This is a somewhat experimental idea that I want to see if it is possible. If this is achievable, it would be hugely helpful for not only adding new functionality to Universal Gestures models, but mapping whole arm movements to our Humanoid robot.

This task involves researching how it might be possible to estimate where the user’s elbow is, based on where their hand is relative to their head. You can imagine if a person’s hand is very close in the horizontal plane to their head, that the elbow would be fully bent. If their hand is far away, their arm is probably fully straight.

There may be some Meta API already in the All-in-One SDK that can do this. Last time I heard about Meta talk about elbow estimation, it was something they were actively researching. So the first stop would probably be to check if this exists in Meta documentation.

If Meta’s advanced estimation isn’t something we can use yet, we can use more primitive methods. Most VR games like VRChat have very simple character rigs that animate the arm bending in and out based on hand position. Try finding some YouTube videos that might teach you how to implement this on our character. If we’re able to render a pretty good point of where the elbow of a virtual arm is (as a red dot), that’s a start. If we can also get an estimated number for how bent the elbow is (in degrees), that’s even better.

If the Unity task is successful in being able to get this information, we can move onto a new PyTorch task that involves creating a new model that also takes in elbow position/angle as an input. That way, we add the ability to recognize a gesture like facepalm, using elbow flexion as a new input.