Our goals are to get computer vision algorithms running on our Qualcomm, RB5, and EB5. There's a few different questions that this team's gonna be answering for research.

<aside> 💡

First off, we need to ensure that everyone on the team understands the hardware at a high level. Therefore, an onboarding document for this would be prudent. We just need information to get started ASAP so it won’t be that long in length.

</aside>

Intro to CV

I will write these soon

• Vision of the headset
• The CV algorithms we want
  • SLAM
  • Hand tracking
  • What does it mean to have spatial understanding
• Anatomy of the Standalone hardware from a software POV
• Crash course in ML inference
• Where this could lead members in terms of job search :yipee:

Team Tasks

• Hand estimation software: that should be the most trivial. We need hand positions and joint poses. Kind of like how we do this for Universal Gestures. Perhaps we want to use Mercury, which is already used by Monado (Monado is already on our headset) or maybe MediaPipe Hand Landmarker, although I'm not sure about the overhead because I think MediaPipe is Python (someone needs to look into that as well).
• We need to figure out what SLAM algorithms can be on device, and is supported by our hardware. Perhaps the Basalt XR, OpenVINS, VINS-Fusion, ORB-SLAM3, OKVIS2. I need someone to do research on these and figure out what they all mean
• We're tracking image features and adding acceleration in angular velocity, such that we can input this information into the computer vision algorithms. So we also need to figure out a way to, get the inputs for the 4 tracking cameras, perhaps the hardware team is already doing this, so I need to sync with them,
• Stretch goal: we need to be able to recognize places and maintain an environment map. This is further along in the scope as I think spatial understanding is a bit difficult. We need to manage memory/create some kind of agentic architecture instead of using just a pure CV algorithm.
• Gaze estimation. We need eye image processing. Eye/gaze representation, camera-to-headset transforms, user calibration. This does sound a bit like what AdHawk (a company in Waterloo) is doing, and they were in contact with UWRL at one point so maybe we could ask them. In the meantime, open-source alternatives could be: pupil-detectors, or pye3d.
• We need to execute computation on the processors. How are we going to run on each processing block?
  • For the CPU, it is my understanding that we can use existing algorithms without changing anything.
  • For the GPU, we may need to use Vulkan so that it's cross-platform. There’s an existing implementation called VkVIO, which describes the Vulkan versions of Basalt frontend operations.
  • For the DSP and HVX, we must write Qualcomm Hexagon kernels. I am not familiar with these! Someone’s going to have to give us a deep dive on these. There is something called FastRPC which “provides CPU-to-DSP calls”. Moreover, there is something else called FastCV which provides optimized CV functions.
  • Neural acceleration for executing model computation graphs: someone will have to look into SNPE or a compatible QNN backend
  • ISP, vision coprocessor, FPGA, other DSPs. I need to sync with Vincent/hardware to understand what these mean.