The RB5's names are untangled on the names page.

1. Hand tracking: UmeTrack

UmeTrack: camera images in, hand poses out

UmeTrack: camera images in, hand poses out

Meta's code, trained weights and dataset are public [U2]. Licence CC BY-NC 4.0. Archived repo; no hand detector, so it evaluates recordings but does not track live.
About 10 ms for 4 crops on a 2080 Super [U1].

2. The plan

Step	Do	Done when
1. Lab PC	Run UmeTrack's known-skeleton evaluation on the test set.	Mean keypoint error 9.3 to 9.5 mm, separate hands [U2].
2. RB5 CPU	Run only the image encoder on the board and feed its output back into the PC pipeline.	ms per crop; error within 0.1 mm of the PC.
3. RB5 DSP, then GPU	Rerun the encoder on the DSP as int8 and on the GPU as float.	ms per crop and error change per unit.

Deliverable: one table on Sohil's card, with the commands used. Exact commands: UmeTrack on the RB5: the commands

3. SLAM

SLAM: cameras and IMU in, head pose out

SLAM: cameras and IMU in, head pose out

Basalt [S1] via Monado already runs on the RB5: 4.4 cm trajectory error on EuRoC with 2 cameras; the RB5 reaches 92 to 94 °C and the CPU slows about 30 % [L1]. Plan: Edwin's card [N1].

Sources

[U1] UmeTrack: Unified multi-view end-to-end hand tracking for VR: https://arxiv.org/abs/2211.00099
[U2] https://github.com/facebookresearch/UmeTrack; dataset: https://github.com/facebookresearch/UmeTrack_data
[L1] Jacky Liu, Discord #standalone-sw, 2026-10-30
[S1] https://gitlab.freedesktop.org/mateosss/basalt
[N1] SLAM System adaptation on RB5 (Edwin Chen): https://app.notion.com/p/402bc072402f80b7973ad542c273bd00