Standalone architecture rev 2

Hardware Components

Tracking cameras: AR0234 ×4
Hand/eye cameras: SC035HGS ×4
RGB camera: IMX678 x1
IMU: model and quantity unspecified. The diagram connects the IMU to the FPGA over SPI.
FPGA: model and quantity unspecified. Functions: triggering, timestamping, and CSI aggregation.
Compute platform: RB5; board variant and quantity unspecified. The diagram routes MIPI CSI-2 virtual channels to RB5.
Display panels: DSI panels; model and quantity unspecified.
Optics: Northstar; specific hardware and quantity unspecified.

Software High Level | Detailed

flowchart TD
    HW["Hardware platform<br/>(hardware lead)"]
    KERNEL["Kernel drivers<br/>cameras, IMU, display, GPU, DSP"]
    PLATFORM["Platform services<br/>sensor service, Vulkan, OS"]
    CV["CV algos<br/>SLAM, hands, eyes, spatial"]
    RUNTIME["XR runtime (Monado)<br/>OpenXR, compositing"]
    LAUNCHER["Launcher<br/>Godot, 2D windows, browser"]
    APPS["Apps<br/>OpenXR and 2D"]

    HW -->|"sensor data"| KERNEL
    KERNEL -->|"frames + IMU"| PLATFORM
    PLATFORM -->|"frame sets"| CV
    CV -->|"poses, hands, gaze"| RUNTIME
    KERNEL <-->|"display output / vblank"| RUNTIME
    RUNTIME <-->|"OpenXR"| LAUNCHER
    RUNTIME <-->|"OpenXR"| APPS
    LAUNCHER <-->|"Wayland"| APPS

    classDef abstraction fill:#F1EFE8,stroke:#5F5E5A,stroke-dasharray:5 5
    class HW abstraction

flowchart TD
    HW["Hardware platform (hardware lead)<br/>sensors, FPGA + firmware, sync and timestamping,<br/>IMU, RB5 compute units, displays"]

    subgraph KERNEL["Kernel drivers"]
        CAMDRV["Camera capture<br/>camss V4L2, sensor + FPGA drivers"]
        IMUDRV["IMU driver<br/>IIO"]
        KMSDRV["Display<br/>msm DRM/KMS + VS035ZSM panel"]
        GPUDRV["GPU<br/>msm DRM (Adreno)"]
        RPCDRV["DSP/NPU access<br/>FastRPC"]
    end

    subgraph PLATFORM["Platform userspace (Arch Linux ARM)"]
        SENSOR["Sensor service<br/>capture, frame sets, IMU stream"]
        TURNIP["Mesa Turnip (Vulkan)<br/>used by compositor, Godot, CV"]
        HEXRT["Hexagon/NPU runtime<br/>toolchain TBD"]
        OSSVC["Boot and services<br/>systemd units, package repo"]
    end

    subgraph CV["CV algos"]
        subgraph TRK["Perception"]
            SLAM["SLAM"]
            HAND["Hand tracking"]
            EYE["Eye tracking"]
            SPATIAL["Spatial understanding"]
        end
        CVCAL["Camera calibration<br/>offline tool"]
        COMPUTE["DSP/NPU/CPU optimization<br/>compute backends"]
    end

    subgraph RUNTIME["XR runtime (Monado)"]
        MDRV["Headset device driver<br/>xrt_device + builder"]
        OXR["OpenXR context"]
        DISPCAL["Display/optics calibration"]
        subgraph XRCOMP["XR compositing (hw accelerated)"]
            UNDISTORT["Undistort"]
            TIMEWARP["Time warp"]
        end
    end

    subgraph LAUNCHER["Launcher/desktop environment"]
        GODOT["Godot 3D environment"]
        WAYLAND["Wayland compositing<br/>2D app windows"]
        BROWSER["In-launcher browser"]
        WEBXR["WebXR support?"]
    end

    OTHERXR["Other OpenXR apps"]
    NONXR["Non-XR apps<br/>Wayland clients"]

    HW -->|"frames, timestamped"| CAMDRV
    HW -->|"samples, timestamped"| IMUDRV
    HW <-->|"DSI scanout / vblank"| KMSDRV
    HW <--> GPUDRV
    HW <--> RPCDRV

    CAMDRV -->|"V4L2 buffers (dma-buf)"| SENSOR
    IMUDRV -->|"IIO samples"| SENSOR
    GPUDRV <--> TURNIP
    RPCDRV <--> HEXRT

    SENSOR -->|"frame sets + IMU"| TRK
    SENSOR -->|"raw frames"| CVCAL
    CVCAL -.->|"calibration files"| TRK
    TRK --- COMPUTE
    TURNIP <-->|"Vulkan compute"| COMPUTE
    HEXRT <-->|"offload"| COMPUTE

    TRK -->|"XR sub system interfacing<br/>poses, hand joints, gaze, scene"| MDRV
    MDRV --> OXR
    OXR -->|"layers + predicted poses"| XRCOMP
    DISPCAL -.->|"distortion model"| UNDISTORT
    KMSDRV <-->|"atomic commits / vblank timestamps"| XRCOMP

    OXR <-->|"XR communication<br/>OpenXR API"| GODOT
    OXR <-->|"OpenXR API"| OTHERXR
    OXR -.->|"OpenXR (tentative)"| WEBXR
    WAYLAND -->|"window textures (dma-buf)"| GODOT
    BROWSER -->|"Wayland client"| WAYLAND
    BROWSER --- WEBXR
    WAYLAND <-->|"Wayland protocol"| NONXR
    GODOT -.->|"launches"| OTHERXR

    classDef abstraction fill:#F1EFE8,stroke:#5F5E5A,stroke-dasharray:5 5
    classDef tentative stroke-dasharray:4 4
    class HW abstraction
    class WEBXR,HEXRT tentative

Computer Vision | OpenXR Runtime

Rough Rsch

flowchart TD
    SENSOR["Sensor service"]
    TURNIP["Mesa Turnip (Vulkan)"]
    HEXRT["Hexagon/NPU runtime<br/>toolchain TBD"]

    subgraph CV["CV algos"]
        subgraph TRK["Perception"]
            SLAM["SLAM"]
            HAND["Hand tracking"]
            EYE["Eye tracking"]
            SPATIAL["Spatial understanding"]
        end
        CVCAL["Camera calibration<br/>offline tool"]
        COMPUTE["DSP/NPU/CPU optimization<br/>compute backends"]
    end

    MDRV["Monado headset device driver"]

    SENSOR -->|"frame sets + IMU"| TRK
    SENSOR -->|"raw frames"| CVCAL
    CVCAL -.->|"calibration files"| TRK
    TRK --- COMPUTE
    TURNIP <-->|"Vulkan compute"| COMPUTE
    HEXRT <-->|"offload"| COMPUTE
    TRK -->|"XR sub system interfacing<br/>poses, hand joints, gaze, scene"| MDRV

    classDef external fill:#F1EFE8,stroke:#5F5E5A
    classDef externalTentative fill:#F1EFE8,stroke:#5F5E5A,stroke-dasharray:4 4
    class SENSOR,TURNIP,MDRV external
    class HEXRT externalTentative

flowchart TD
    TRK["CV algos (Perception)"]

    subgraph RUNTIME["XR runtime (Monado)"]
        MDRV["Headset device driver<br/>xrt_device + builder"]
        OXR["OpenXR context"]
        DISPCAL["Display/optics calibration"]
        subgraph XRCOMP["XR compositing (hw accelerated)"]
            UNDISTORT["Undistort"]
            TIMEWARP["Time warp"]
        end
    end

    KMSDRV["Display driver<br/>msm DRM/KMS + VS035ZSM panel"]
    GODOT["Godot launcher"]
    OTHERXR["Other OpenXR apps"]
    WEBXR["WebXR support?"]

    TRK -->|"XR sub system interfacing<br/>poses, hand joints, gaze, scene"| MDRV
    MDRV --> OXR
    OXR -->|"layers + predicted poses"| XRCOMP
    DISPCAL -.->|"distortion model"| UNDISTORT
    XRCOMP <-->|"atomic commits / vblank timestamps"| KMSDRV
    OXR <-->|"XR communication<br/>OpenXR API"| GODOT
    OXR <-->|"OpenXR API"| OTHERXR
    OXR -.->|"OpenXR (tentative)"| WEBXR

    classDef external fill:#F1EFE8,stroke:#5F5E5A
    classDef externalTentative fill:#F1EFE8,stroke:#5F5E5A,stroke-dasharray:4 4
    class TRK,KMSDRV,GODOT,OTHERXR external
    class WEBXR externalTentative