Changes to Spec:

Point Persons:

Table of Contents

Introduction

https://lh7-rt.googleusercontent.com/docsz/AD_4nXfAhUT28rynNlZwiyoJWw1l8fIkaPH0ohD-bkfJYqiF9-k3q_EbXJIS4cJgOlt-F7lApl1CJxTRIviOwHY8aATEWhDtRoH67vU4J7ChI1JZUG-phDr-lQPk4VNLS2JdbbLooMpPI5UR8imND2KCf0wG9tBR?key=saRBwJK3_3Nj56HudTMcag

When you prompt a virtual assistant (for example Meta AI on Raybans glasses), what happens when you ask “What am I looking at”? Currently, the pipeline seems rather simplistic. The cameras on the glasses take a picture, that picture is passed through a model that can assign text labels to images, and finally that text label describing the whole image is passed into an LLM. This process, especially the step where a model must describe everything in an image using words, is often inaccurate.

What if we could build a system that…

• …provides a richer text summary of a virtual environment, complete with descriptions of how objects compose each other, are placed within/next to/on top of each other?
• …also describes how you, the user, is interacting with that environment at any moment? Could we assign additional text to describe that you are pointing at a specific object, or reaching out for one?
• …runs in real time, that is, can constantly update every frame to provide an updated description. That way, we wouldn’t have to wait for text generation, and we could create a live captioning system?
• …runs entirely on-device, meaning this information is never sent to the cloud?

If we created this, we could use it for…