Confidential · Internal
Architecture · Video Intelligence

Overwatch — the live video-understanding agent

Point a camera at anything. Deterministic code does the perception — detect, track, measure — and LLMs do the meaning: judging significance, narrating in any persona, watching for conditions, acting on them, predicting what happens next, and answering questions about everything it has seen. The two layers check each other: every narrated claim is verified against measured geometry, and every prediction is scored against what actually happened.

Browser Deterministic — no LLM, local LLM — gated Memory / storage The frame’s journey Measurements verify claims Your input re-enters
L1 Frames the camera stream, thinned Source webcam · video file · sample clip Motion-gated capture ~6 fps baseline, bursts on movement → WebSocket BROWSER jpeg frames L2 Object tracking what is where, over time Detect — open vocabulary YOLOE · profiles: generic 640 / wide 1280 Persistent IDs + re-ID BoT-SORT tracks · appearance gallery re-links returns Trajectories → motion measured paths · a wave ≠ a slide NO LLM · LOCAL objects + measured paths L3 Event triggers when is it worth looking? Triggers — incl. relationships appeared · moved · stopped · gone · near(A,B) · handoff(A,B,item) Cost gates — the spend firewall idle · cadences · backoff · salience: background clutter = $0 NO LLM · GATES only moments worth spending on L4 Object identification what is each thing? Batch naming — scene-aware crops interpreted in the classified scene ("a soccer match…") Active perception — look closer zooms into a persistent low-confidence object to resolve it LLM · FLASH-LITE named, tracked things L5 Understanding what does it mean? Narrator — significance + personas judges the 4-frame moment; may stay silent · packs = data Grounding verifier — the lie detector claim vs L2 measurements → ✓ grounded / ⚠ unverified Predictor — predict, then verify calls the next event, scores itself → accuracy % Watcher — conditions & if-then actions sees the condition, composes & speaks the “then” LLM · FLASH committed moments + labels L6 Memory & recall what happened — ask anything Object registry taught names & galleries — survive restarts (identity store) Timeline narrated moments + frame sequences; identity is a property World store — full-rate video 12fps record, sparse detect · object↔frame index · REST /store History recall + World Explorer LOOK NOW / moment / BACK / AT a time · names join retroactively STORE · LLM ON ASK the story + your answers L7 Delivery back to the human Overlay boxes · trajectories · zoom pulse Conversation narration ✓/⚠ · alerts · answers Voice + anticipation spoken lines · prediction card · 🔮 % BROWSER measurements verify claims your tasks · questions · if-then
01 · The loop

One frame’s journey: the browser sends a motion-gated JPEG → the tracker updates IDs and trajectories → deterministic triggers decide when to look → the memory records everything measurable → and only then, behind cost gates, do LLMs decide what it means — narrate it, verify it, predict from it, or act on it. Results stream back over the same WebSocket as overlay geometry, conversation, and voice.

The core split, and why it matters: tracking never decides significance, and the LLM never does geometry. The deterministic layer decides when to look and supplies measured truth; the LLM decides what it is and whether it matters — and is held accountable to the measurements. This is what a naive “feed frames to a VLM” pipeline cannot do.
02 · Distinct capabilities

Every capability is generic — none of them knows what soccer is. Domain flavor (the “GOAAALLL!”, the hushed documentary awe) lives in persona packs: pure data entries bundling match keywords, narration style, cadence, detection profile, and voice.

Anti-hallucinationGrounded self-verification every narrated claim is cross-examined against measured geometry›

What it isAfter the narrator commits a line, a verifier checks the claim against the tracker’s measured state — each in-scene object’s identity, position, and trajectory-derived motion — and badges it ✓ grounded or ⚠ unverified (with honest abstention when there is nothing trackable to check). The UI highlights the supporting objects and draws their measured paths on the video: language bound to pixels.

Why it mattersThe deterministic layer acts as a lie detector for the LLM — “a dog ran across” is flagged when no dog was ever tracked; “he sprinted away” is flagged when the person measured still.

Active perceptionThe agent decides to look closer a closed perception–cognition loop›

What it isA persistent low-confidence detection makes the agent crop that region, upscale it, and look again with the vision model — resolving what the full-frame pass could not. Task-gated, rate-limited, once per object.

AnticipationPredict-then-verify it calls the next event, then scores itself — a number, not a vibe›

What it isWhile commentating, the agent periodically predicts the single most likely next event as a specific, checkable claim with a time horizon; when the horizon elapses it grades itself against what the narrator actually recorded. Running accuracy is always visible. Unscoreable windows don’t count.

AgencyWatches & conditional actions “if someone holds up a coffee cup, ask them what kind it is”›

What it isWatches inspect the live frame for the condition (a real detector, not keyword-matching), with per-condition cooldowns and scene-change gating. A conditional task carries a then: when the watch fires, the agent composes the actual line from the live frame and speaks it — and the utterance joins the timeline as part of the story.

MemoryVisual memory independent of narration “what WAS on his hat?” is answerable even if narration never ran›

What it isEVERY processed frame is recorded to rolling video segments, with every detection indexed into an object↔frame graph (SQLite locally; Blob + Cosmos in the cloud) — browsable in the World Explorer — a session browser and debug tool (pick any past recording, scrub with raw per-frame metadata, inspect any object's full history). Chat retrieves visually four ways: LOOK NOW (live frame), LOOK <moment> (a narrated moment’s sequence), LOOK BACK (recent snapshots), LOOK AT a time (the recording) — all carrying conversation context so pronouns resolve, all named with what is known NOW.

IdentityAppearance re-ID — identity survives track loss leave the room and return: “🔁 recognized returning: Dennis”›

What it isStable objects get local appearance embeddings (no LLM). When a NEW track stabilizes, it is matched against the galleries of ABSENT objects and merged on a confident match — name, taught facts, and full history inherited, and the object↔frame graph unifies across the re-entry. Identity is a property of the object, not the track: names also apply retroactively to earlier frames (the person who was turned away). Conservative gates: candidates must be absent (never co-present) and require class agreement or overwhelming similarity.

Why it mattersNamed identities also persist to a cross-session store at a stricter threshold — "Dennis" survives a restart ("🌐 recognized from a previous session"). The same gallery matched across feeds is the multi-camera identity mechanism — in the cloud, a vector index shared by all cameras.

Cost disciplineA quiet scene spends nothing every LLM call site sits behind a deterministic gate›

What it isIdle gates, per-capability cadences, adaptive NONE-backoff (strong events punch through), scene-change gating for watches, batching and per-object retry caps for identity, single-flight latest-wins TTS, and strict-JSON verdict roles on the cheapest model tier. Fully idle ≈ $0/hr; measured worst-case persona commentary ≈ $2–3/hr.

03 · Where it fits

Overwatch is the video-intelligence limb of the LumicIQ ecosystem (see the feature architecture map): the same deterministic-perception → gated-LLM pattern that the platform applies to retail event streams, applied to live camera feeds. In production, Overwatch’s deterministic perception layer ships as the platform’s .NET edge service (LumicIQ.Edge, ADR-0009): the same detect → track → re-ID runs at the edge and emits typed video events plus a world-state aggregate onto the spine, while the expensive visual “meaning” becomes a cloud cascade over archived segments — perception at the edge, meaning in the cloud. Its trigger/gate discipline mirrors the platform’s InterestingnessDetector; its grounded verification is the same audit instinct as the immutable LlmCallRecord; and its narrated moments and watch events are natural RelayEvent producers for the store agent.