Behavioral World Models
A generative simulator of human interaction — the engine for believable social presence. · Wojciech Zielonka
The one-line thesis
"Behavior prediction is social intelligence" — the direct analog of "video prediction is robot control." Pretrain on human video at web scale (self-supervised behavior prediction) → the model becomes a simulator of how people communicate. That simulator can then (a) drive autonomous, human-like avatars/agents, and (b) predict/anticipate a real human's behavior for perception, telepresence, and HCI.
What is a behavioral world model?
A world model predicts how the world evolves given actions. A behavioral world model treats the human(s) in the interaction as the world, and predicts the joint next behavioral state given the interaction context. Two things fall out of the same model:
- Generative mode — produce the agent's next behavior → drives an autonomous, expressive, human-like avatar (not keyframed, not puppeted — reactive).
- Predictive mode — forecast the human's next behavior → anticipation for HCI/AR, and driving a telepresence avatar of a real person from sparse signals.
Crucially it is multimodal and simultaneous: real human behavior is speech + face + gaze + gesture + timing happening at once, and it's full-duplex — you listen and react (backchannels, micro-expressions) while the other person is still talking. Today's cascade pipelines (STT → LLM → TTS → talking head) are inherently turn-based, high-latency, and error-propagating. A behavioral world model is end-to-end and full-duplex by construction.
The formulation (what the model actually predicts)
Let s_t be the behavioral state at time t — the joint speech / expression / gaze / head-pose / gesture of the agent and the partner. Given the interaction history and context, the world model learns
p( s_{t+1} | s_{≤t}, context, intent )— a distribution over the next behavioral chunk. The behavior is the "action" (the analog of a robot's motor command), context is the perceived scene + partner, and intent is the optional high-level goal from the brain. Rolling the model forward — sampling s_{t+1}, s_{t+2}, … — simulates a person behaving over time.
The architecture - a Genie for human behavior (autoregressive video model)
It follows Genie's recipe - but the "playable world" is people behaving and interacting with their environment, and the "actions" are human behaviors. Trained on unlabeled human video (Instagram / Reels-scale, no behavior labels) via Genie's three components:
| Genie component | Behavioral Genie (humans) |
|---|---|
| Latent Action Model - infers latent actions from action-less gameplay | infers latent behavioral actions from action-less human video - the learned control space of behavior (no labels). |
| Video tokenizer (VQ / ST-transformer) | tokenizes the human video + behavioral state (face - gaze - pose - gesture) + the environment. |
| Dynamics model - AR transformer: next frame | past + action | autoregressive video model: next behavioral frame | past behavior + latent behavioral action. |
| learns controllable game worlds from unlabeled video | learns controllable human behavior & interaction with the environment from unlabeled video. |
At inference the latent behavioral action is either supplied by a high-level "brain" (an LLM / intent signal) for an autonomous agent, or inferred from a real person for telepresence. The dynamics model rolls out behavior frame-by-frame - an autoregressive video world model of humans, made real-time & streamable by construction (causal AR + KV-cache), drift-controlled (BAgger) for long interactions.
The world-model connection (the concepts, reused)
| World-model concept | In a behavioral world model |
|---|---|
| Pre-train the prior (video prediction learns physics) | Self-supervised behavior prediction on massive human video (Instagram/Reels-scale) → a prior over how people behave. |
| Post-train to ground/act (attach an action head) | Ground to persona / goal / a specific person; the "action" is the next behavior chunk (speech+expression+gaze+gesture). |
| Autoregressive + KV-cache (streamable) | Full-duplex real-time conversation — block-causal, cached, chunked behavior. |
| Causal by construction (Genie-native AR) | Train autoregressive from the start → real-time & streamable natively; no bidirectional→causal distillation (CausVid) needed. |
| Drift / exposure bias (BAgger / Self-Forcing) | Long conversations drift → train on the model's own rollouts to stay coherent over minutes. |
| Dual-system (slow world model + fast action expert) | Slow brain (semantics) + fast behavior head (reactive signals) — decoupled rates. |
| Genie (latent actions, action-less video) | The core recipe here - an LAM learns the behavioral control space from unlabeled human video; an AR dynamics model generates the behavior. |
| Latent world model (V-JEPA / Dreamer) | Predict behavior in latent space for cheap anticipation without rendering (the predictive mode). |
How a small team ships this on tens of GPUs using Meta's data + pretrained models — not 5K GPUs from scratch.
The goal & how to drive it
1 Requirements → 2 the scoping constraint (small team, no 5K GPUs → leverage, don't pretrain) → 3 the end-to-end architecture → 4 the three planes (data · model · serving) → 5 evaluation, privacy, and a phased plan.
Requirements
| Functional | Non-functional |
|---|---|
| • Given multimodal context (partner's speech · face · gaze), generate the agent's next behavior (speech + expression + gaze + gesture). • Multimodal & synchronized (audio-visual, lip + co-speech). • Full-duplex — listen & react while the partner speaks. • Controllable (persona / intent from an LLM brain). • Drive a photoreal avatar; also a predictive mode (anticipate a real human). | • Real-time — perceived latency well under a turn (~100–200 ms). • Streamable (KV-cache, causal). • Data-efficient — little labeled data. • Small compute — trainable on tens of GPUs, not thousands. • Privacy/consent-compliant (GDPR). • Small team (≈2–5) → maximize reuse. |
The scoping principle — leverage > build ★
With a small team and no 5K GPUs, every "train from scratch" is replaced by "adapt something that exists." Three levers make it tractable:
| Lever | Why it collapses the cost |
|---|---|
| Action-less pretraining (Genie) | The Latent Action Model learns behavior controls with NO labels → no expensive behavior annotation, and unlabeled human video (Instagram/Reels-scale, which Meta has endlessly) becomes usable. |
| Reuse pretrained pieces | Work in a frozen pretrained tokenizer / VAE latent (the expensive visual representation is reused, not relearned) and optionally init the AR transformer from a pretrained LLM. You train only the modest behavioral dynamics → tens of GPUs. |
| Causal from the start (Genie-native) | Train the dynamics autoregressively from the start → real-time & streamable by construction; there is no bidirectional teacher and no CausVid step to build or maintain. |
| Reuse the renderer | The avatar decoder is my existing work (INSTA/GEM/SynShot) — the team never trains a renderer. |
The system architecture — data · model · serving
Data — what Meta has (the reason to do it here)
Meta owns an unmatched corpus of in-the-wild human video (Instagram / Reels-scale). It's mostly single-person video (people talking to camera, reacting) — not curated two-person interaction — but that is exactly what the action-less Genie prior feeds on. Curated interaction data (for full-duplex turn-taking) is the scarcer piece, which shapes the phasing. The real constraint throughout is consent/privacy, not availability — which plays to my strength (GDPR + synthetic priors in SynShot).
| Source | What it gives | Caveat |
|---|---|---|
| Codec Avatars capture (Reality Labs, Pittsburgh) | Gold: multiview, tracked, consented face/body/audio — clean behavioral signals + drives the renderer. | limited scale/diversity → pair with in-the-wild. |
| Movie Gen (Meta's video foundation model) | A frozen pretrained tokenizer / VAE to reuse (plus an optional LLM init for the AR transformer) — no from-scratch pretraining. | reuse frozen; we don't retrain its weights. |
| Instagram / Reels (in-the-wild human video) ★ | The primary source — huge, unlabeled video of people talking & expressing → the action-less behavioral prior (Genie LAM). Mostly single-person, which is enough for the prior. | strict consent + privacy filtering; not dyadic. |
| Ego4D / Ego-Exo4D (FAIR) | Egocentric human interaction dynamics. | egocentric viewpoint. |
| VR social (Horizon / Quest) | Avatar interaction with tracked control signals → helps ground latent actions. | avatar-domain, not photoreal. |
| Messenger / WhatsApp calls | — | OFF LIMITS — E2E-encrypted, private. Do not use. |
Data pipeline & captioning
The Genie recipe means no behavior labels — the biggest small-team win. What the pipeline does:
- Curate: filter for people/faces & social content (dyadic where available), shot/scene cuts, face/person detection, quality, dedup, consent/safety.
- Extract behavioral signals with existing pretrained trackers (my INSTA/3DMM tracking, gaze, pose, ASR, audio features) → the per-frame multimodal behavioral state. No manual annotation.
- Caption (light): a VLM caption + ASR transcript for the semantic/language stream (feeds the LLM brain & conditioning) — cheap, batch.
- Multimodal tokenize once: audio codec (EnCodec/Mimi) + video VAE + text tokens + behavior params → pre-tokenized sharded dataset (the data-efficiency trick: encode once, not every epoch).
Model & training
- Tokenizer (reused): a frozen pretrained tokenizer/VAE (video) + audio codec + text — reuse the visual representation, don't relearn it.
- Genie core (what we train): a multimodal Latent Action Model (infers behavioral actions from unlabeled video) + a causal autoregressive dynamics model over the interleaved audio+video+text+behavior token stream (Moshi-style multi-stream). Optionally init the AR transformer from a pretrained LLM.
- Small & native: the dynamics is causal from the start (real-time by construction) and modest (narrow domain + small action space) → tens of GPUs, weeks. No teacher–student distillation to run.
- Real-time & drift: the model is streamable by construction (no CausVid); BAgger / self-forcing controls drift over long conversations (any AR rollout drifts).
- Renderer: reuse the existing INSTA / GEM / SynShot avatar pipeline — behavior parameters in, photoreal head out. The team never trains a renderer.
Serving (real-time full-duplex)
The real-time toolkit is standard: causal + KV-cache (O(n) decode), native autoregressive decode (one pass per token — no diffusion-step distillation), behavior chunking (buffer ~0.5 s locally), latent-space (emit behavior tokens, render once). Target a ~100 ms perceived budget. Full-duplex = the model runs a listening stream (backchannels, micro-reactions) and a speaking stream simultaneously (Moshi's multi-stream). Server-side for the MVP; on-device later for VR.