Every world model the Lab has covered this year answers the same question: can you generate a place? Marble hands you splats, Genie hands you a playable dream, Cosmos reasons a scene into being. WorldDirector, dropped on arXiv on July 2 by a team from HKUST and Ant Group, asks a harder one: can you direct that place — and will it remember what happened while you weren’t looking?
The Story
Here’s the dirty secret of AI video-world models: they have no memory. Pan the camera away from an object and back, and the model quietly re-hallucinates it. Your red sports car comes back blue. The dog you left in the corner is gone, or it’s a different dog. Because these systems entangle physics and pixels in one giant denoising pass, “what should exist” and “what it looks like” are the same tangled variable — so consistency evaporates the moment something leaves the frame.
WorldDirector’s core move is to rip those two jobs apart. A large language model acts as the director: given a text description, it plans the actual 3D trajectories of every object and the camera through time — where the woman walks, where the car sits, where you’re standing to watch. Those 3D plans get projected down into 2D location controls (literally colored boxes telling the video model “car goes here, person goes there”). Only then does the video generator paint pixels, guided by those controls plus an appearance condition — a locked reference of what each object looks like — so identity is bound to the object, not re-guessed every frame.
The result is what the authors call persistent dynamic object memory. Objects don’t just sit frozen off-screen — they keep evolving (the car keeps rolling, the person keeps walking) according to the 3D plan, and when the camera swings back they’re exactly where causality says they should be, wearing exactly the appearance they left with. Because the world is generated causally, chunk by chunk, WorldDirector can string together long-horizon sequences — a horse trots off, a bus rumbles through, a person walks up — all in one coherent take, with the camera free to roam.
Why You Should Care
If you make anything with a camera and a story, “the world forgets” has been the wall. It’s why AI video clips are 5 seconds long and why “playable world” demos fall apart the second you turn around. WorldDirector reframes world generation as blocking a scene — the exact thing a film director or a level designer already does in their head. You’re not praying to a prompt; you’re placing actors and a camera in 3D space and letting the model render the coverage.
That’s a big deal for the people who read this Lab: previz artists who want to choreograph a shot before touching a render farm; game devs who need an NPC to still be there (and be the same NPC) after the player looks away; comic and animation makers fighting the eternal character-consistency war. And the LLM-as-director layer means the interface is language — you describe the blocking, the model handles the geometry. It’s the same “reason first, render second” philosophy we saw in Cosmos 3 and PixelDiT, now pointed at object permanence instead of image quality.
Try It / Follow Them
- Project page & demos (watch these first): worlddirector.github.io
- Paper: arXiv:2607.02517
- Code (full inference released): github.com/pPetrichor/WorldDirector
- Weights —
WorldDirector-14Bon Hugging Face: hlwang06/WorldDirector
This is a 14B model with a real appetite for VRAM, so it’s a “clone the repo on your beefy GPU” affair, not a one-click web toy — yet. But the inference code and weights are already public, which for a paper this fresh is genuinely generous. Fair warning from the honest-enthusiast desk: the open release is the research build, the demos are cherry-picked, and long takes still drift. This is a frontier you tinker with, not a product you ship Monday.
IK3D Lab Take
We’ve spent 2026 watching world models get bigger — more splats, more resolution, more square kilometers. WorldDirector is the first that gets meaningfully smarter about the boring problem that actually blocks creative work: memory. Decoupling the choreography from the rendering, and letting an LLM hold the clipboard, is the kind of architectural honesty that tends to age well. It won’t out-pretty Marble this week. But “the world remembers, and you can direct it” is the sentence every previz, game, and animation pipeline has been waiting to hear — and this is the first paper that backs it with open code. Bookmark it. This idea is going to get borrowed.
