For years, text-to-motion AI has been a beautiful lie for anyone building interactive worlds. You type a prompt, wait, and get a gorgeous clip — after the whole thing has been computed offline. Great for a render. Useless for a game character that has to react now. NVIDIA’s new ARDY, unveiled for SIGGRAPH 2026, finally collapses that wall: it streams high-fidelity human motion in real time while you steer it with live text prompts and drag its trajectory around like a joystick.
The Story
ARDY — AutoRegressive DYnamics, if you’re keeping score — is a paper by Kaifeng Zhao, Mathis Petrovich, Haotian Zhang, Tingwu Wang, Siyu Tang and Davis Rempe, accepted to ACM Transactions on Graphics (SIGGRAPH 2026) and dropped on arXiv on July 10, 2026. It comes out of NVIDIA’s Spatial Intelligence Lab, and unlike so many “we’ll release code later” papers, the code and models are already on GitHub.
The problem it solves is a genuine fork in the road for motion AI. Offline methods (think MDM, the many MotionDiffusion descendants) give you gorgeous, precisely controllable clips but need the entire sequence in hand before they compute — latency measured in seconds. Online, streaming methods run fast enough for a game loop but historically pay for it: they lose control fidelity, choke on nuanced text, or forget their long-horizon goal because their context window is tiny. ARDY’s whole pitch is refusing to pick a side.
Two ideas do the heavy lifting. First, a hybrid representation: instead of shoving the entire body into one latent blob, ARDY keeps the root (the pelvis trajectory — where the body goes and which way it faces) as explicit, human-readable features, and compresses the rest of the body into a learned latent embedding. That split is the trick: explicit root = precise navigation control; latent body = efficient, natural-looking generative learning. Second, a two-stage autoregressive transformer denoiser with a variable history window, so it can lean on more past context when a motion needs continuity and less when it needs to pivot fast.
Why You Should Care
Because “controllable” here isn’t a single knob — it’s a whole console. ARDY accepts online text prompts (change your mind mid-motion and the body responds), plus a stack of kinematic constraints layered on top: root waypoints and full trajectories, full-body keyframes, and sparse individual joint positions or rotations — over long horizons. That’s the vocabulary a technical animator or a gameplay programmer actually works in.
- Game devs: a locomotion system you drive with a stick and retarget with a prompt is exactly the missing middle between clip-based state machines and expensive physics controllers. Steer the root, whisper “now sneak,” and keep the frame budget.
- Previz & virtual production: block a shot by dragging the trajectory and typing intent, watch it play back instantly, iterate before you commit a mocap day.
- 3D & character artists: because the output is SMPL-style body motion driven by keyframes and joint constraints, it slots into the rigging and retargeting pipelines you already run in Blender, Maya or Unreal.
- Humanoid robotics: the authors explicitly frame streaming, constraint-aware motion as a stepping stone for embodied agents — the same reason a robot arm and a game avatar increasingly share a research lineage.
Try It / Follow Them
- Project page (video results, the interactive demo shown above): research.nvidia.com/labs/sil/projects/ardy
- Code & model releases: github.com/nv-tlabs/ardy
- The paper: arXiv 2607.08741 (ACM ToG / SIGGRAPH 2026)
- Lead author Kaifeng Zhao: zkf1997.github.io
IK3D Lab Take
We’ve spent months here watching Gaussian splats teach machines to see and rebuild space. ARDY is the other half of that story: teaching machines to move through it, live, under your hand. The hybrid trick — keep the trajectory explicit, hide the body in a latent — is one of those ideas that feels obvious the second you read it and quietly reframes the problem. It’s the same “give the human the steering wheel, let the model handle the physics of the wrists” instinct we keep applauding.
Honest caveats: this is body motion, not fingers or faces (pair it with something like Audio2Face for the head), the fidelity ceiling is still set by the mocap data it trained on, and real-time on your hardware will depend on the model tier NVIDIA ships. But code is on GitHub today, and a real-time, promptable, joystick-steerable motion engine is precisely the kind of primitive that ends up quietly powering a hundred tools next year. Clone it, drag a trajectory, type “sneak, then sprint,” and watch the wall between offline polish and interactive control come down.
