Models#Video generation#Model inference

Ant opens Duo v1, a full-duplex world model that listens while it talks

Ant Research opened Duo v1, a full-duplex world model that listens as it replies in 708 ms chunks. Code is Apache-2.0; weights exclude the US, EU, UK and Korea.

The teaser figure from the Duo paper: ten split-screen interaction scenes around the Duo logo, labelled Customization, Action, Guidance and others; bottom left shows a user raising a bottle and the character responding

Ant Research, the research arm of Ant Group, announced the release of Duo v1 on November 13: weights on Hugging Face, inference code on GitHub under Apache-2.0, and a paper that went up on arXiv as 2611.15131 two days earlier. Duo v1 is a full-duplex world model. It slices an interaction into chunks of 17 video frames — about 708 ms at 24 fps — and in each chunk an Intent module decides whether to speak (<speak> plus text) or keep listening (<wait>), before a Realization module turns that decision into synchronized audio and video. The team says the model can interject while you are still talking, or stay silent, rather than waiting for its turn.

Key points

  • Full-duplex means two streams at once: user audio and video keep arriving while the model keeps producing audio and video, with a speak-or-wait decision every 708 ms. The comparison is a world model steered by keyboard, mouse or text, and the turn-based ASR→LLM→TTS avatar pipeline.
  • The 93.3% is the authors grading their own work: 30 cases against six binary criteria gave 180 checks, and 168 passed. Language adherence, speech intelligibility and visual identity stability went 90/90; content responsiveness reached 22/30 (73.3%). The paper has no head-to-head comparison with another model.
  • Code and weights carry different licenses: the GitHub code is Apache-2.0; all four Hugging Face weight repos use the MiniMax H3 community license, which excludes the EU, UK, South Korea and the US. The repo ships inference code only, with no training scripts or entry point.

Background

Most world models simulate physical scenes: they watch past frames and control signals, predict what happens next, and let a user steer with a keyboard, mouse, gamepad or text prompt. We covered Google DeepMind’s Genie 4, which generates audio natively and lets you rewrite a running scene, but the loop is still command-in, world-out. The other reference point is the digital human: speech recognition, a language model, then speech synthesis, one turn at a time.

Duo v1 targets that interface. The paper treats intelligence and physical reality as equally fundamental to a world, and proposes the Reciprocal World Model (RWM): the character inside the world should perceive the user continuously and decide when to take part. Duo v1 is the first modular implementation, with Intent handling cognition and participation and Realization handling coherent audiovisual behavior.

It is not a single model trained from scratch. The paper says Realization is initialized from MiniMax H3 weights. OrcaRouter, which read the repository configs, found that Intent is a Qwen3VLForConditionalGeneration model with a rank-8 LoRA and four added special tokens: <wait>, <blank>, <chunk_708> and <speak>. That inheritance explains the license problem below: the MiniMax H3 community license travels with the derivative weights.

Ant’s open-weight line has mostly been language models. In October we wrote about Ling-3.1-flash, released under MIT; Duo v1 applies the same “weights plus inference code” playbook to an audiovisual world model, but with very different terms.

The facts

The core of Duo v1 is a 708 ms chunk clock. Section 1 of the paper defines a chunk as 17 video frames, about 0.71 s at 24 fps; Section 6 gives the sampling window as 17/24 s, roughly 708 ms. Inside a chunk, Intent reads the user audio, video and ASR text available up to that boundary and emits either <speak> with text or <wait> with none. Its hidden states, not just the decoded words, are passed to Realization. Realization is a causal audiovisual diffusion transformer that generates audio and video latents jointly with flow matching; the attention masks prevent the current chunk from seeing future input.

Training runs in three stages: supervised fine-tuning of Intent’s participation decisions and text, causal joint audiovisual training for Realization, and dual-rollout distillation down to a four-step generator. The third stage matters most. Intent autoregressively generates the assistant turn (the language rollout), and Realization conditions on those intent features plus its own synthesized audiovisual history (the media rollout), so training conditions match inference. Without it, the paper says, training uses recorded assistant text and media while inference has to use the model’s own output, and the distributions drift apart.

The latency figures come from Section 6.3 and assume 768×576 generation:

StagePaper’s figureNote
Intent inference per chunk90–160 msVaries with input and context length
Realization joint generation200–450 ms768×576, varies with context length
Transfer and displayabout 40 msIncluded in the paper’s total
End to end per chunk330–650 msExcludes first-frame decode and chunk wait
Chunk-boundary waitup to about 708 msWhen an utterance ends mid-window
Interaction chunk17 frames @ 24 fps ≈ 708 msPaper Sections 1 and 6

Jiqizhixin’s report describes Realization as running at 720p; the paper’s latency table says 768×576. This article uses the paper’s number. The hardware gap is easier to miss: the 330–650 ms figure comes from an eight-GPU layout, with Intent on GPUs 0–3 in four-way tensor parallelism, the diffusion transformer across all eight, and the streaming VAE tiled four ways on GPUs 4–7. The README’s offline demo asks for two CUDA GPUs, one for Intent and the codecs and one for Realization. Both numbers are honest; they answer different questions.

Weights and licensing are the part to read before anything else. Hugging Face’s API lists Duo-1.0-causal-base and Duo-1.2-causal-base at about 153.46 GB each; Duo-1.0-distill is a 2.79 GB overlay and Duo-1.0-distill-quant is a 47.03 GB FP8 build. All four model cards say license: other and license_name: minimax-h3-community-license. The license defines an “Applicable Territory” as worldwide minus excluded territories, and the exclusions are the European Union, the United Kingdom, South Korea and the United States. Teams in those places are told to contact MiniMax for a separate license. The GitHub repository is Apache-2.0, but that covers the code only.

Evaluation is Section 7.1. Thirty real-time duplex cases were replayed side by side and judged on six binary criteria: content responsiveness, language adherence, speech intelligibility, visual identity stability, audiovisual synchronization, and audiovisual emotional consistency. Of 180 checks, 168 passed, for 93.3%.

CriterionPass rate
Language adherence30/30
Speech intelligibility30/30
Visual identity stability30/30
Audiovisual emotional consistency96.7%
Lip-sync alignment90.0%
Content responsiveness73.3% (22/30)
Overall168/180 (93.3%)

The paper explains the weak spot itself. One failure mode is the long tail of conversation — jokes and odd remarks that need a reading beyond the literal words. The other is intent that only becomes clear when speech and video are read together. Its example: a user indoors says they just unlocked a shared bike when it starts to rain. The model treated it as a request for advice and suggested waiting; given the indoor setting, a complaint was the better reading. The gap is not generation quality. It is reading the person.

The bottle catch belongs in the same careful category. It appears in the Action panel of Figure 1 and in Jiqizhixin’s opening anecdote, as a demo scene. It is not a controlled experiment: the evaluation protocol has no object-interaction metric, and the project page’s Moments section labels some examples as training-data clips. Treat it as a direction, not a measured capability.

What others say

Jiqizhixin’s November 17 report presents Duo v1 as a joint Ant and Xinchen AI result, leads with the bottle and a West Lake day-to-night transition, and describes the architecture and three training stages in line with the paper. It does not mention the license restriction or that the evaluation was self-scored.

OrcaRouter’s November 13 teardown reads the release differently. It counted 153.46 GB of weights across 80 files (Hugging Face’s API now counts 81), and noted that the release had “no launch post, no hosted endpoint, no pricing page and no independent benchmark.” It calls the license the part that “decides your week,” and points out that 93.3% is the authors marking their own exam: a binary rubric with written failure conditions is more falsifiable than a leaderboard Elo, but the judges are the authors. The only quantitative comparison in the paper is a data-synthesis ablation — over a 60-second sequence, the baseline loses 34.2 luma levels by the ninth six-second block while the proposed method stabilizes around 99 — with no head-to-head against Moshi, Qwen-Omni, Wan-Streamer or similar systems.

The paper’s own conclusion leaves room: “Duo v1 remains preliminary and currently supports basic audiovisual conversation,” with response timing, consistency over longer sessions and recovery from misunderstandings still needing broader evaluation. The team’s stated v2 direction is to condition Realization directly on user audio and video, which is the visual interaction the bottle demo implies.

Two discrepancies are worth flagging. On authorship, the paper’s first page carries only the Ant Group logo, with no affiliation line and no mention of Xinchen AI; the joint-authorship claim is Jiqizhixin’s, not the paper’s. On resolution, Jiqizhixin says 720p and the paper says 768×576. Neither changes the main story, but citations should keep them apart.

Our take

For people working on real-time interaction, the useful idea is the 708 ms clock and the separation of “when to speak” from “what to say.” Most audiovisual generators leave timing to the user or to a fixed turn structure. Duo makes it a per-chunk prediction and uses causal masks to keep the future out. That design travels to other models even if you never run Duo.

For teams building avatars, companions or game NPCs, read the license before the benchmark. The weights are not Apache-2.0: the US, EU, UK and South Korea sit outside the licensed territory, and teams there need a separate deal with MiniMax. Add the 153 GB download, the two-GPU minimum for the offline demo, the eight-GPU layout behind the low latency, and the absence of a hosted API or inference provider. Trying it means deploying it.

Three caveats. First, “world model” here means an audiovisual interaction model, not a general robot model: it outputs speech, expression and video, not robot actions or physical simulation state, and the paper has no robot-control experiments. Second, the evaluation is self-scored, and 73.3% content responsiveness says the hard part is still reading the human; until a third party reruns the rubric, 93.3% is an internal number. Third, demos and measurements are different things: the bottle and the West Lake transition are showcase scenes, while the quantitative result is 30 cases scored on six criteria.

Three things to watch: a third-party run of the same six criteria; whether the license adds the excluded territories; and whether v2’s visual interaction turns “see an object and respond” from a demo into a metric.

How to try it

The offline path is short. You need Linux, Python 3.10+, FFmpeg, two CUDA GPUs and two separate Python environments, because qwen-asr pins its own Transformers version:

git clone https://github.com/inclusionAI/Duo.git
cd Duo
python -m venv .venv-model && source .venv-model/bin/activate
pip install -e '.[realization]'
deactivate
python -m venv .venv-speech && source .venv-speech/bin/activate
pip install -e '.[speech]'
deactivate
hf download inclusionAI/Duo-1.2-causal-base --local-dir weights
cp configs/inference.example.json configs/inference.json
.venv-model/bin/python run_demo.py --config configs/inference.json --port 4173

Open http://127.0.0.1:4173/video/index.html, record or upload an MP4/WebM, and wait for the audiovisual reply. For streaming, the README points to the inference/vllm-omni-duo branch (recommended) and inference/sglang-omni-duo. The 330–650 ms figure belongs to that path, not the offline demo.