StreamEval: one benchmark for interactive streaming video models

Real-time interactive video models are multiplying, but most evaluations still judge a finished clip for picture quality. That cannot show whether a model holds a coherent scene while new instructions arrive for several minutes. StreamEval is a benchmark built for that case: five tasks, 300 test cases, sessions from 15 to 300 seconds, with every prompt, piece of code, input asset and scoring rule published. The repository is Apache-2.0 Python, created on October 15, 2026, with the paper at arXiv:2610.28402.

Core features

• Five tasks, 60 cases each: digital-human dialogue, avatar performance, world generation, instruction editing and reference editing. The README says the paper compares 19 existing benchmarks and that each covers only one or two of these tasks, while StreamEval puts them in one framework.
• Inputs arrive over time: Instead of one prompt up front, speech, text instructions, reference images and video frames are delivered along a timeline, so the test shows whether a system can take new requests while keeping earlier state.
• Four dimensions, ten metrics: task execution, state consistency, visual fidelity and audio-visual coordination, each scored from 0 to 100. A task only uses the metrics that apply to it and gets its own Overall; the five tasks are never merged into one total.
• A reproducible public process: Every case, the test code, each case’s input material, instruction timeline, expected result and scoring rule are open, so anyone can rerun the same evaluation.
• A leaderboard updated monthly: The README says the StreamEval Team maintains it and adds new models and new versions each month.

Benchmarks

The figures below come from a Chinese article’s summary of the paper and the public leaderboard, and we did not check the leaderboard itself item by item. The article says the paper evaluated 37 models in 47 model-task configurations, and reports one consistent pattern: picture quality and identity consistency score high, while following instructions over time scores much lower. Avatar performance tops out at 36.90 on Following, world generation at 25.93, and reference editing at 15.00 on Edit Success. The article’s category leaders include Vidu S2-Avatar for dialogue and performance (93.98 and 81.71), MemFlow for world generation (33.36), and Vidu S2-Editing for both editing tasks (45.43 and 39.02). Treat streameval.ai and the paper as the source of record for these numbers.

Typical use cases

• Model teams: Connect a model to the public test set and compare it under the same rules as commercial systems.
• Research and reproduction: The cases and scoring code are public, so the numbers in the paper, and the scoring rules themselves, can be checked.
• Product teams choosing a model: Read the leaderboard by task rather than looking for one composite score, because the five tasks use different metrics.

If you want a single real-time video model rather than a benchmark, see EdenLive.

Quick start

You need Python 3.10 or later, on Linux or macOS. The first run can be the CPU example, which generates its own media and needs no API key, GPU or dataset download:

git clone https://github.com/StreamEval/StreamEval.git streameval
cd streameval
python -m venv .venv && source .venv/bin/activate
python -m pip install -e ".[runtime,judge,dev]"
python -m streameval.benchmark.synthetic --out .smoke

After that, download the full data. The README puts it at about 3.1 GB, available from Hugging Face or ModelScope.

Summary

StreamEval suits teams and researchers who need to compare streaming video models, and anyone who wants to audit how such an evaluation is built. It does not suit a reader looking for one “best model” answer: each task has its own board, and the README is explicit that they are not combined.

A few things to know first. The scores cited here come from a Chinese article’s account of the paper and leaderboard. We did not open the leaderboard or read the paper’s body, so they are second-hand. The evaluation consumes model audio and video, and the scoring chain includes automated screening and judge configuration; the README says every model on a task uses the same judge setup, but the implementation needs a code read to confirm. The full dataset is about 3.1 GB, and a complete run needs the matching compute and model access.

The repository was created on October 15, 2026, and its last push was on October 22, 2026. It is not archived, had 125 stars as of October 23, 2026, and uses the Apache-2.0 license. We checked the README and repository metadata, and we did not install or run the benchmark.