EdenLive: real-time rendering of gray-box video into detailed frames

If you are building a game prototype or an interactive world, the rules and state can work long before the visuals do. EdenLive is a real-time generative renderer from VAST, published on GitHub, and its README draws the split clearly: the program runs the rules and keeps the world state, while EdenLive renders the reference video into more detailed frames, guided by a target first frame and a text prompt. The paper’s abstract says it generates 720P video at more than 24 FPS on a single GPU, the repository was created on October 12, 2026 (UTC), and the Tencent Research Institute’s AI digest mentioned it on October 21.

Core features

Five parts make up EdenLive, listed in the order a reader is most likely to need them.

• Gray-box input, learned appearance: The input is a proxy or reference video, a target first frame and a prompt. The model card says references can come from 2D or 3D scenes, existing engines or recordings. The sample proxy video is 1280×704, and the prompt can be one static text or a set of timed prompts.
• Two models, two modes: The 5B and 14B weights each come in a bidirectional mode, which denoises all frames together, and a few-step mode, which samples chunk by chunk with four steps per chunk. The paper uses the few-step mode for real-time serving.
• WebRTC streaming: The real-time service takes game frames as input and streams video over WebRTC. You set up the TURN relay yourself. LiveTalking also streams real-time video over WebRTC, but its digital human is driven by text or audio, while EdenLive’s frames are driven by a reference video, a first frame and a prompt.
• EdenGolem, a playable game from one text prompt: The README says EdenGolem turns a text prompt into a “coarse playable game”, and the paper calls the output a playable 3D gray-box game. The workbench puts a game library, a chat that builds games and a play window in one interface.
• An accelerated path on the RTX 5090: According to the README, LightX2V’s low-bit quantization, operator fusion, KV-cache work and CUDA Graphs bring the 5B model to about 24.7 FPS inference throughput at 1280×704 on one RTX 5090. The README gives no test date.

This is the sample gray-box first frame, the proxy image the README describes. The ground and the prop are low-detail geometry, and the model renders its appearance on top of that layout.

The image above is cover.png, which the README describes as a photoreal render of the gray-box frame.

Typical use cases

EdenLive’s uses below each depend on checking hardware and licenses first.

• Solo developers prototyping a game: Keep the rules and state in an engine or script, and let EdenLive handle the look. The demo video at the top of the README shows a small underwater wreck game, with EDENLIVE and REAL-TIME RENDER in the corner.
• Researchers restyling recorded footage: Offline inference reads a recording manifest, and the paper mentions offline re-rendering and stylization of recorded videos. The README’s 14B examples use two GPUs (--nproc_per_node=2).
• Embodied AI teams working on data augmentation: The demo video includes a robot kitchen example labeled “Embodied Data Augmentation”. It is the project’s own example, and the README gives no benchmark for that task.

Quick start

EdenLive’s quick start follows the README’s install and inference steps. It skips the real-time service dependencies (step 2) and runs offline, single-GPU 5B few-step inference.

git clone https://github.com/VAST-AI-Research/EdenLive.git EdenLive
cd EdenLive
conda create -n edenlive python=3.10 -y
conda activate edenlive
pip install torch==2.13.0 torchvision==0.28.0 --index-url https://download.pytorch.org/whl/cu130
pip install av==17.1.0 decord==0.6.0 pandas==2.3.3 pyarrow==25.0.1 imageio==2.37.4 imageio-ffmpeg==0.6.0
pip install --no-deps -e .
hf download VAST-AI/EdenLive --local-dir ./models/EdenLive
python -m edenlive.scripts.infer \
  --config edenlive/configs/wan22_5b_720p_few_step.yaml

The README verifies on Linux x86-64 with 80 GB NVIDIA GPUs, a CUDA 13 driver and Python 3.10. The weights total about 210 GB, so check disk space first. The README lists the SM90, BF16 and FA4 kernel requirements (flash-attn-4==4.0.0b31 and nvidia-cutlass-dsl==4.7.1) only for the default 5B bidirectional profile. This few-step config reads the two samples listed in examples/samples/manifest.csv, writes output to inference/wan22_5b_720p_few_step, and lists the native attention backend at runtime.

Benchmarks

EdenLive’s speed numbers come from two places: the tech report’s H100 runs and the README’s LightX2V result on an RTX 5090.

Setting	Hardware	Inference throughput	Source
5B, 832×480	1 × H100	73.52 FPS	Tech report, Table 2
5B, 1280×704	1 × H100	29.80 FPS	Tech report, Table 2
14B, 832×480	8 × H100	26.76 FPS	Tech report, Table 2
5B, 1280×704, LightX2V	1 × RTX 5090	about 24.7 FPS	README

The tech report is dated October 2026. Its timings use roughly 30-second offline proxy inputs and run from model-side condition processing until decoded frames reach CPU memory. The README’s RTX 5090 figure has no test date or software version. All four throughput figures are above 24 FPS. Latency needs its own line: Table 3 of the report gives a first-frame latency of 1,161.79 ms for the 5B model at 1280×704 on one H100, with peak allocated memory of 36.78 GiB per GPU.

Summary

EdenLive suits a team with an 80 GB-class NVIDIA GPU that wants to study interactive worlds or game visuals. It is a poor fit for anyone without an SM90-class GPU who wants the default 5B bidirectional profile. If you want the workbench in a commercial product, its FLUX-based first-frame model rules that out.

Read the license by component. The code is Apache-2.0. The model card lists the DiT, VAE, text encoder and decoder weights as Apache 2.0, CLIP as MIT, the vision encoder under the DINOv3 License, and the FLUX.2 [klein] 9B first-frame model under a non-commercial license. The Hugging Face license field reads other (edenlive-mixed). The vision encoder carries control signals into the diffusion transformer, and the config enables it, so “5B and 14B weights are Apache 2.0” is only part of the story. The workbench’s game generation also runs the claude CLI with --dangerously-skip-permissions, which the README states openly.

We read the README, the model card, the tech report and the config file, and we did not run inference.