Audio#Voice cloning#Video generation

LiveTalking: a streaming digital human that talks back live

LiveTalking by lipku is a real-time streaming digital human: synced audio-video, interruptible speech, multi-session, low-latency WebRTC, Apache-2.0.

Project facts

GitHub Ecosystem
Repositorygithub.com/lipku/LiveTalking
License
Apache-2.0
Language
Python
Stars
9,728
Data checked
2026-10-06

Snapshot figures reflect the check date and may change over time.

For always-on live streams and avatar customer service, realism matters less than latency: when a viewer says something, the avatar has to answer now. LiveTalking by lipku is a real-time interactive streaming digital-human engine with synchronized audio-video dialogue; the README states it already sees wide commercial use. It has been updated continuously since December 2023 — the latest commit, September 13, 2026, fixed a real leak: disconnected sessions previously dropped only a dict reference while render, inference, and TTS threads kept spinning, about 3 CPU cores per leaked session.

Core features

  • Swappable models: one framework drives ernerf, musetalk, wav2lip, and Ultralight-Digital-Human — pick per quality-versus-compute budget.
  • Interruptible speech: users can cut in mid-sentence; flush_talk clears the pending speech queue instead of making you wait.
  • Three output paths: WebRTC for low-latency interaction, RTMP push into a live room (relayable to platforms like Bilibili or YouTube), or a virtual camera feeding OBS.
  • Motion choreography: custom videos play during idle stretches so the stream never freezes.
  • Multi-session and custom avatars: several sessions run in parallel, each with its own sessionid, with your own footage as the avatar.
  • Pluggable LLM and TTS: the LLM engine speaks to OpenAI-compatible backends including DashScope and OrcaRouter; the TTS side supports EdgeTTS, GPT-SoVITS, CosyVoice, and voice cloning; a decentralized plugin registry (registry.py) lets developers extend the TTS, avatar, and output modules themselves.

Typical use cases

  • 24/7 unmanned livestreams: an LLM drafts the pitch, the avatar speaks it live, choreography fills the gaps.
  • Knowledge-base customer service: users ask by voice, the avatar answers in real time, mistakes get interrupted and re-answered.
  • E-learning and lobby displays: batch-generate lessons through the API, or drive an interactive presenter via the /human endpoint.

Benchmarks

The README’s own real-time inference numbers, per single session:

ModelGPUFPS
wav2lip256RTX 306060
wav2lip256RTX 3080Ti120
musetalkRTX 3080Ti42
musetalkRTX 409072

These come from the project’s own “Performance” section in the README (no test date given). The backend logs inferfps (GPU inference rate) and finalfps (final streaming rate), and both need to clear 25 to count as real time; concurrency while idle is CPU-bound, concurrency while everyone is talking is GPU-bound — the two numbers worth checking before picking a card.

Quick start

git clone https://github.com/lipku/LiveTalking.git
conda create -n livetalking python=3.12
conda activate livetalking
pip install torch==2.9.1 torchvision==0.24.1 torchaudio==2.9.1 --index-url https://download.pytorch.org/whl/cu128
pip install -r requirements.txt
python app.py

Per the README, this was tested on Ubuntu 22.04 with Python 3.12, PyTorch 2.9.1, and CUDA 12.8; model weights download separately. Once the service is running, /human accepts text in either echo or LLM-chat mode, and /humanaudio plays an audio file directly, with each connection assigned its own sessionid for multi-user concurrency; a separate Admin API handles global configuration, session monitoring, and forced stops, which is useful for wiring into an existing ops backend.

Summary

LiveTalking fits live, interactive scenarios; if you only need finished clips without latency constraints, an offline model like LongCat-Video-Avatar 1.5 renders better quality. Apache-2.0, written in Python, about 9.7k stars as of 2026-10-06. Four caveats: the environment is heavy and CUDA versions must line up; lip quality depends on the model you pick — wav2lip is fast but coarse, musetalk finer but hungrier; the README explicitly requires that any video made with it and posted to platforms like Bilibili, Shipinhao, or Douyin carry LiveTalking’s watermark and attribution, so check that before a commercial release; and a paid commercial edition exists on top of the open-source three-model stack (adding wav2lipls plus adaptive motion and real-time voice interaction), while the open-source path itself leans on GitHub issues for support, with no SLA.