airllm: run large models on small GPUs, one layer at a time

Running a 70B-class model usually means buying a card with a lot of VRAM. airllm takes another route. During inference, it keeps one layer on the GPU at a time and swaps in the next layer when the current one finishes. It is an Apache-2.0 Python library. The README says it runs 70B models on a single 4GB card without quantization, distillation or pruning. The repository has about 40,000 stars, and its latest release, v4.0.0, came out in September 2026.

Core features

• Layer-by-layer loading: VRAM use depends on the size of one layer, not the whole model. The README’s table lists about 4 GB for Llama 3.x 70B, about 8 GB for Llama 3.1 405B and about 12 GB for DeepSeek-V3 (671B). These are the maintainer’s figures, and we did not measure them.
• Optional block-wise compression: Passing compression='4bit' or '8bit' at initialization speeds inference by up to 3 times, according to the README. The README calls the accuracy loss almost negligible. The chart above shows this comparison, but it does not name the model or the hardware.
• Measured runs in the changelog: A July 2026 entry reports Kimi K3 (2.8T) running end to end on one RTX 6000 Ada in 3.72 GB of VRAM. An August 2026 entry reports Qwen3.8-Flash-Next at 5.95 GB on one RTX 4090.
• Streamed LoRA training in v4.0.0: Frozen weights stay on disk, and only the adapters stay on the GPU. The README says Qwen3.8-Flash-Next trains under 6 GB, and Qwen3.8-27B trains in about 2 GB at sequence length 512.
• Hugging Face model IDs: You pass a repository ID to AutoModel.from_pretrained, and the call looks much like loading an ordinary Transformers model.

Typical use cases

• A student or researcher with one consumer GPU who wants to try a large model: It runs slowly, but the full inference path works.
• A developer who can accept slow batch jobs: For example, an overnight summary of long documents, where latency per request does not matter.
• Someone running LoRA experiments on little VRAM: The v4.0.0 examples list VRAM figures for Flash-Next and the 27B model to compare against your own hardware.
• A reader who wants to understand layer streaming: The README table makes the link between VRAM and model size easy to see.

Quick start

pip install airllm

Load a model by its Hugging Face repository ID:

from airllm import AutoModel

model = AutoModel.from_pretrained("Qwen/Qwen3-32B")

The README also lists Qwen/Qwen3.8-27B at 3.33 GB and deepseek-ai/DeepSeek-V3 at about 12 GB. Swap in the repository ID to try them.

The v4.0.0 training example needs newer packages. Run pip install -U airllm, then install transformers from its main branch with pip install git+https://github.com/huggingface/transformers.git. Qwen3.8-Flash-Next depends on qwen4_exp, which is only on that branch.

Gated models on Hugging Face need an access request on their page first. Without one, loading fails with a 401 Client Error, as the README’s FAQ explains.

Summary

airllm suits people who want to run large models on limited VRAM and can accept slow output. It does not suit low-latency online serving. The README gives no tokens-per-second figure, and it says the bottleneck with compression enabled is mainly disk loading.

A few things to know first. Most of the VRAM numbers in the README are the maintainer’s measurements or estimates, and we did not run the library on any of those cards. The claim that compression costs almost no accuracy comes without test results, and we did not check it. The README’s license badge links to another repository, so we took the license from this repository’s LICENSE file, which is Apache-2.0.

The repository was created on June 12, 2023, and its last push was on October 21, 2026. It is not archived, and as of October 21, 2026, it had 39,988 stars. GitHub lists the main language as Jupyter Notebook, but the library itself is a Python package.