vLLM and Ant Group launch Humming quantized kernels; project reports 14–43% more throughput on H20

Humming is a quantized inference kernel library from the vLLM community and Ant Group, licensed under Apache-2.0. It compiles GEMM kernels just in time for the target GPU and matrix shape. On October 21, the vLLM blog reported an end-to-end test on eight H20 GPUs: swapping only the MoE backend, from Marlin W4A16 to Humming W4A8, raised DeepSeek-V4.1-Flash output throughput by 14% to 43%. Those are the project’s own numbers, and we found no independent rerun.

Key points

• License and package: The repository is Apache-2.0. The PyPI package humming-kernels reached version 0.1.18 on October 21, and its x86_64 wheel is 517,199 bytes. Since vLLM 0.29.0, CUDA installs of vLLM list it as a dependency.
• Test setup: Eight H20 GPUs, tensor parallel size 8, and DeepSeek-V4.1-Flash with its MXFP4 weights left unchanged. The post gives no request lengths, dataset or benchmark tool.
• How to enable: Pass --linear-backend humming and --moe-backend humming to vllm serve. Both options default to auto in vLLM’s source, and the post’s examples set humming explicitly.

Background

Humming’s code was public long before this post. The vllm-project/humming repository was created on February 11, 2026, and the older inclusionAI/humming address redirects to it. humming-kernels has 19 versions on PyPI, from 0.1.0 (uploaded May 13) to 0.1.18. vLLM 0.29.0, released September 9, added Humming to its CUDA dependencies, and its release notes list several Humming changes, including an MoE path for MXFP4 weights with block-FP8 activations (#51332). The post is titled “Announcing Humming” and says the two groups “are open-sourcing” it, but the repository and package were already available. Treat this as a formal introduction, not a first release.

The post’s serving example uses DeepSeek-V4.1-Flash-1017, whose model card we covered on October 17. DeepSeek’s DeepInfer framework, which we covered on October 20, has a vLLM link too: its README says it was “originally built primarily on vLLM” (our DeepInfer story). The vLLM team’s original developers also founded Inferact, whose TPU inference kernels we covered in September (Inferact story). The post adds that Humming already fully supports T-Head’s Zhenwu 810E and plans to extend to the Zhenwu M890.

The facts

The post’s end-to-end numbers for Humming are in this table. The setup is eight H20 GPUs at tensor parallel size 8, with only the MoE backend changed. The change columns show Humming relative to Marlin, taken from the post’s own table. In that table, input throughput moves by the same percentages as output throughput.

Concurrency	Marlin W4A16 output (tokens/s)	Humming W4A8 output (tokens/s)	Output change	TTFT change	TPOT change
1	88.30	100.77	+14.1%	−29.3%	−8.5%
2	149.33	176.38	+18.1%	−30.0%	−10.9%
4	231.99	286.12	+23.3%	−31.7%	−12.9%
8	287.04	383.04	+33.4%	−32.3%	−22.2%
16	348.96	475.89	+36.4%	−25.1%	−27.1%
32	398.83	560.49	+40.5%	−30.5%	−28.5%
64	442.37	633.94	+43.3%	−29.0%	−30.5%

Three limits matter. First, the two sides do not use the same activation precision: Marlin runs BF16 activations and Humming runs tokenwise FP8 activations, so this is a precision change, not a like-for-like swap. Second, the post gives no accuracy or quality figures for this test, so we cannot say whether output quality moved. Third, the post does not state request lengths, the dataset or the load generator, so the numbers cannot be reproduced from the post alone. The body text says TPOT falls by about 31% at concurrency 64, but the table shows 30.5%. We use the table.

The post also reports kernel microbenchmarks run by the vLLM community. At small batch sizes, dense W4A16 runs roughly 3 to 4 times faster than cuBLAS on several devices. On H200 at large batch sizes it reaches about twice Marlin’s speed. On H20 and H200, MXFP4 W4A16 MoE reaches roughly twice Marlin’s speed at medium to large batch sizes. The post also highlights compile time: for one W4A16 kernel on H200 with CUDA 13.4, compilation fell from 23.35 seconds with CUTLASS and NVCC to 1.82 seconds with Humming and NVCC, with caches disabled. These are single-kernel or compile-stage results, not whole-server throughput.

We read the package facts directly from PyPI and the vLLM repository:

Item	Value	Source
Latest humming-kernels	0.1.18, uploaded 2026-10-21	PyPI metadata
x86_64 wheel	517,199 bytes	PyPI file list
Base dependencies	torch≥2.7, triton, numpy, safetensors, jinja2, cuda-bindings, tqdm, tabulate	PyPI requires_dist
Pin in vLLM 0.29.0	humming-kernels[cu13]==0.1.12	vLLM requirements/cuda.txt

What others say

Pandaily, an English-language outlet, reported the same figures on the same day and wrote that they “have not been independently verified.” It also repeated the README’s claim that Humming needs only PyTorch and NVCC and that the pip package is about 100 KB. PyPI metadata does not support that. Version 0.1.18 lists triton, numpy, cuda-bindings and other dependencies, and its x86_64 wheel is 517 KB. We follow PyPI.

Sina Finance reported on January 29 that T-Head had put the Zhenwu 810E on its website, listing 96 GB of HBM2e memory and 700 GB/s of chip-to-chip bandwidth. The same report, citing unnamed industry sources, said the chip’s overall performance was comparable to NVIDIA’s H20. It gave no test conditions, and we found no public test behind that comparison.

An audit document in a personal GitHub repository, dated August 2026, targets the SM87 Jetson Orin. It examined the vLLM and Humming main branches from August 7 and 8, with a local comparison environment of vLLM 0.26.0 and Humming 0.1.10, and it explicitly reports no performance result. One of its observations concerns automatic selection: in the selector it inspected, the A16 automatic route still picks Marlin, so calling Humming the automatic production path for Qwen3.5 on SM87 is not accurate. That is a single-device observation on an older version, and we did not recheck it on v0.29.0.

Three Humming-related issues in the vLLM repository were still open when we checked on October 21. Issue #60162 reports that on Ampere, Humming’s FP8 linear path crashes on per-tensor or per-channel FP8 weights, so FP8 models will not start with VLLM_BATCH_INVARIANT=1. Issue #60700 says the W4A8 MoE assertion requires hidden and intermediate sizes to be multiples of 256 for every backend, so Humming also rejects shapes that CUTLASS rejects, such as an intermediate size of 640. Issue #62136 reports that quantization="mxfp8" with moe_backend="humming" crashes during weight conversion with a KeyError: 'weight_scale'.

Our take

Humming’s 14% to 43% range spans concurrency 1 to 64, and the gain grows with concurrency. At concurrency 1 it is 14.1%. A team that serves at low concurrency should not expect the top of the range.

Throughput is not cost. The post gives no H20 price, power draw or cost per token, so the numbers do not translate into savings.

Check versions before you pin. vLLM 0.29.0 pins humming-kernels 0.1.12, while PyPI has 0.1.18. The README says only PyTorch and NVCC are needed, but the 0.1.18 metadata lists more dependencies. Confirm what pip actually installs.

Precision is the biggest open question. The end-to-end comparison moves activations from BF16 to FP8, and the post reports no accuracy figures. Measure output quality on your own evals before you switch.

The post calls the Zhenwu 810E CUDA-compatible. The January report describes a self-developed software stack and says nothing about a compatibility layer, so we cannot confirm that claim.

Three things to watch: whether issues #60700 and #62136 close; whether the autotune feature the post plans ships (it says Humming will benchmark candidate configurations on the target device, and we did not confirm a release); and whether anyone publishes an independent test on the Zhenwu 810E.

Teams running quantized MoE models on vLLM, especially on H20 or other Hopper and Blackwell cards, are the readers who should reproduce this test, using their own request lengths and concurrency. Teams on the Zhenwu 810E have only the post’s word so far.

How to try it

If you use a CUDA build of vLLM 0.29.0 or newer, you do not need to install Humming separately. The post’s serving example runs with eight-way tensor parallelism:

vllm serve deepseek-ai/DeepSeek-V4.1-Flash-1017 \
  --tensor-parallel-size 8 \
  --linear-backend humming --moe-backend humming

To try the kernel library on its own, run pip install "humming-kernels[cu13]" for CUDA 13 or pip install "humming-kernels[cu12]" for CUDA 12. The README includes a Python example that uses HummingLayer. We did not run these commands on a GPU machine, and we did not reproduce the numbers.