Compute#AI chips#Model inference

Twelve used V100s now hold every DeepSeek V4.1 Flash expert at 4-bit

A lawyer runs DeepSeek-V4.1-Flash on 12 used V100s for about $9,450: all 15,360 experts stay in VRAM, 153–184 tok/s on one code stream.

PCIe switch cards and braided cables inside an open frame server, with red and blue status lights on

On November 19, 2026, Nick Mermiges published the fourth release of a twelve-card build in his GitHub repository: twelve Tesla V100-SXM2 32GB cards from 2018, split into three pipeline stages, holding all 15,360 routed experts of DeepSeek-V4.1-Flash-1017 in NVIDIA’s NVFP4 format in card memory, at 153–184 tokens per second on a single code request. The twelve cards cost about $9,450.

Key points

  • The memory arithmetic closes. Twelve cards at 32,510 MiB each come to roughly 381 GiB; the NVFP4 experts need 278.9 GiB, or 73% of it, leaving about 100 GiB for a 524,288-token context, KV cache and activations. Eight cards gave only 254 GiB, which is why the earlier build could not fit them.
  • All the speed numbers are single-request. The 153–184 tok/s band covers one code stream at six context lengths from 1K to 256K; three streams together sum to 344–443 tok/s, and the author writes plainly that this “is not a batched-decode engine.”
  • Nothing is compressed further. All 15,360 experts sit in card memory across 12 of 12 workers — 5,376 / 5,376 / 4,608 by stage, zero in host RAM. The eight-card build had to hold every expert at 2 bits and swap the 5,949 most-used ones (38.7%) back to NVFP4 until the cards filled up.

Background

DeepSeek-V4.1-Flash-1017 is an open-weight update DeepSeek shipped on October 17 under the MIT license. Its published config.json lists 40 layers, 384 routed experts per layer, six experts active per token, a 1,048,576-token position limit, and fp4 as the expert dtype. Forty times 384 is 15,360, the number the build keeps citing, and we covered the model’s benchmark changes when it landed.

Those experts are where the parameters live. Each is three 5,120 × 2,304 matrices, about 35.4 million parameters, and 15,360 of them come to roughly 543.6 billion (our arithmetic). Left in host memory, every layer’s routing decision would drag them across PCIe and decode would collapse, which is why local deployment of this class of mixture-of-experts model is a fight over VRAM capacity, not over compute.

The builder is not new to this. Since late October he has published three eight-card variants — Volta sPLIt, Volta Max uPLIft and Volta Full NVFP4. The upstream three-stage patch in 1Cat-vLLM (#1031) ran this model on twelve cards at about 3 tok/s on an eager engine, which is the baseline the current 150 tok/s figure grew from. The card itself is near the end of the road: NVIDIA’s R580 branch is the last driver line that supports Volta.

The facts

The speed table comes from the repository’s docs/measurements.md, test window b4a and the 14:09Z bench on November 19, 2026. Conditions: greedy decoding, thinking off, 1,024 tokens out, four slots, one request at a time, each cell the mean of two repeats.

ContextCode (tok/s)Legal (tok/s)Prose (tok/s)First token, cold (s)Prefill (tok/s)
1K183.8101.586.10.651,567
8K175.592.989.62.513,267
32K175.892.594.77.324,478
64K157.493.483.313.954,697
128K157.493.185.628.074,670
256K153.492.484.558.034,518

The 58-second first token at 256K is the number to remember: the previous eight-card build needed 108.4 seconds for the same wait. Ask again about a document the server has already read and the first token arrives in 0.43 s at 128K and 0.64 s at 256K — prefix-cache hits and cold reads differ by two orders of magnitude.

ItemValue
Cards12 × Tesla V100-SXM2 32GB (2018), 32,510 MiB each
Total VRAMabout 381 GiB (arithmetic)
NVFP4 experts278.9 GiB, all 15,360, none in host memory
Context524,288 tokens per request, four slots
TopologyPP3 × TP4, layers 0–13 / 14–27 / 28–39, NVLink inside a board, PCIe between
Power cap175 W per card, 2,100 W for the twelve (arithmetic)
Clock cap1,452 MHz (cards run at 1,447)
Purchase priceabout $9,450, or roughly $790 per card (arithmetic)

The price needs a caveat. The $9,450 covers twelve cards on three four-card boards, not one card and not a whole machine: the earlier eight-card build put the boards at about $2,400 each when they were bought, when a card cost around $400. By November 9, Tom’s Hardware found used V100 SXM2 32GB cards listed at about $900 on eBay and still climbing. The host, RAM and power supplies already existed — a Threadripper Pro board, eight 32 GB DDR4 RDIMMs, two PSUs totalling 2,800 W — so the $9,450 reads as cards and carrier boards.

Power and noise deserve the same care. The builder raised each card’s cap from 150 W to 175 W and lowered its clock ceiling from 1,522 MHz to 1,452 MHz, and says the trade bought a noticeable gain. Twelve times 175 W comes to 2,100 W (our arithmetic), which matches the ceiling IT之家 reported — the number covers the cards, not the host. The same report calls the machine loud enough to rule out an ordinary office. Pushing a card to 200 W for a day ended with the host tripping once under four long chats.

Why Volta can run FP4

The V100 has no FP4 or FP8 tensor cores, only FP16 mma.sync.m8n8k4. The repository states the cost flatly: every 4-bit format has to be unpacked in software into FP16 operands, so “the low-bit experts pay by reading fewer bytes, not by faster math.” Tom’s Hardware drew the obvious conclusion — a 2018 card running a Blackwell-era format is a story about 900 GB/s of memory bandwidth, not about compute.

The profile of a decode step agrees. Using the engine’s own timers and the profiler, the builder attributes 29% of a round to experts, 21% to dense multiplies, 10% to the drafter and 14% to work outside any kernel, with attention and its index taking most of the rest. A decode round takes 26.25 ms on a short chat, 28.1 ms at 128,000 tokens and 31.0 ms at 249,000, and nearly all of the growth is the sparse-attention index.

Board topology is the other engineering detail. Each of the three boards holds four cards, shares NVLink internally and reaches the others only over PCIe: two boards sit behind one PEX88096 switch, so their hand-off stays inside it, while the third sits behind a separate PLX 8749 and its hand-off crosses additional host bridges. Hence the 0–13 / 14–27 / 28–39 layer split — each stage’s work stays on its own board.

What others say

Tom’s Hardware’s November 20 report is the fullest outside account, and it fills in the memory history: at eight cards every expert sat at 2 bits with 5,949 swapped to NVFP4 until the cards filled; at twelve, “none compressed further.” The builder told the outlet that the expert values “equal the ones in DeepSeek’s own release,” while other parts of the model stay in compact forms.

IT之家’s Chinese report on November 21 carries the numbers accurately — 58.0 s to first token at 256K, 4,518 tok/s prefill, 28.1 s at 128K all match the repository’s tables — but one sentence mistranslates the expert count: it says each expert was compressed to about 15,360 parameters. 15,360 is the number of routed experts (384 × 40 layers); by our arithmetic from the published config, one expert holds about 35.4 million parameters.

The builder’s own caveats are stricter than most of the coverage. The repository says the figures show how closely the build tracks an all-native model and “do not by themselves say that its answers are better for a given kind of work”; answer quality in use “is not established.” He also discloses that the Engram table is a 4-bit version (103.0 GiB against the official 188.8 GiB) and that no equivalence check exists for it.

The claim that the server beats “any published 4x DGX Spark stack” at half the price of three Sparks comes from his Reddit post and has not been reproduced by anyone else. Against the DGX Spark prices we recorded in October — $4,999 for the 64GB unit, $6,950 for the 128GB — three units cost $14,997 or $20,850, so “half” only roughly holds for the larger configuration.

Our take

The point of this build is not that old cards still work. It is that someone published the bill of materials and the measurement method for a workload that has to keep a large mixture-of-experts model resident: 2018 hardware, about $9,450, and better than 150 tok/s for one user reading long documents. For a law firm, a clinic or a small team that cannot send files out, that combination is more realistic than three DGX Sparks — if they are willing to inherit the software work.

Almost all of the difficulty is software. The engine is a community SM70 fork of vLLM with the builder’s own three-stage patches, eleven environment variables, a 31,998 MiB cap per card, PCIe ACS redirect disabled, and a 164-to-197-second load. The kits, measurement scripts and judges are public; a one-command install is not. Following the documentation is still a porting project.

The costs are real too. It serves one request at a time; three concurrent streams sum to 344–443 tok/s, and cold reads interfere — three 128K code requests released together got their first tokens after 77.4, 90.5 and 90.5 seconds where one alone takes 28.1. At 2,100 W of GPU power plus the noise, the machine belongs in a server room, not an office. R580 is the last driver branch, so the cards are on borrowed time. And the 4-bit Engram table means “experts match the official release” is true of the experts and nothing else.

So 184 tok/s is not a general performance figure. It is one cell: one code request, 1K of context, greedy decoding; repeats of the same text stay within 2% and different texts spread by about ±9%. This is one person, one machine and a published method — not a benchmark, and not yet reproduced.

What to watch

Batched decoding is first on the builder’s list — “the one thing the Sparks have that I want.” Then the conversation store, so a matter’s documents survive a restart, and a quality window for the Engram reader. There is also a smaller debt: the restart tool after a power trip is still pinned to an older kit. Until those land, twelve V100s are at “it runs,” not yet at “run it every day.”