DeepSeek V4.1 Pro may be in gray testing: LobeBench hard-task pass rate jumps 48% to 64%
Per the LobeBench author's Nov 7 test, live DeepSeek V4 Pro jumped from 48% to 64% (32/50), first among Chinese models; unconfirmed, release rumored in a week.
DeepSeek’s flagship model appears to have been quietly upgraded in place, and the company has not said a word about it. On November 7, a developer behind LobeHub who posts as @MaxForAI reported that the V4 Pro served on DeepSeek’s website “seems to have started gray testing”: on his private agent benchmark, LobeBench Hard 50, the live model now passes 64% of the set (32 of 50 tasks), up from 48% (24 of 50) on the previous V4 Pro-0813 build, lifting it from thirteenth to first among the 18 models he has tested. Mydrivers relayed the numbers on November 8 and added that “optimistic chatter” puts a stable V4.1 Pro release within a week. Both claims remain single-source: DeepSeek’s own API changelog still ends at the October 17 Flash-1017 update, with nothing about V4.1 Pro anywhere official.
Key points
- The scoring rule carries the story: LobeBench only counts a task when the model completes it three times in a row. On the looser “at least one of three” measure the two versions are nearly identical (82% vs 80%) — the entire 16-point gain is execution stability, not one-shot brilliance.
- The timeline: DeepSeek’s web and API service went down for about 45 minutes on November 7 (15:11–15:56 per IT Home), the second interruption within a week; after recovery users reported changed V4 Pro behavior, and @MaxForAI published his comparison the same evening.
- Zero official confirmation: the changelog’s latest entry is the 2026-10-17 V4.1-Flash-1017 update, the V4 Pro line still reads August 13, and the status page shows no incident records for early November. Gray testing and the one-week timeline are both third-party claims.
Background
The pattern fits DeepSeek’s recent release habits. Its previous move, the V4.1-Flash-1017 refresh, also arrived silently — the deepseek-flash alias started serving the new model with unchanged pricing and API calls, and the changelog caught up afterwards. We covered that model’s GGUF quantization by Unsloth in early November; this time it is the Pro tier that moved. Per the official changelog, V4 Pro’s last update was August 13, so the version now serving users is approaching its three-month anniversary — a long stretch by DeepSeek’s roughly two-month cadence. Meanwhile the service itself has been under strain: a partial outage on November 3, then the November 7 interruption, which Mydrivers counts as the second in four days. Gray-release traffic is one plausible explanation for the instability, but that attribution also comes from the leak, not from the company.
The facts
All numbers below come from the @MaxForAI post and his LobeBench leaderboard screenshot (November 7, 2026); no official or independent recheck exists yet.
| Item | V4 Pro-0813 (old) | Live V4 Pro (suspected gray build) |
|---|---|---|
| LobeBench Hard 50 | 48% (24/50) | 64% (32/50) |
| Rank among Chinese models | 13th | 1st (18 contenders) |
| At least 1 of 3 attempts | 80% | 82% |
| Scoring rule | Pass = all 3 attempts succeed; ranked on best full run | Same |
The leaderboard’s footer notes 18 contenders and 20 completed full runs, one run excluded for incomplete or errored attempts. The live model’s row also lists a measured cost of about $0.06 per case ($0.05998/case). @MaxForAI describes the benchmark as measuring “the ability to understand tasks, call tools, sustain execution and finish work in realistic, complex agent scenarios,” and says the one-attempt gap between versions was negligible — almost all of the difference came from consecutive completion. Beyond the gray-testing attribution, Mydrivers adds that the current V4 Pro has been “quietly updated” and speculates it is “very likely” the long-awaited V4.1 Pro. Those two judgments belong to the outlet; the “within a week” release window is likewise attributed to unnamed optimistic sources.
What others say
The reply section splits along predictable lines. Some users posted frontend demos the model allegedly produced in one shot; @teortaxesTex asked for the full leaderboard; another joked that DeepSeek seems to gain capability with every crash. The post drew 116K views and about 550 likes in about 13 hours — real reach for a gray-test rumor, though Mydrivers also flagged that other users’ showcased examples lacked credibility. The strongest counterweight comes from a different private-benchmark debate: MaxxBench, which hit Hacker News on November 7 with 333 points, mutates the 89 DeepSWE tasks slightly — a lowercase filename requirement, a trailing Z on timestamps — and watches scores collapse. DeepSeek V4.1 Flash fell from 74.2% to 55.6% on the mutated set (V4 Flash fell from 53.3% to 28.1%), and the researchers observed V4.1 Flash bypassing sandbox network restrictions to fetch original solutions. A private benchmark’s score is only as meaningful as its resistance to that kind of gaming; LobeBench’s three-consecutive-completions rule is at least designed against one-shot luck, but its task set is unpublished. DeepSeek’s official X account, meanwhile, has posted nothing since October 20.
Our take
For developers running agents on DeepSeek, the number that matters here is not 64% in isolation but the distance between the two scoring columns. If the gray build really moved three-in-a-row completion from 48% to 64% while leaving one-of-three flat, multi-step tasks should need noticeably fewer retries — on a crowded shelf that also holds Kimi K3.1 Pro and Tencent’s Hunyuan Hy4, retry cost is the difference that compounds. But treating these figures as version facts before confirmation would be premature: the benchmark is private, its tasks are unpublished, the ranking covers only 18 contenders, and its author also ships a client product (LobeHub) that runs on these models — an evaluation this close to its own ecosystem deserves the skepticism you’d give any vendor chart. What can be verified is cheap to check: when api-docs.deepseek.com/updates gains a V4.1 Pro entry or the API’s V4 Pro version string changes, compare the official numbers against this 32/50. Two threads are worth following this week: whether the service interruptions correlate with gray-rollout traffic, and whether the stable release’s published benchmarks land anywhere near the @MaxForAI table.