Models#Benchmark#Model training
Z.ai releases GLM-Legal, an agentic legal model built on GLM-5.3-Flash
Z.ai shipped GLM-Legal, a legal agent model built on GLM-5.3-Flash: 19.2% all-pass on Harvey LAB, API by application, 7th of 79 on vals.ai.
Zhipu’s Z.ai released GLM-Legal on October 12, an agentic model built for legal work: legal-corpus mid-training plus reinforcement-learning post-training on the GLM-5.3-Flash base, scoring a 19.2% all-pass rate on Harvey’s Legal Agent Benchmark (LAB) — 12.5 percentage points above that base, per the official blog. There are no downloadable weights, API access opens by application, and no pricing has been published.
Key points
- 19.2%, graded all-or-nothing: the official blog (October 12) reports 19.2% on Harvey LAB, +12.5 points over GLM-5.3-Flash. LAB draws on real law-firm workflows across 24 practice areas and only credits a task when every rubric item passes.
- Seventh on the independent board: vals.ai’s Harvey LAB leaderboard (updated October 12, 79 models) puts GLM Legal at number 7 with a 19.17% task pass rate at $0.15 per test. Muse Spark 1.2 leads at 25.42%, and every top-five slot belongs to the Muse Spark or Wonder Full families.
- Application-gated API: beta access runs through z.ai/consultation/legal; Z.ai says the model is coming soon to its desktop agent platform AutoClaw. On open weights, the blog says nothing at all.
Background
Harvey LAB belongs to Harvey, the legal-tech company, and Harvey has shaped these models before. In June it partnered with Applied Compute to post-train GLM-5.1 with reinforcement learning — per Harvey’s own blog post dated June 22, 2026, post-training lifted the rubric pass rate to 0.913, which the company said beat GPT-5.5 xhigh and Opus 4.8 Max. Watch the units: that is a rubric pass rate from Harvey’s own runs, not the same yardstick as this week’s 19.2% all-pass figure, and the two don’t convert. On Zhipu’s side, the GLM line has been open-weight until now — GLM-5.3 was open enough to get named in Anthropic’s threat report as the strongest open-weight attack model — and on the same day as this release, the speed-tuned GLM-5.3-Highspeed was quietly rolling out in Zcode. This time the blog stays silent on openness, and early replies on Hacker News asked it outright: “Looks like this is not an open model.”
The facts
The numbers behind GLM-Legal, each with its source and measurement:
| Item | Figure / claim | Source |
|---|---|---|
| Harvey LAB all-pass | 19.2%, credit only when every rubric item passes | Official blog, 2026-10-12 |
| Gain over base | +12.5 points (implies a base of roughly 6.7%) | Official blog |
| vals.ai pass rate | 19.17%, ranked 7 of 79 | vals.ai, updated 2026-10-12 |
| vals.ai cost per test | $0.15 ($0.075 input / $0.25 output) | vals.ai |
| LegalBench | Vendor claims parity with top closed-source frontier models at a leaner parameter count | Official blog, vendor claim |
| PLawBench | Vendor claims parity with leading frontier systems on Chinese legal consultation, case analysis and drafting | Official blog, vendor claim |
| Training recipe | Legal mid-training, RL post-training across tens of thousands of simulated legal environments, co-evolved with an agent harness | Official blog |
| Availability | API by application; coming to AutoClaw; no pricing | Official blog |
The two sets of numbers agree: vals.ai scores GLM 5.3 Flash at 6.67%, and adding the officially claimed 12.5 points lands exactly on GLM Legal’s 19.17% — the independent board corroborates the vendor’s arithmetic. What disagrees is the framing. The blog says the model rivals closed-source frontier systems; the leaderboard that confirms the score still shows six models ahead of it, with 6 points to the leader.
What others say
The Hacker News thread (369 points and 209 comments when checked on 2026-10-13) aims most of its fire at the benchmark itself:
- User mathis-l wrote that he had studied agent traces on Harvey’s firm-knowledge task subset and found “multiple problematic tasks and rubrics,” listing three examples. His issue in Harvey’s open-source harvey-labs repository (number 191, still open as of October 13, filed October 9) gets specific: task 113’s rubric allows only three case files for requests for a preliminary injunction, yet matter 1011-00004 in the same dataset also qualifies.
- User jimz went further: on the immigration and white-collar defense tasks, following the “correct” answers would get clients detained or deported, or rewarded for informing on them; the rubric assumes the state’s initial position is right, and it seems not to account for Loper Bright, the decision that reshaped US administrative law. His verdict: at minimum, incompetent.
- otterley, who says he went to law school before returning to engineering: “An 80% miss rate on a benchmark suggests to me that automation has a very long way to go, even for the drafting and research phases” — and even when the output matches a competent attorney, the attorney still has to check every line.
- User postalcoder’s first reaction was that the model “is not an open model,” wondering how sentiment toward Chinese labs shifts as they keep weights proprietary — and whether the timing connects to reports of Harvey getting squeezed on margins by the US labs.
Our take
The absolute number is not the story; the recipe is. Domain mid-training plus RL post-training, with the model and its agent harness iterating together, bought 12.5 points inside one vertical — the playbook coding agents proved out, now aimed at law. Compare that with Opus 5.5 Max topping Arena’s general agent leaderboard on its debut: “frontier” is per-domain now, and the general and vertical boards no longer agree. Read Z.ai’s “rivals frontier models” next to vals.ai’s number 7 — each is true on its own scale, and neither makes sense without the other. For teams building legal tech, three things remain unsettled: no weights, no pricing, and a benchmark whose own quality is under challenge (issue 191 is still open). In a field where even the leader stays under 26%, treat GLM-Legal as a research signal, not a production dependency.
How to try it
- API: apply at z.ai/consultation/legal with a description of your use case, and wait for review.
- AutoClaw: Z.ai says the model is coming soon to the desktop agent platform at autoclaw.z.ai — worth following.
- Check the board yourself: vals.ai’s Harvey LAB page lists all 79 models with per-task pass rates and cost, and Harvey’s harvey-labs task repo is open — you can read the contested task in full.