Models#Benchmark#Model inference
StartLux-Decision v1.1 tops a community Decision Index at 69.12, first of 173 models
StartLux-Decision v1.1 27B ranks first of 173 models on the community-run Jev Decision Index at 69.12, first on all three parts; weights are non-commercial.
StartLux-Decision v1.1 27B is a 27B open-weights decision model: give it a state (text, JSON, up to eight images) and typed questions with listed options, and it returns a probability for every option in one forward pass without generating text. On October 16 a community-maintained Space on Hugging Face, the Jev Decision Index, published its 0.3.2 data and placed the model first of 173 entries with a Full score of 69.12; the runner-up, vLLM-SR d3 (27B), scored 66.26, a gap of 2.86 points.
Key points
- StartLux leads all three parts of the index, not only the total: 72.88 on the public benchmarks, 64.82 on private tests of the same skills and 61.41 on private tasks from new domains. The Full score weights them 20, 50 and 30 percent.
- 69.12 is the only score above 69 in the 173-row table. Jev 1.14, the reference row, scores 61.53 with twelve models ahead of it; Jev 1.13 scores 60.11.
- The leaderboard is a Space whose README says “Unofficial and community-maintained; not affiliated with TypeSafe AI”. The model weights are CC BY-NC 4.0, so commercial use needs a separate license from StartLux Labs.
Background
Decision models answer with a choice instead of a sentence, and the form was popularized by TypeSafe’s Jev in September. OpenAI and Amazon shipped their own within two weeks; Amazon’s Strands Decider 2B also sits on this leaderboard, at 21.34 and rank 135. It is a 2B model, so that line measures size more than it measures approach. Our October 3 entry covered the category forming and TypeSafe’s reported valuation; this board was not part of it.
The Jev Decision Index takes a different route: one maintainer collects open reproductions on Hugging Face and runs all of them through one frozen suite. StartLux shipped v1.1 on October 14. The maintainer committed the 0.3.2 data that added it at 16:36Z on October 16, and the model was renamed from “27B Instant” to “27B” minutes later. The company’s own README only gained the line “ranks first of 174 entries” at 03:07Z on October 17.
How the index scores
The 0.3.2 data file was generated at 2026-10-16T16:39:08Z. Its suite records 42 benchmarks, a 37-benchmark public panel and 110,201 requests, of which 109,759 are scoreable. The Full score is 0.20 × public + 0.50 × private same skills + 0.30 × private new domains, with the private parts mapped onto the public scale using untuned stock models. The page header shows both “173 open models” and “174 models”: the extra row is Jev 1.14’s reference line, which is not one of the 173 entries in the models array.
| Metric | StartLux v1.1 27B | vLLM-SR d3 (27B) | Jev 1.14 |
|---|---|---|---|
| Full score | 69.12 (1st) | 66.26 (2nd) | 61.53 (13th) |
| Public | 72.88 (1st) | 65.45 (5th) | 60.70 (20th) |
| Private, same skills | 64.82 (1st) | 62.83 | 59.28 |
| Private, new domains | 61.41 (1st) | 60.51 | 55.60 |
The leaderboard ran the private tests itself on a single RTX PRO 6000, using the author’s serving setup. Decision agreement with the submitted rows came out at 99.85 percent over 831 rows, against a bar of 98.5 percent. The note says the submitted public run was adopted with a kit rescore of 72.88, equal to the submitted value; the model card’s own public-kit figure is 72.83.
The 173 rows are a mixed field: 82 full fine-tunes, 37 LoRA adapters, 28 LoRA plus a head, 23 pure inference techniques, two head-or-adapter entries and one hosted API. Twelve entries score above Jev 1.14’s 61.53.
Public set as dev set
The caveat is printed in two places. Section 4.1 of the technical report says “The public subset of DI also served as the development benchmark” and that the authors compared Stage 2 data recipes, prompt-attention variants and the seed averages behind the Instant checkpoints by their public scores, so “these scores reflect model selection and are not independent test results”. The model card repeats it: the 72.83 figure is their own run of the public kit, “not leaderboard submissions”.
The leaderboard’s note adds that “Public 0.3 scores guided the recipe and the seed-average choice”, and that, by the authors’ disclosure, training data targets 33 of 38 index benchmarks using official train splits or same-layout generators. The 38 is the authors’ count; the current panel is 37 public benchmarks and 42 Jev benchmarks, so the note’s figure and the panel’s figure are not the same denominator.
Split the lead into its parts and it narrows unevenly. On the public set StartLux is 3.39 points ahead of the next model (KnowLine 35B-A3B Gen1 at 69.49). On new domains it leads by 0.90, and on private tests of the same skills by 0.52, where Perplexity Decider v1.2 sits at 64.30. The same-skills section tests familiar skills on unpublished questions, and it is the section with the smallest margin — the pattern you would expect when public scores have been part of the development loop.
What others say
Chinese coverage arrived on October 18. 36Kr carried a short flash that day; its body sat behind a security check when we fetched it, so nothing from it is quoted here. TMTPost’s longer piece is by ITValue, a contributor post under the site’s 钛度号 label, and it frames the event as “Huggingface 发榜” — Hugging Face issuing a ranking. The data was committed directly by the Space’s maintainer, multimodalart, in commit b7e7ea6770, and Hugging Face as a company has no role in it.
The same piece offers two comparisons that need their denominators. It puts 27B AutoThink’s 73.37 against “Jev (hosted) 57.96” as a lead of about 27 percent; 57.96 is Jev 1.13’s public-set score, so that is public against public. On the Full score Jev 1.13 is 60.11 and Jev 1.14 is 61.53. It also says the first-generation 27B scored 63.88 on DI 0.2.1 and led Jev 1.13’s 57.91 on 31 of 38 benchmarks. The archived 0.2.1 data in the Space, generated on September 28, contains no StartLux row; its leader is Surogate Rune 26B-A4B v3 at 57.44. That 63.88 cannot be checked against the leaderboard.
A second leaderboard tells a different story. Artificial Analysis published its own Decision Index v1.0 on October 13, with Jev 1.13 leading decision models at 58, GLiDE and Microsoft-Decision-1 at 54, and no StartLux entry. Hacker News has no thread on this result; the October 2 post about Decision Index 0.2.1 drew 4 points and no comments. TypeSafe’s Diogo Almeida told 量子位 in early October that public benchmarks are “极易被操纵” — easily gamed — and that he trusts accumulated product experience over a general leaderboard. That position now gets tested by a leaderboard built by someone else.
License and deployment
The weights of StartLux-Decision v1.1 27B are CC BY-NC 4.0: free for research and other non-commercial use with attribution, commercial use by separate license from StartLux Labs. Only the inference package is Apache-2.0. The ranking and the license have to be read separately; here open weights do not mean a product you can ship.
The model card lists BF16 latency on one H200 with one request in flight: 52 ms at 256 tokens, 98 ms at 1,024, 358 ms at 4,096 and 737 ms at 8,192. Two limits matter in practice. One read covers at most 26 options, so longer lists go through group rounds. And none_probability is an alarm score rather than a calibrated probability.
The first step takes three commands: git clone the repository, pip install -e '.[vllm]', then startlux-decide serve --model <local weights> --launch-vllm. Requests go to /v1/systemone, which follows the System One format. Setting think_mode to auto or always returns 422; the Instant build answers once and does not deliberate.
Our take
Two kinds of reader get something direct from this. Teams doing routing, triage or moderation, where the job is picking one of a fixed set of options, can read 69.12 as a structured signal: 80 percent of that number comes from private tests nobody can train on, which is a better proxy for generalization than a public score. Buyers comparing decision models get a second useful fact from the same table: Jev 1.14’s 61.53 has already been passed by twelve models, so price and license start to carry more weight than the reference implementation.
Carry the discounts with the number. The public set fed the recipe and seed choice, and the same-skills margin is 0.52 points, so the real advantage is smaller than 2.86 suggests. CC BY-NC 4.0 keeps commercial deployment out unless you negotiate a license or pair the Apache-2.0 inference package with other weights. And the phrase “top of Hugging Face” should be “top of a community-maintained Space on Hugging Face”; the two claims carry different weight.
What remains unknown is the stability of the board itself. The Space maintainer takes entries by pull request, rows get renamed within minutes of landing and models are re-ranked as versions change. Today’s first place may not be comparable to next week’s reference line.