Research#Benchmark#Model inference
Weight-space programming edits Qwen3.5-0.8B weights, with scores rising as loops fall
MIT CSAIL reported on Nov. 2 that GPT-6 Astra changed 0.016% of Qwen3.5-0.8B’s weights; MATH-500 rose from 7.9% to 48.1%.
Weight-space programming treats model parameters as objects that can be read and edited directly, rather than values updated only through gradient training. In a November 2, 2026 research blog, MIT CSAIL researchers described eight experiments in which GPT-6 Astra read small networks, wrote networks from scratch, and edited Qwen3.5-0.8B. For the last experiment, they reported that MATH-500 rose from 7.9% to 48.1%. The jump is worth examining, but it mainly tracks the repair of a repetitive reasoning loop; it does not establish that the model learned new mathematical ability.
Key points
- Experiment 8 changed 131,072 Qwen3.5-0.8B parameters, about 0.016% of its 0.8 billion weights; the authors reported gains on MATH-500, GSM8K, and IFEval.
- Astra had a Python environment for weight operations, practice questions from GSM8K and MATH, and up to 50 forward-pass evaluations. The workflow used neither gradient descent nor a training loop.
- Independent work verified the released weight diff and reproduced a similar MATH-500 score, but could not reproduce how the closed GPT-6 Astra model discovered the edit. The result looks more like a targeted loop repair than a general math upgrade.
What weight-space programming did
The team presented eight experiments in a research blog, not a peer-reviewed paper. They cover reading weights, repairing a control policy, writing networks from scratch, and editing a small language model. The most readily checked result is Qwen3.5-0.8B: the model repeatedly generated reasoning text and often ran out of its 32,768-token output budget before answering. The researchers say Astra traced the loop to a direction written by the MLPs in layers 15 and 19, then subtracted it from 64 output columns of each layer’s down_proj.
This was not a model rewriting its own brain unaided. Astra had a Python environment for manipulating weights, 7,469 GSM8K and 7,275 MATH practice questions, and 10, 25, or 50 forward-pass evaluations. There was no gradient descent or training loop. In the 50-pass setting, the authors report changing 131,072 parameters, or 0.016% of the model, in two tensors. The cover diagram puts the blog’s read, fix, and improve experiments side by side; the Qwen comparison is the panel on the right.
The reported scores
The authors’ November 2 post used thinking-on inference with up to 32,768 generated tokens. Sampling counts differ by benchmark; MATH-500 used four samples per question. The headline scores need the loop rate beside them to be interpreted.
| Qwen3.5-0.8B | Base model | After 50-pass edit |
|---|---|---|
| MATH-500 | 7.9% | 48.1% |
| GSM8K | 13.8% | 67.1% |
| IFEval | 21.0% | 62.3% |
| MATH-500 answers hitting token limit | 93.0% | 33.1% |
The last row offers the clearest explanation: many base-model answers stalled in repetitive loops, while more edited-model answers finished. The team’s own control also narrows the claim. On Gemma 3 270M, which rarely looped, MATH-500 moved from 6.8% to at most 7.7%, a gain the authors say was not established. “The score rose after a loop was repaired” fits the evidence better than “a tiny weight edit generally improves mathematical reasoning.”
What independent checks found
The Hugging Face pngwn/weight-space-repro project checked the public diff tensor by tensor and reports that separate independent runs of the 50-pass edit scored 50.2% and 51.0% on MATH-500 under the thinking-on protocol. Its authors could not test whether GPT-6 Astra would discover the same change because Astra is closed; they could check the released diff and the behavior of the edited model, not the discovery capability. A separate DEV Community audit re-evaluated 500 questions with four samples each, also scoring 51.0%, and found that 31.6% of answers still hit the token limit, close to the original 33.1%.
A separate NYU Shanghai RITS roundup adds reports of other replications: Qwen 3.8 27B and Kimi K3.1 Pro were used to repeat some small-network identification tasks, and Claude Opus 5.5 was used for a localized edit to Gemma 4. These involve different models, tasks, and evaluation settings; they are not replications of Qwen’s math score. RITS also notes that the experiments on real language models used Python. The Qwen run included practice questions and repeated forward-pass evaluation, so “direct weight editing” should not be read as tool-free understanding of an arbitrary model.
Our take
The strongest engineering signal so far is a small, inspectable patch to public weights: the diff can be reconstructed tensor by tensor, and independent evaluation of the edited Qwen model lands near the authors’ MATH-500 score. This differs from fine-tuning the whole model. The method tries to locate and change a direction associated with a particular behavior, which may be useful to researchers testing localized failure fixes, reversible patches, or model-behavior hypotheses.
The limits are equally specific. Most of the reported MATH-500 increase comes from one Qwen setup with a severe repetition loop; the same study did not establish a math gain on its Gemma control. Python weight manipulation, practice data, and multiple forward evaluations are part of the workflow. Rebuilding the public diff independently does not show that another model would find the same patch, nor that small edits can give arbitrary models new skills. For a related look at evaluation design, compare our RoboColiseum multi-agent benchmark; for the product context around GPT-6 Astra, see our coverage of the GPT-6.1 Astra launch plan.
How to try it
The RikiyaT/lm_understand-logs repository contains the authors’ prompts, responses, code, and evaluation records; the Hugging Face model card hosts the edited Qwen3.5-0.8B weights. Independent evaluators have also published their reconstruction and scoring steps. To check the result, compare the edited model and its base under the same inference settings, thinking mode, sampling parameters, and grader, while recording the share of answers that hit the token ceiling. Comparing only 7.9% and 48.1% misses how much of the change comes from completing answers that previously looped.