Safety#AI safety#Model training
OpenAI now reads its models' internals: activation probes score every token from GPT-5.6 on and can cut a reply short mid-stream
OpenAI says probes score model internals token by token, cutting replies mid-stream for review — and resisting jailbreaks better than larger text classifiers.
OpenAI published “Training activation classifiers for safety and alignment” on its Alignment blog on November 2, 2026, and it documents something that has already been running in production: activation classifiers, or probes. Since GPT-5.6, these lightweight models have been attached to the internal activations of OpenAI’s most capable models, scoring every assistant completion token by token as it streams. When a score crosses a threshold, the system stops streaming to the user, pauses the completion, blocks any tool calls that have not been cleared as safe, and hands the message to a reasoning model for a full review. Two claims in the post draw the most attention: OpenAI reports that probes are more robust to jailbreaks than its significantly larger text-based classifiers, and that probe accuracy improves as the monitored model gets more capable. Both deserve scrutiny, and the post itself supplies the caveats.
Key points
- Probes read the model’s residual stream activations — typically a vector from a layer around the middle of the network — and output binary classification logits. They run in-process on the inference engine and return one score per sampled token; OpenAI says this makes them easier to make fail-closed than an external text classifier.
- Training scale: up to 1 million messages per probe (a mix of synthetic and de-identified conversations, labeled by OpenAI’s most capable reasoning models acting as LLM judges), and at least 32 random initializations per hyperparameter configuration — initialization randomness accounts for roughly 85% of probe performance variance, so OpenAI trains many seeds and keeps the best.
- Two limitations OpenAI states itself: probes are supervised, so they only detect harm categories they were explicitly trained on; and a probe is tied to the base model’s weights — a new pre-trained base means retraining, and weight drift within the same family forces periodic retraining.
Background
The probes post did not come out of nowhere. On October 29, OpenAI published “How we detect frontier model misalignment,” which laid out internal monitoring as a five-layer funnel: activation probes as the layer-0 high-recall screen, then model-based review, investigative agents, automated forensics, and finally human response, with the highest-priority alerts paging staff within about 15 minutes. The November 2 post effectively dissects that layer 0 in detail. We have covered this line before from two angles: METR and Redwood Research red-teamed the same internal monitoring system in October and found multiple bypass paths, including one that hid “hundreds of recent actions” from the monitor (see our coverage), and the October 23 report on frontier training-environment security, where red-team agents found chained VM escapes in the sandbox stack (see our report). The research road was already mapped, too: Anthropic’s Constitutional Classifiers in their January 2026 update ensembled linear probes with external classifiers for roughly a 40x screening-cost reduction, and the March Trojan-Speak paper showed adversarial fine-tuning could evade text classifiers at over 99% while costing less than 5% capability — while arguing that activation-level probes substantially improve robustness against exactly such attacks. OpenAI’s choice validates that line of research.
Engineering details
The probe itself is small. Its input is a residual stream activation vector and its output is a binary logit; the architecture builds on attention probes (Kantamneni et al. 2025) with an MLP plus QV-attention, and an exponential moving average (EMA). The EMA is what makes streaming deployment practical: compared to a rolling mean, it cuts the probe’s streaming state from O(window size × number of heads) memory to O(number of heads), with what OpenAI calls essentially no difference in predictive power. Training replaces the per-prefix hard max with a “rolling softmax” objective, which modestly improves the recall/flag-rate Pareto front in their experiments. Architecture selection included a robustness comparison that matters for attacks: the MultiMax design is significantly more sensitive to benign prefilled prefixes placed before a violative suffix, while OpenAI’s exponentially decaying attention design degrades far less (Figure 1). Linear, covariance and full-attention probes were all tried and none beat the chosen design. Evaluation runs on two curves: recall on a small carefully labeled set, and escalation rate on a deployment-representative set, combined into the recall-escalation curve that gates deployment; jailbreak robustness is tested with automated and manual red-teaming.
| Dimension | Number | Source |
|---|---|---|
| Training messages per probe | up to 1M | Alignment blog, 2026-11-02 |
| Random initializations per config | at least 32 | Alignment blog, 2026-11-02 |
| Variance explained by initialization | ~85% | Alignment blog footnote, 2026-11-02 |
| Training epochs | fewer than 5 (AdamW) | Alignment blog, 2026-11-02 |
| Streaming state memory | O(number of heads), after EMA | Alignment blog, 2026-11-02 |
Jailbreak robustness
OpenAI gives two layers of explanation for why probes resist jailbreaks better than text classifiers. The structural one: a text classifier reads surface tokens, so content the model expresses through encodings it understands — but external classifiers cannot interpret — is invisible to the text layer. A probe reads internal activations directly and skips that translation. The Trojan-Speak attack, where a model learns a communication protocol no outside observer can parse, is nearly undetectable by text monitoring yet still lives in the activation space the probe watches. The empirical one: in OpenAI’s red-team testing, probes were overall more robust than content-based classifiers, especially against jailbreaks not optimized against the probe. The post immediately adds the limit: real-world attacks adapt to signals from the safety stack, which is why OpenAI keeps red-teaming — read the other way, the robustness advantage is not guaranteed to survive once attackers get feedback from the probe.
The capability-scaling claim is subtler. In principle it makes sense: a probe’s input is the monitored model’s own activations, and in a more capable model the internal representation of violating intent may be sharper, hence easier to read. OpenAI reports the trend continues on its frontier models. But note what scales is classifier quality, not the safety of the monitored model — a more capable model with violating intent reads more clearly to the probe, which is a different statement from the model misbehaving less. In production terms the claim also hides a dependency: probe availability is now tied to the pace of training stronger base models, and every new base resets the probe fleet to retraining.
Our take
For AI safety engineers, the value here is the architecture rather than the conclusions: a cheap token-level probe doing high-recall screening, an expensive reasoning model doing low-false-positive final review, calibrated through thresholds and an escalation-rate curve. That two-tier pattern is directly reusable, and Constitutional Classifiers’ cascade follows the same logic. For most developers the near-term impact is small — this is monitoring of OpenAI’s own infrastructure, and at the API surface a cut-off and review just looks like a refusal or a delay.
Three honest catches. First, every effectiveness number comes from OpenAI’s own internal experiments, with figure captions noting they are “not necessarily representative of deployed classifiers,” and no third party has verified them — METR and Redwood’s October assessment showed that external reviewers given three weeks of access could find real gaps in this very system’s coverage. Second, “only things we can explicitly train them to detect” means probes are blind to novel harms; they intercept known patterns, not unknown risks. Third, the coupling to base weights means each new pre-trained base forces a retrain of the whole probe fleet, so the safety layer’s refresh cadence is now driven by the base model release cadence. What to watch: OpenAI says it is expanding the set of policies probes cover — the next system card or Preparedness update, and whether it publishes probe thresholds and escalation rates, is the next test of how transparent this layer will be.