Laya: structured decisions in one forward pass, no generation

Support-ticket triage, phishing detection, agent routing: what these jobs share is that they need a definite answer, not generated prose you have to parse. Laya is a non-autoregressive “System 1” decision engine — feed it text together with typed questions (a choice, a score, a yes/no) and a single forward pass returns structured results with calibrated probabilities, no text generation in between. The README measures about 33 ms per question on a T4 GPU, or 7.2 ms per question batched, across 100+ languages; as of October 27, 2026 the repository sits at roughly 34,500 stars under Apache-2.0. It writes no code and no emails. It does one thing — typed questions over given text — and pushes that one thing’s latency well under a third of what a generative call costs.

Core features

• Typed decision primitives: choice (options with criteria), score (an ordered scale) and noul (a calibrated P(true) yes/no) — one call can ask several at once, and results are option-order invariant.
• One forward pass: the models are encoder-sized (ModernBERT-large, 421M parameters; mmBERT-base, 322M) with no autoregressive decoding; the README reports 33 ms per question on a T4 and 7.2 ms batched (checked October 2026).
• 100+ language routing: a built-in Router detects language and script and dispatches each request to the right checkpoint; 48 of 51 tested languages score above three times the random baseline in the README’s table.
• Confidence gating and abstention: set a threshold so the model abstains visibly when unsure instead of guessing; since v0.4.16 abstention behaves identically across the Python, Rust and .NET ports.
• Full integration surface: an HTTP server (laya[serve]), an MCP server, LangChain/LlamaIndex/CrewAI selectors, ONNX and a TileLang GPU fast path, plus a separately versioned TypeScript SDK (the laya-ts npm package).
• Fine-tuning workflow: RLCD — reinforcement learning against strictly proper scoring rules — lifts the official typed-decisions benchmark from 0.362 on the base checkpoint to 0.766 fine-tuned.

Typical use cases

• Ticket triage: one support email in, three answers out — which department, how urgent, churn risk or not — each with a probability; high-confidence tickets route automatically, the rest go to a human.
• Agent routing: in a multi-agent system, let Laya classify intent and priority before handing off, saving an order of magnitude of latency over generate-then-parse.
• Content-risk screening: batch scoring for phishing, spam and moderation signals, fully self-hosted so the data never leaves your network.

Quick start

python -m pip install laya

from laya import Router

router = Router()  # downloads a checkpoint on first use
result = router.predict(
    "Hi, we were billed twice for March. Refund today or we cancel.",
    {"department": {"type": "choice",
                    "criteria": {"billing": "invoices, payments, refunds",
                                 "technical": "bugs, outages, errors"}}},
)
print(result["answers"]["department"]["choice"])  # billing

The result is a plain Python dictionary with the answer, its probability and the model the Router picked. To try it without installing anything, there’s a laya-demo Space on Hugging Face.

Summary

Laya fits teams that want to pull classification, scoring and yes/no calls out of a generative pipeline: millisecond latency, structured output, explicit abstention and self-hosting, exactly the shape of high-frequency small decisions like ticket triage and agent routing. Be honest about the limits: it only answers typed questions within its training distribution — open-ended understanding and writing still need a generative model; long-context capacity has a hard edge, with the official benchmark showing accuracy turning volatile past roughly 5,000 tokens of preceding text; and matching the benchmark’s fine-tuned numbers means building your own labeled data. Apache-2.0, Python first, with Rust and .NET ports. For more routing context, see the orchestration notes in everything-claude-code, or agentplay to visualize several coding agents’ sessions before you decide what routes where.