Models#Benchmark#Model training
NVIDIA's Nemotron for Science adds a 49k-question chemistry set and a post-training recipe
NVIDIA's Nemotron for Science pairs Nemotron 3.5 Super VL with LitmusBench v0.2, 47,118 chemistry questions, and a NeMo Gym post-training recipe.

NVIDIA announced Nemotron for Science on November 16, bundling the Nemotron 3.5 Super VL model it open-sourced a month earlier (120B total parameters, 12B active, up to 1M tokens of context) with the chemistry dataset v0.2 that has been public since October 15 and a post-training recipe. LitmusBench v0.2 holds 47,118 training questions and 2,056 test questions, roughly nine times v0.1’s question count. The press release says both model and dataset are on Hugging Face and that the model was optimized to run on a single 128GB DGX Spark.
Key points
- The recipe is the real addition: LitmusBench v0.2 draws on ChEMBL, ExpansionRx and Biogen data, covering 126 molecular properties and 12 assay endpoints in short-answer form, about 64 MB in total. Version 0.1 had 5,232 training and 482 test questions from ChEMBL alone.
- The licence is split per question: the model card is OpenMDW 1.1, not Apache, and the dataset licenses ChEMBL-derived questions under CC BY-SA 4.0 with everything else under OpenMDW 1.1.
- The recipe is code in a blog post, not a script: NVIDIA documents how to turn a CSV of experiments into a verifiable-reward environment in NeMo Gym, ships GRPO and SFT configs for NeMo RL, and runs its example on 16 GPUs.
Background
Nemotron for Science builds on the October 15 release. That launch shipped Super VL’s weights, dataset documentation, NVFP4 checkpoints and RL environments; our October 16 entry covered it and flagged that every headline number was vendor-run. This round adds a dataset, a recipe writeup and three customer stories. The model itself did not change: the BF16 repository’s card was last modified on October 22 and never mentions LitmusBench or a science edition, and the single-device instructions live on the separate NVFP4 card.
The timing points at SC26. NVIDIA’s technical blog files the post under an SC26 tag, and the same day’s newsroom carried a Vera CPU platform expansion and a quantum computing collaboration, with scientific computing as the thread. LitmusBench itself is not new: v0.1 last changed on September 29, and v0.2 widens the sources from one to three and replaces a uniform CC BY 4.0 licence with a per-source split.
The facts
| Item | Value | Source |
|---|---|---|
| Model | 120B total / 12B active, Mamba-2, MoE and attention hybrid, C-RADIO vision encoder | HF model card |
| Context | Up to 1M tokens, native speculative decoding via MTP | HF model card |
| Model licence | OpenMDW 1.1 | HF model card |
| Dataset size | 47,118 train + 2,056 test, 126 properties, 12 assay endpoints, about 64 MB | HF dataset card |
| Dataset licence | ChEMBL-derived questions CC BY-SA 4.0, the rest OpenMDW 1.1 | HF dataset card |
| Previous version | v0.1: 5,232 + 482 questions, ChEMBL only, CC BY 4.0 | HF dataset card |
| Post-training example | Drug-metabolism prediction, 16,384-token output budget; SFT +12%, RL a further +17% | NVIDIA technical blog |
| Example scale | 4 nodes × 4 GPUs = 16 GPUs; GRPO 100 steps, SFT 55 steps | NVIDIA technical blog |
| Deployment | BF16 example on 4× H100; NVFP4 on one DGX Spark, single sequence | HF model cards |
The customer numbers arrive one organization at a time, and NVIDIA is the only source for all of them. Lila Sciences post-trained on laboratory feedback and says quantum-dot synthesis accuracy nearly doubled to 61%, above GPT-6 Astra’s 54% at 10–20× lower inference cost; the same passage claims frontier-level results for designing execution-ready lab protocols before any extra fine-tuning. Terray Therapeutics extracts reactants, products and reaction conditions from chemistry diagrams and recovered about 76% of annotated reactions on the RxnScribe test under that benchmark’s matching criteria. Therna post-trained for mRNA design, reports close to double the frontier accuracy on protein-expression ranges, and says scientists preferred its explanations over the base model’s in 96% of comparisons. KTH Royal Institute of Technology built an agent that repairs broken GROMACS molecular dynamics setups: one overnight run on a single DGX Spark fixed 81% of test cases with reasoning enabled, each fix verified by GROMACS itself.
Institutions are described as users, not as benchmarks. Argonne National Laboratory is wiring Nemotron into its StructBioReasoner multi-agent system for protein-based therapeutic candidates; Jülich Supercomputing Centre is integrating it into drug and scientific discovery agents; RIKEN put the Nemotron family on its RIKYU platform through the RiVault inference gateway. The release gives no performance figures for any of them.
Inside the recipe
The technical blog is the most useful document in the bundle because it shows the machinery. Four pieces turn a CSV of experimental records into an RL environment: a converter to JSONL, a resource server that scores submissions, a registration YAML, and the trainer wiring. The score is Morgan-fingerprint Tanimoto similarity at radius 2 and 2048 bits; the model must call a submit_metabolite tool with a SMILES string, and that number becomes the reward, with no preference labelling.
The code sits in two Apache-2.0 repositories, NeMo Gym and NeMo RL, at roughly 1,338 and 2,135 stars on November 17, both pushed on November 16–17. Distillation runs Anthropic’s Claude Sonnet 5.5 over 1,934 training prompts and keeps 1,123 reasoning traces at a Tanimoto threshold of 0.7 or better, which then serve as an SFT warm start.
It is a tutorial, not a product. The GRPO config names the model NVIDIA-Nemotron-3.5-Super-VL-120B-A128-BF16, with an extra 8 in the parameter suffix; the SFT config file is named for LoRA while its lora_cfg.enabled is false; and there is no notebook or Dockerfile. Reproducing the example takes four nodes of four GPUs and some familiarity with the NeMo stack.
What others say
Unite.AI’s writeup on the same day adds the caveat NVIDIA leaves out: the gains are for bounded tasks and do not amount to laboratory autonomy, and what decides the outcome is verifier design, task choice and held-out evaluation. The piece attributes the results to the vendor and does not claim any independent replication.
Developer attention has not followed. As of November 17 the Hacker News submission of the press release sat at 3 points and no comments, while a Nemotron 3.5 Transcribe post in late October drew 134 points and 26 comments. That may be SC26 crowding the calendar, or it may be that people building science agents have not picked this up yet.
The naming is inconsistent across NVIDIA’s own materials: the press release says LitmusBench v0.2, the blog says Litmus-Bench V2.0, and the Hugging Face repository is Nemotron-RL-litmus-bench-v0.2. Same dataset, three strings for search engines to split. The vendor-benchmark problem also carries over from Qwen4-Max: nothing here has been re-measured by anyone outside NVIDIA.
Our take
The idea worth copying is the environment, not the model. Take experimental records you already have, turn them into a task with an automatic score, and let RL close the loop; 1,934 prompts plus a strict verifier moved a 120B model by 12% with SFT and a further 17% with RL on drug metabolism in the vendor’s own example (the post does not say whether these are percentage points or relative gains). For a university group with a handful of GPUs, that is a more realistic path than waiting for a bigger general model.
The catches are concrete. The 61% versus 54% comparison has no published question set, scoring script or sample size, so it is impossible to tell model capability from task construction. The DGX Spark deployment runs one sequence at a time, which suits a single researcher and not a shared lab service. The config errors suggest nobody re-ran the tutorial end to end before publishing. And the hardware is real money: the 128GB DGX Spark lists around $6,950, while NVIDIA’s own BF16 example serves on four H100s.
Three things stay open. The scientific benchmark results in the blog are a bar chart with no numbers table. The dataset card does not say whether the test split is drawn from the same sources as training, which decides how much the scores mean. And no post-trained science checkpoint has been published: the three customer models stay in-house, so nothing fine-tuned on a lab’s data is on Hugging Face.
How to try it
Start with the dataset: nvidia/Nemotron-RL-litmus-bench-v0.2 has one JSONL each for train and test, short answers only, with provenance per question. To run the recipe, clone NeMo Gym, work through the CSV-to-JSONL step, swap metxbiodb.csv for your own records, then replace the scoring function in resources_servers/metxbiodb/app.py; check that the verifier grades sensibly before starting GRPO, and keep the first task narrow. For inference only, use the NVFP4 checkpoint with the LiLiCorr drafter and set --max-num-seqs 1 in the vllm serve command.