What RL on expert-built tasks teaches Nemotron 3.5 Super, one LoRA adapter at a time
Sushant Mehta and Edwin Chen ran reinforcement learning on NVIDIA Nemotron 3.5 Super with five expert-built task sets, one small LoRA adapter per set. Gains ran from +4.5 to +13.8 points. Here is how the method works and how to read the results.
- Reading time
- 16 min
- Sections
- 9
- Code examples
- 1
- Tables
- 2
- Sources
- 10
- Published
- 19 Nov 2026
- What changed
- Surge AI posts the Nemotron 3.5 Super RL study
- Why it matters
- The paper supports a modest claim.
- What to do
- Start with the grader
On 17 November 2026, Sushant Mehta and Edwin Chen of Surge AI posted a paper on arXiv called "What RL on Expert-Built Tasks Teaches Nemotron 3.5 Super". They took NVIDIA's open Nemotron 3.5 Super model and ran reinforcement learning (RL) on it five separate times. Each run used one expert-built task set and trained one small LoRA adapter. There was no supervised fine-tuning stage first, and no copying of expert answers. The model only learned from its own attempts and a score.
The authors report gains on all twelve headline benchmarks they chose, from +4.5 points on IFBench to +13.8 points on GDP.pdf. They also tried to explain the gains: they compared base and trained attempts side by side and labelled what changed. That second part shows which habits RL changed, not only which numbers moved.
This matters for anyone who builds on open-weight models. If the recipe holds up, a team with good tasks and good graders can add a measurable skill with a small adapter, without touching the base weights.
What happened
Surge AI posts the Nemotron 3.5 Super RL study (17 November 2026)
The paper runs to 18 pages. The authors give Surge AI as their affiliation. Surge AI also built the five training datasets and four of the twelve headline benchmarks: ComplexConstraints, HANDBOOK.md, Chartography and GDP.pdf.
The five task sets cover:
- Software engineering (RL-SWE): a repository plus an expert change request, passed only if every hidden test passes.
- Complex constraints (RL-Constraints): a single-turn request with a rubric of yes-or-no checks.
- Long-horizon tool use (RL-ToolUse): a request inside a simulated company, with mail, chat, calendar, tickets and files exposed as MCP tools.
- Chart understanding (RL-Charts): a professional chart and a value question, with a Python kernel and a crop tool.
- Enterprise document reasoning (RL-Docs): a PDF, a question and a rubric of expected values, with page-view, zoom, search and Python tools.
NVIDIA's Nemotron 3.5 Super is the base (technical report dated 27 October 2026)
NVIDIA's technical report describes Nemotron 3.5 Super as a 120-billion-parameter multimodal model with 12 billion parameters active per token. It is a hybrid Mamba-Transformer mixture of experts. NVIDIA's own post-training already includes RL, with GRPO-trained specialist teachers merged by on-policy distillation. So the Surge study is RL on top of an already RL-tuned model, not RL on a raw base.
Tinker is a post-training API from Thinking Machines. Its October 2026 anniversary post says the user keeps the data and training loop, calls four functions (sample, forward_backward, optim_step and save_state), and can now pick Nemotron models.
How RL post-training with verifiable and rubric rewards works
Think of RL post-training as a driving lesson with a strict examiner. Nobody shows the learner the perfect route. The learner drives, the examiner marks the result, and over many tries the learner keeps what earned marks and drops what lost them. Supervised fine-tuning (SFT) is the opposite: it is copying an expert's route turn by turn.
In each training step of this paper, the loop is simple:
- Sample. For one task, the model produces a group of attempts at temperature 1.0. Agentic attempts can be long: up to 250 turns for software engineering.
- Score. Each attempt gets a reward. Some rewards are verifiable: the hidden tests pass or they do not. Others come from a rubric marked by a judge model. The paper names GPT-6.1 Sol as the judge for the constraints, tool-use, chart and document sets.
- Compare within the group. The advantage of an attempt is its reward minus the group's mean reward. The authors do not divide by the standard deviation. If every attempt in a group gets the same reward, the group carries no signal and is dropped.
- Update. Tokens in better-than-average attempts become more likely, and tokens in worse ones become less likely.
The reward design carries most of the domain knowledge. For RL-ToolUse, the reward is the share of requirements met, but breaking a prohibition drops it to zero or below. For the constraint and document sets, the score is a weighted rubric formula. Small shaping terms penalise overlong answers, cut-off turns and, for tool use, over-persistence.
Rubric rewards are only as good as the judge (see LLM-as-judge). The paper shows the sensitivity: when the authors audited the GDP.pdf rubrics, the gain moved from +13.8 to +18.6 points.
The setup is deliberately bare. The authors report no supervised stage, no KL penalty that holds the model near the base, and no entropy bonus. The optimiser is AdamW with a peak learning rate of 1e-4 for four runs and 2e-4 for software engineering. The paper does not report batch size, group size or step counts, so you cannot fully reproduce the compute budget from the text.
What CISPO changes compared with PPO and GRPO
Most RL for language models descends from PPO. PPO reuses each batch of samples for several updates. Because the model changes between updates, it reweights each token by an importance ratio: how likely the token is under the current model divided by how likely it was when it was sampled. To stop updates from running away, PPO clips that ratio to a narrow band around 1. When a token's ratio leaves the band in the direction of the update, its gradient is cut to zero.
GRPO, introduced in the DeepSeekMath paper as a variant of PPO, keeps this clipping but replaces the learned value model with a group baseline, the same "reward minus group mean" idea used here. That saves memory.
The MiniMax team, in its MiniMax-M1 report from June 2025, found a problem with clipping. Reflective words such as "However", "Recheck", "Wait" and "Aha" act as forks in a chain of reasoning. The base model gave them low probability, so their ratios grew large quickly, and they were clipped out after the first on-policy update. The rare tokens that most needed learning stopped learning.
CISPO (clipped importance-sampling weight policy optimisation) moves the clip. Instead of zeroing the update for an out-of-band token, it caps the importance weight and treats that capped weight as a constant (a stop-gradient). Every token still sends a gradient; outliers just cannot send a huge one. MiniMax reports that CISPO matched the DAPO method in half the training steps on a Qwen2.5-32B test, and admits the gradient is slightly biased by the weight clipping.
An analogy: PPO is a teacher who ignores any answer that changed too much since last week. CISPO is a teacher who still reads every answer, but gives no single answer more than a fixed amount of weight.
| Method | Baseline for advantage | What gets clipped | Effect on rare, fast-moving tokens |
|---|---|---|---|
| PPO | Learned value model | The per-token update, via the ratio band | Gradient can drop to zero |
| GRPO | Mean of a group of attempts | The per-token update, as in PPO | Gradient can drop to zero |
| CISPO (MiniMax-M1) | Group-relative, GRPO style | The importance weight only, with stop-gradient | Gradient kept, but capped |
| CISPO in this paper | Reward minus group mean, no std scaling | Weight clipped to the range 0 to 4 | Gradient kept, capped at 4x |
The paper also caps "staleness", meaning how many model versions old the sampler may be: from 2 for tool use and documents to 8 for charts. Stale samples let importance weights drift further from 1, which is where CISPO's cap does its work.
Why one small LoRA adapter is enough for RL
LoRA freezes the base weights and learns a small pair of low-rank matrices beside each weight matrix. The 2021 LoRA paper reports, for GPT-3 175B, 10,000 times fewer trainable parameters and 3 times less GPU memory, with no added inference latency. You can see the idea animated in the LoRA simulator.
Here the adapter is rank 32 on attention and MLP weights, including the experts, plus the unembedding layer. The vision encoder stays frozen in the chart and document runs.
Why should so few parameters be enough? Thinking Machines gave an information-based answer in "LoRA Without Regret" (September 2025). In supervised learning, every target token carries information, so an episode delivers roughly as many bits as it has tokens. In policy-gradient RL, the learning signal per episode is a single number, the advantage. That is about one bit's worth, however long the attempt. Their estimate for a MATH training run was about 320,000 bits absorbed in total, compared with about 3 million parameters in a rank-1 LoRA on an 8B model. In their tests LoRA matched full fine-tuning for RL even at rank 1.
A school report card is a fair everyday picture. A whole term of work comes back as one grade. So RL rarely fills even a small adapter. The same post adds two conditions: apply LoRA to all layers, especially MLP and mixture-of-experts layers, and use a learning rate about ten times higher than for full fine-tuning.
Serving one adapter per domain
Five adapters on one base model raise a serving question. Because LoRA leaves the base untouched, a server can keep one copy of the 120B base in memory and swap or batch small adapters per request. The S-LoRA paper (2023) showed this at scale: thousands of adapters on one or a few GPUs, with a shared memory pool for adapter weights and the KV cache, and up to 4 times the throughput of earlier approaches, by the authors' measure. The S-LoRA authors note that serving engines such as vLLM already had basic LoRA support, which their design extended.
In practice you would route code tasks to RL-SWE and document questions to RL-Docs.
How to read the per-benchmark gains
The table lists the twelve headline benchmarks, with base and trained scores in percent and the gain in points. The brackets show 95% paired bootstrap confidence intervals, as reported by the authors. Base scores come from the same Tinker deployment as each adapter.
| Benchmark | Adapter | Base | Trained | Gain [95% CI] |
|---|---|---|---|---|
| SWE-bench Pro V2 | RL-SWE | 86.9 | 92.1 | +5.2 [+3.3, +7.1] |
| Terminal-Bench 2.1 | RL-SWE | 63.9 | 70.3 | +6.3 [+2.4, +10.6] |
| DeepSWE | RL-SWE | 27.9 | 32.6 | +4.7 [-0.7, +9.8] |
| ComplexConstraints | RL-Constraints | 27.3 | 35.5 | +8.2 [+2.3, +14.0] |
| AdvancedIF | RL-Constraints | 50.6 | 58.3 | +7.7 [+5.8, +9.7] |
| IFBench | RL-Constraints | 72.3 | 76.8 | +4.5 [+2.4, +6.7] |
| MultiChallenge | RL-Constraints | 58.3 | 66.6 | +8.3 [+5.5, +11.2] |
| AutomationBench | RL-ToolUse | 14.8 | 27.5 | +12.7 [+9.3, +16.2] |
| HANDBOOK.md | RL-ToolUse | 54.8 | 67.6 | +12.8 [+9.5, +16.2] |
| Chartography | RL-Charts | 39.4 | 50.6 | +11.2 [+7.5, +15.0] |
| ChartMuseum (format-lenient) | RL-Charts | 79.35 | 85.80 | +6.45 [+4.80, +8.10] |
| GDP.pdf (published rubrics) | RL-Docs | 10.4 | 24.2 | +13.8 [+7.8, +20.2] |
Four reading rules help.
Look at the interval, not just the point. DeepSWE's interval crosses zero, so that gain could be noise. Some secondary results are flat: the paper reports -0.5 on SWE-bench Verified and +1.6 with a wide interval on Terminal-Bench under the Terminus 2 harness.
Check the scorer. ChartMuseum uses a format-lenient score; the official scorer gives 76.90 to 84.25, a +7.35 gain. GDP.pdf rose to +18.6 after a rubric audit, but the authors keep the published-rubric number as the headline.
Check what tools were allowed. RL-Charts gained +11.2 on Chartography with tools, but only +0.2 with no tools. The adapter learned to use its tools better, not to see charts better.
Count the tokens. The trained checkpoints sample 1.23 times (ChartMuseum) to 2.07 times (SWE-bench Pro V2, second run) as many tokens per attempt as the base. Plan cost and latency budgets accordingly.
Transfer across domains
The most interesting single number is +8.0. The document adapter never saw API tools in training, yet its AutomationBench pass rate rose from 13.5 to 21.5 percent. The tool-use adapter, trained for that domain, gained more at +12.7. Something general moved. The side-by-side data suggest it was the habit of looking things up: RL-Docs attempts that never read Sheets or Drive fell from 27.5 to 11.7 percent. But transfer had a cost. The share of AutomationBench tasks where RL-Docs broke at least one rule rose from 34.7 to 40.0 percent, while RL-ToolUse, which was punished for prohibitions, cut it from 37.3 to 32.5 percent. Skill transferred; caution did not.
Trained only on single-turn requests, the constraint adapter still gained +8.3 on the multi-turn MultiChallenge benchmark.
Continue learning
How this article connects
10 pages link to or from it. Drag to turn, click a dot to open it.
Reading the failure-mode analysis
The authors took pairs where the adapter solved a task the base failed, and labelled why. Claude Opus 5.5 agents labelled; a second set relabelled and agreed on 29 of 30 tool-use pairs and 23 of 30 software pairs. Samples were small: 30 pairs per adapter, 24 for charts and 11 for constraints.
The labels point to habits rather than knowledge:
- Tool use. The most common base failure was not finding the data or the target (12 of 30). The share of attempts that never opened the inbox or Slack fell from 74.3 to 41.2 percent. In plain terms, the trained model checks its sources before acting.
- Software engineering. The top base failure was a lost requirement (13 of 30). Attempts that wrote their own test or scratch file on SWE-bench Pro V2 rose from 14.3 to 32.6 percent.
- Charts. The base often judged close calls by eye (10 of 24). Attempts with no measurement code on non-text questions fell from 67.9 to 44.4 percent.
- Constraints. Most labelled base failures (8 of 11) came from an earlier turn not carried into the answer. But one check got worse: the line-indentation constraint on IFBench fell from 56.9 to 33.3 percent met. RL can trade one habit for another.
The method has a built-in tilt: it only reads tasks the adapter newly solved, so it explains wins rather than losses. For more on reading evaluation claims with care, see a score is only as good as its report.
What it means in practice
The paper supports a modest claim. On a strong open model that has already been RL-tuned by its maker, a further round of RL on a narrow, well-graded task set still moves related benchmarks by several points. A rank-32 adapter carries that change, and no expert demonstrations were needed.
The value sits in the tasks and graders, not the optimiser. The domains with the sharpest graders, such as hidden tests and prohibitions that zero the reward, produced behaviour that is easy to name: write a test, open the inbox, measure the bar.
For agent builders, per-domain adapters pair well with a router: one base, several adapters. The post-training simulator compares this with SFT and preference methods.
What to do now
- Start with the grader. Write the reward before the training loop. Prefer checks that can be verified, like tests, and use rubrics only where you must.
- Add hard negatives to the reward. Copy the RL-ToolUse idea: an action that breaks a rule should zero the reward, not just lower it.
- Decontaminate. The authors removed repositories, charts and documents that overlap their benchmarks and ran a 13-gram overlap check. Do the same before you trust a gain.
- Use LoRA on all layers. Include MLP and expert weights, start from a learning rate about ten times your full fine-tuning rate, and test a small rank first.
- Evaluate base and adapter on the same deployment. The paper found base log-probabilities differed by 0.40 nats per token between two deployments. Small serving differences can fake a gain.
- Report intervals and tool settings. Publish paired bootstrap intervals and run with and without tools, as the chart results show why.
- Track tokens per attempt. Measure cost per solved task, not just pass rate.
- Test for rule-breaking after transfer. If you route an adapter outside its domain, measure its prohibition-violation rate, not only its success rate.
Risks and open questions
The authors grade some of their own work. Surge AI built the training sets and four of the twelve benchmarks. Gains also appear on outside benchmarks such as IFBench, but independent replication matters.
Judges shape the results. Rubric rewards and several benchmarks rely on GPT-6.1 Sol as judge. The HANDBOOK.md gain was +13.1 under a different judge, close but not identical, and the GDP.pdf audit moved its gain by nearly five points.
The compute budget is unclear. Batch size, group size and step counts are not reported, so it is hard to price the recipe or compare it with full fine-tuning.
One base model. All results come from Nemotron 3.5 Super on Tinker.
Combining adapters is unexplored. We do not know whether the five adapters can be merged, stacked or distilled into one model without losing the gains or the safeguards.
Watch next: independent re-runs on public task sets, a release of the adapters or tasks, results on a second base model, and any study that merges per-domain RL adapters and measures both skill and rule-breaking.
Frequently asked questions
What is the main finding of the paper?
RL on one expert-built task set, carried by a single rank-32 LoRA adapter, improved Nemotron 3.5 Super on twelve related benchmarks. The authors report gains from +4.5 to +13.8 points. They also show which habits changed, such as writing test files or checking sources first.
Why did they skip supervised fine-tuning?
Skipping SFT isolates RL. Without it, every change comes from the model's own attempts and the grader's score, not from copying expert answers. That makes the behaviour changes easier to attribute to RL.
What is CISPO in one sentence?
CISPO is an RL objective from the MiniMax-M1 report that clips the importance-sampling weight instead of clipping the token update. Every token keeps some gradient, which helps rare reasoning tokens keep learning.
Can a rank-32 adapter really change a 120B model?
For RL, yes, according to this paper and to Thinking Machines' earlier LoRA study. RL gives roughly one scalar of feedback per episode, so it needs little capacity.
Should I trust these numbers?
Treat them as the authors' reported results with clear caveats. Most intervals exclude zero, but some do not, and several benchmarks were built by the authors' employer. Use the paper as a strong recipe and test it on your own tasks.
Summary
Surge AI trained five rank-32 LoRA adapters on Nemotron 3.5 Super with CISPO on Tinker, one per expert-built task set, with no SFT. The authors report gains from +4.5 on IFBench to +13.8 on GDP.pdf, an +8.0 transfer gain on AutomationBench, and new habits such as checking sources and writing tests. Small adapters suffice because RL feedback carries little information per episode, and CISPO keeps rare tokens learning. The caveats are self-built benchmarks, judge sensitivity, longer attempts, unreported compute and one base model. For builders, the tasks and graders carry most of the value.
Sources
- Paper arXiv: What RL on Expert-Built Tasks Teaches Nemotron 3.5 Super (abstract)
- Paper arXiv: What RL on Expert-Built Tasks Teaches Nemotron 3.5 Super (full text PDF)
- Official NVIDIA Research: Nemotron 3.5 Super and Lightning Technical Report
- Paper arXiv: MiniMax-M1, Scaling Test-Time Compute Efficiently with Lightning Attention (CISPO)
- Paper arXiv: DeepSeekMath (GRPO)
- Paper arXiv: LoRA, Low-Rank Adaptation of Large Language Models
- Web Thinking Machines: LoRA Without Regret
- Web Thinking Machines: One Year of Tinker
- Web Thinking Machines: Tinker documentation
- Paper arXiv: S-LoRA, Serving Thousands of Concurrent LoRA Adapters
Cite this article
Kelahmet, Ö. C. (2026, November 19). What RL on expert-built tasks teaches Nemotron 3.5 Super, one LoRA adapter at a time. Özer Cem KELAHMET. https://www.ozercemkelahmet.com/articles/rl-expert-tasks-nemotron-lora-adapters
BibTeX
@online{kelahmet2026expert,
author = {Kelahmet, {\"O}zer Cem},
title = {{What RL on expert-built tasks teaches Nemotron 3.5 Super, one LoRA adapter at a time}},
year = {2026},
month = {nov},
url = {https://www.ozercemkelahmet.com/articles/rl-expert-tasks-nemotron-lora-adapters}
}Version historyfirst version
- Published
Factual corrections are listed here with what changed. Wording and formatting edits are not listed.