Sample Page Title

Submit-training Giant Language Fashions (LLMs) for long-horizon agentic duties—akin to software program engineering, net shopping, and sophisticated software use—presents a persistent trade-off between computational effectivity and mannequin generalization^{. Whereas Supervised High-quality-Tuning (SFT) is computationally cheap, it ceaselessly suffers from out-of-domain (OOD) efficiency degradation and struggles to generalize past its coaching distribution. Conversely, end-to-end reinforcement studying (E2E RL) usually preserves OOD capabilities and achieves excessive in-domain accuracy, but it surely incurs huge compute prices as a result of necessity of repeated, many-turn on-policy rollouts for each parameter replace.}

NVIDIA researchers have launched PivotRL, a framework designed to bridge this hole^{. By working on current SFT trajectories, PivotRL goals to ship the generalization advantages of E2E RL whereas sustaining the info effectivity related to SFT.}

The Structure of a Pivot

The core of PivotRL is the transition from full-trajectory rollouts to focused, turn-level updates^{. The framework identifies and makes use of two major mechanisms: Pivot Filtering and Useful Rewards.}

1. Pivot Filtering

In turn-level agentic coaching, each assistant completion at a model-call boundary is taken into account an motion. PivotRL begins by extracting all assistant turns from an SFT dataset right into a ‘pivot candidate’ pool.

The system then profiles these candidates offline utilizing a frozen reference coverage, π₀. To optimize the coaching price range, PivotRL filters for pivots: particular states the place native, on-policy rollouts exhibit excessive variance in outcomes. The filtering standards are outlined by two situations:

• Nonzero empirical reward variance: $hat{sigma}^2(s) > 0$.
• Low reward imply: $hat{mu}(s) < lambda_{diff}$

This strategy addresses the uninformative-turn bottleneck. In group-normalized RL—particularly Group Relative Coverage Optimization (GRPO)—turns the place actions both uniformly succeed or uniformly fail lead to a normalized benefit of zero, offering no significant gradient replace. By specializing in mixed-outcome turns that stay troublesome for the reference coverage, PivotRL concentrates compute on states that present the strongest studying sign.

2. Implementing Useful Rewards

Customary SFT-to-RL variations typically depend on actual string matching with the demonstration knowledge to assign rewards^{. Nonetheless, in generative motion areas (e.g., shell instructions or search queries), a number of functionally equal actions could diverge from the precise string within the coaching knowledge.}

PivotRL replaces strict matching with useful rewards, $r_{func}(s, a) = 1[a in mathcal{M}(s)]$, the place $mathcal{M}(s)$ is the set of domestically acceptable actions decided by a domain-specific verifier. These verifiers can vary from normalized schema checks and string similarity to light-weight LLM-as-a-judge scoring.

Theoretical Foundations: Gradient Sign and OOD Retention

The effectiveness of those design decisions is supported by two major theoretical outcomes:

• Theorem 3.2 (Reward Variance and GRPO Sign): The analysis workforce proved that the Fisher norm of the pure gradient of the statewise reward goal scales with the reward customary deviation. Particularly, the inhabitants GRPO rating, $gamma_{s, beta}, equals frac{sigma}{beta^2}$. This validates the technique of filtering for mixed-outcome pivots to maximise the native in-domain studying sign.
• Theorem 3.3 (Minimal KL Change): This theorem demonstrates that useful reward-based RL shifts chance mass towards acceptable actions whereas preserving the reference coverage’s relative chance ordering for actions unrelated to the coaching process. As a result of the relative rating of task-unrelated actions stays unchanged, PivotRL considerably mitigates the catastrophic forgetting and OOD degradation frequent in SFT.

Efficiency and Effectivity

The analysis workforce evaluated PivotRL utilizing Qwen3-30B-A3B-Pondering-2507 as the bottom mannequin throughout 4 agentic domains: conversational software use $(tau^2-Bench)$, software program engineering (SWE-Bench Verified), terminal management (Terminal-Bench), and net shopping (BrowseComp).

In-Area Accuracy Positive aspects

In comparison with SFT on an identical knowledge, PivotRL achieved superior in-domain outcomes:

• Common Acquire: +14.11 factors over the bottom mannequin, in comparison with +9.94 factors for SFT.
• Area Specifics: PivotRL outperformed SFT on $tau^2-Bench$ (+5.37), Terminal-Bench (+6.25), and BrowseComp (+9.80).

Out-of-Area Retention

Probably the most vital benefit was noticed in OOD stability^{. Whereas SFT brought about a median regression of -9.83 throughout eight OOD benchmarks (together with math and science QA), PivotRL maintained a near-zero common change of +0.21. Notably, PivotRL achieved +10.04% greater OOD accuracy in non-agentic duties in comparison with SFT.}

Compute Effectivity on SWE-Bench

On SWE-Bench Verified, a rigorous customary for long-horizon brokers, PivotRL demonstrated a considerable discount in coaching overhead:

• Flip Effectivity: PivotRL reached accuracy ranges corresponding to E2E RL utilizing 4x fewer rollout turns.
• Temporal Effectivity: Coaching was ~5.5x sooner in wall-clock time than E2E RL when utilizing the identical variety of compute nodes.

Key Takeaways

• Hybrid Effectivity: PivotRL combines the compute effectivity of Supervised High-quality-Tuning (SFT) with the out-of-domain (OOD) generalization of Finish-to-Finish RL.
• Pivot Filtering: The framework identifies ‘pivots’—crucial intermediate turns the place sampled actions present excessive variance in success/failure, offering the strongest studying alerts.
• Useful Verifiers: As a substitute of requiring actual textual content matches, PivotRL makes use of domain-specific verifiers to reward any functionally equal motion.
• OOD Stability: Not like SFT, PivotRL preserves the mannequin’s efficiency on unrelated duties (e.g., math) by sustaining the reference coverage’s chance ordering for task-unrelated actions.
• Manufacturing Velocity: It achieves accuracy corresponding to E2E RL with 4x fewer rollout turns and ~5.5x sooner coaching time, as confirmed in NVIDIA’s Nemotron-3-Tremendous.

Try the Paper. Additionally, be happy to observe us on Twitter and don’t overlook to affix our 120k+ ML SubReddit and Subscribe to our Publication. Wait! are you on telegram? now you may be a part of us on telegram as effectively.