Open Models Almanac
Methodspost-trainingrollout & RL infrastructure

methods · post-training

rollout & RL infrastructure

98 methods filed at this node or below it, from the sources of 17 models.

In this branch 98

Everything filed at this node or below it, with one collapsible heading per child node.

filed here 98

Asynchronous reinforcement learning core · 14 sources · 18 quotes
Partial rollout used · 3 sources · 5 quotes
Adaptive Rollout Concurrency used · 2 sources · 2 quotes
Co-located rollout and training used · 2 sources · 2 quotes
On-policy rollout generation used · 2 sources · 2 quotes
Seamless Rollout Engine used · 2 sources · 2 quotes
SLIME used · 2 sources · 3 quotes
Token-in-Token-out (TITO) used · 2 sources · 3 quotes
88 from one source, or only mentioned as not used
Agent Loop core · 1 source · 1 quote
Agent-centric rollout execution core · 1 source · 1 quote
Compute-node sharding into scale units core · 1 source · 1 quote
Hierarchical trajectory data organization core · 1 source · 1 quote
One rollout worker per non-controller node core · 1 source · 1 quote
Payload Porter core · 1 source · 1 quote
Persistent multi-tenant harness pool core · 1 source · 2 quotes
Predictive Rollout Dispatch core · 1 source · 2 quotes
Sample-level dispatch default · 1 source · 1 quote
Adaptive Rollout Scheduling used · 1 source · 1 quote
Autoscaling rollout actors used · 1 source · 1 quote
Autoscaling supporting services used · 1 source · 1 quote
Bounding off-policy sample ratio used · 1 source · 1 quote
Capping trajectory staleness used · 1 source · 1 quote
Centralized trajectory bin-packing used · 1 source · 1 quote
Data re-sampling strategy used · 1 source · 1 quote
Data Scheduler used · 1 source · 2 quotes
Decoupled control plane and data plane used · 1 source · 2 quotes
Dialogue prefix matching used · 1 source · 1 quote
Distributed rollout workers used · 1 source · 1 quote
Fixed-usable-trajectory-count batching used · 1 source · 1 quote
Heterogeneous Agent Harnesses used · 1 source · 1 quote
Incremental multimodal-delta transfer used · 1 source · 1 quote
Independent parallel agent rollouts used · 1 source · 1 quote
Long-horizon rollout budgets used · 1 source · 1 quote
Maximum-staleness replay-buffer eviction used · 1 source · 1 quote
Mixed-task rollout infrastructure used · 1 source · 1 quote
Mixing tasks and harnesses in one RL batch used · 1 source · 1 quote
Multi-harness rollouts used · 1 source · 1 quote
Multi-Task Rollout Orchestrator used · 1 source · 1 quote
Multimodal metadata/payload separation used · 1 source · 1 quote
Overlapped RDMA weight-transfer writes used · 1 source · 1 quote
Per-dataset rollout concurrency limits used · 1 source · 1 quote
Per-step NCCL weight updates used · 1 source · 1 quote
Preserving the environment sampling mix used · 1 source · 1 quote
Provider failover routing used · 1 source · 1 quote
Replay-buffer shared-prefix deduplication used · 1 source · 1 quote
Routed-Expert Replay used · 1 source · 1 quote
Sample Mixer used · 1 source · 1 quote
Sample replay used · 1 source · 1 quote
Sample-grained garbage collection used · 1 source · 1 quote
Shared composable interface for RL tasks used · 1 source · 1 quote
Teacher-generated rollout trajectories used · 1 source · 1 quote
Token-level interruption of generation used · 1 source · 1 quote
Tool Manager used · 1 source · 2 quotes
Toolbox used · 1 source · 2 quotes
Asynchronous group-wise grading optional · 1 source · 1 quote
Discarding early-returned short samples optional · 1 source · 1 quote
Deficit-corrected scheduling evaluated · 1 source · 2 quotes
Batch-level rollout dispatch not used · 1 source · 1 quote
Prompt-level rollout dispatch not used · 1 source · 1 quote

By model

Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

Modelmethods
DeepSeek-V4.1-Flash coreAsynchronous reinforcement learning coreCompute-node sharding into scale units coreSample-level dispatch defaultAsynchronous reinforcement learning on synthesized tasks usedBounding off-policy sample ratio usedCo-located rollout and training usedDecoupled sandbox and worker-container architecture for RL rollouts usedPer-dataset rollout concurrency limits usedReuse of persisted rollout state without re-prefilling usedSample-grained garbage collection usedScaffold-agnostic rollout control and trajectory normalization usedState-preserving suspension and resumption of rollouts usedToken-granular persistence of rollout state usedToken-level interruption of generation usedDiscarding early-returned short samples optionalBatch-level rollout dispatch not usedPrompt-level rollout dispatch not used—
MiMo-V2.6-Flash coreAgent Loop coreAgent-centric rollout execution coreAsynchronous reinforcement learning coreHierarchical trajectory data organization corePayload Porter corePersistent multi-tenant harness pool corePredictive Rollout Dispatch coreAdaptive Rollout Concurrency usedAdaptive Rollout Scheduling usedBackground-thread offloading of blocking rollout work usedComposite early-stop strategy that zeroes reward and masks affected contexts usedDecoupled control plane and data plane usedDialogue prefix matching usedDuration- and acceptance-adjusted per-source oversampling usedHeterogeneous Agent Harnesses usedIncremental multimodal-delta transfer usedIsolated resettable sandboxes for agent rollouts usedMixed-task rollout infrastructure usedMultimodal metadata/payload separation usedPartial rollout usedPer-source priors for rollout length estimation usedSample Mixer usedSample replay usedTraining–inference consistency mechanisms usedAsynchronous group-wise grading optionalDeficit-corrected scheduling evaluated—
NVIDIA-Nemotron-3-Ultra-550B-A55B coreOne-step off-policy asynchronous reinforcement learning coreAsynchronous reinforcement learning usedOn-policy rollout generation used—
DeepSeek-V4-Pro usedPreemptible rollout service with write-ahead logging used—
GLM-5.2 usedAsynchronous Agent Reinforcement Learning algorithms usedAsynchronous reinforcement learning usedMulti-Task Rollout Orchestrator usedSLIME usedToken-in-Token-out (TITO) used—
Inkling coreAsynchronous reinforcement learning core—
Kimi K3 usedCo-located million-token agentic RL system with resumable sandboxes usedCo-located rollout and training usedPartial rollout used—
Kolibri-1 coreAsynchronous reinforcement learning coreOne rollout worker per non-controller node coreRouting sequences to the same inference replica coreAsynchronous batching of parallel retrieval queries usedCapacity allocation balancing trainer and inference throughput usedDistributed rollout workers usedIsolated resettable sandboxes for agent rollouts usedMaximum-staleness replay-buffer eviction usedNon-blocking weight synchronization to inference usedOverlapped RDMA weight-transfer writes usedPreserving the environment sampling mix usedRDMA weight distribution through one replica per node usedReplay-buffer shared-prefix deduplication usedRouted-Expert Replay used—
Laguna-S-2.1 usedBlocking in-flight rollout steps during weight updates usedCapping trajectory staleness usedClosed-loop multi-turn synthetic-data rollout usedExact rollout-to-production chat-template alignment assertion usedLong-horizon rollout budgets usedMulti-harness rollouts usedRL sandboxing with selective network blocking and artifact caching usedToken-in-Token-out (TITO) used—
MiMo-V2.5 usedContinuous rollout with asynchronous rewards and early termination usedData re-sampling strategy usedData Scheduler usedPartial rollout usedSeamless Rollout Engine usedStaleness-aware truncated importance sampling usedTool Manager usedToolbox used—
MiMo-V2.5-Pro usedContinuous rollout with asynchronous rewards and early termination usedData re-sampling strategy usedSeamless Rollout Engine used—
MiMo-V2.6-Pro coreAgent Loop coreAgent-centric rollout execution coreAsynchronous reinforcement learning coreHierarchical trajectory data organization corePayload Porter corePersistent multi-tenant harness pool corePredictive Rollout Dispatch coreAdaptive Rollout Concurrency usedAdaptive Rollout Scheduling usedBackground-thread offloading of blocking rollout work usedComposite early-stop strategy that zeroes reward and masks affected contexts usedDecoupled control plane and data plane usedDialogue prefix matching usedDuration- and acceptance-adjusted per-source oversampling usedHeterogeneous Agent Harnesses usedIncremental multimodal-delta transfer usedIsolated resettable sandboxes for agent rollouts usedMixed-task rollout infrastructure usedMixing tasks and harnesses in one RL batch usedMultimodal metadata/payload separation usedPartial rollout usedPer-source priors for rollout length estimation usedSample Mixer usedSample replay usedTraining–inference consistency mechanisms usedAsynchronous group-wise grading optionalDeficit-corrected scheduling evaluated—
Mistral-Large-4 coreAsynchronous reinforcement learning coreShared environment implementation across RL, evaluation, and synthetic data generation coreActor-pool orchestration for asynchronous RL usedAdaptive Rollout Concurrency usedAutoscaling rollout actors usedAutoscaling supporting services usedBackpressure- and concurrency-controlled rollout admission usedCentralized trajectory bin-packing usedDiscarding infrastructure-failure rollouts in RL usedExcluding auxiliary environment-model calls from training usedFixed-usable-trajectory-count batching usedLow-staleness optimization for long-trajectory RL usedPause and resume in-flight rollouts across policy updates usedPer-step NCCL weight updates usedPrioritizing sibling rollouts from started groups usedProvider failover routing usedRemote teacher hidden-state transfer with trainer-side logit reconstruction usedShared composable interface for RL tasks usedSubmission-only termination for environments with a submit tool optional—
NVIDIA-Nemotron-3.5-Lightning-30B-A3B usedAsynchronous reinforcement learning usedIndependent parallel agent rollouts usedOn-policy rollout generation usedOne-step-stale asynchronous response generation usedTeacher-generated rollout trajectories used—
NVIDIA-Nemotron-3.5-Super-VL-120B-A12B usedIndependent parallel agent rollouts usedOn-policy rollout generation usedOne-step-stale asynchronous response generation usedTeacher-generated rollout trajectories used—
Qwen3.5-397B-A17B usedAsynchronous RL framework for large-scale agent scaffolds and environment orchestration used—
SmolVLA evaluatedAsynchronous reinforcement learning evaluated—
↑