Open Models Almanac
Methodspost-trainingrollout & RL infrastructure

general family · filed under post-training

Asynchronous reinforcement learning

An asynchronous reinforcement learning infrastructure decouples sample generation (rollout) from training in the RL post-training pipeline.1,2 It colocates rollout and training on the same devices, time-sharing their execution and maintaining high rollout concurrency to mitigate the long-tail bottleneck in the rollout phase.3,4

How this definition was written

Summarised by glm-5.3-flash from 2 sources; every sentence is checked against the quote it cites.

  1. asynchronous reinforcement learning infrastructure that drastically improves post-training efficiency by decoupling generation from training — GLM-5: from Vibe Coding to Agentic Engineering
  2. further decouples generation from training to maximize GPU utilization — GLM-5: from Vibe Coding to Agentic Engineering
  3. We colocate rollout and training on the same physical devices and time-share their execution — DeepSeek V41 Tech Report
  4. significantly mitigates the long-tail issue in the rollout phase by maintaining a sufficiently high level of concurrency — DeepSeek V41 Tech Report

Also called asynchronous RL architecture, asynchronous training paradigm, asynchronous training, asynchronous RL framework, fully asynchronous training, fully asynchronous RL training pipeline.

sources
14
models
10
labs adopt it
7
strongest
core

Documented in

Evidence

18 quotes from 14 sources, one card per model, each opening with its strongest.

How sources treat it
evaluated 1used 10core 7

Kolibri-1

Aleph Alphaadopted3 sources

We trained in an asynchronous Reinforcement Learning setup, where vLLM inference ran in parallel to training.

coreoptimizationin KolibriAleph Alpha
3 more from this model

use asynchronous training, where we generate training data on our environments using the current model in parallel to training the model on already generated data.

usedoptimizationin KolibriAleph Alpha

Both measures include the policy staleness induced by our asynchronous setup.

usedoptimizationin KolibriAleph Alpha

Rollouts and optimisation run continuously and asynchronously.

coresoftware implementationin KolibriAleph Alpha

MiMo-V2.6-Flash · MiMo-V2.6-Pro

Xiaomiadopted1 source

we scale batch size and training throughput through fully asynchronous training, processing thousands of long-horizon rollouts and billions of tokens per step

coresoftware implementationin MiMo-V2.6 RL trainingXiaomi
1 more from these models

These advances are supported by asynchronous training, mixed-task rollout infrastructure, training–inference consistency mechanisms

usedoptimizationin MiMo-V2.6

we employ an asynchronous RL architecture that decouples training from inference and leverage multi-token prediction to accelerate rollout generation

usedpost trainingin Nemotron 3 familyNVIDIA
1 more from this model

It utilized an asynchronous RL architecture that decouples training from inference and leverages MTP to accelerate rollout generation.

usedpost trainingin Nemotron 3.5 LightningNVIDIA

NVIDIA-Nemotron-3-Ultra-550B-A55B

NVIDIAadopted2 sources

It utilized an asynchronous RL architecture that fully decouples training from inference across separate GPU devices

usedoptimizationin Nemotron 3 UltraNVIDIA
1 more from this model

asynchronous RL architecture that fully decouples training from inference across separate GPU devices

usedoptimizationin Nemotron 3 UltraNVIDIA

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑