A 65.4-billion-token stage at 256K context uses repositories, issue–pull-request pairs, commit histories, build and CI logs, and agent trajectories.
4 more from this model
The mid-training mixture combines whole repositories, issue–pull-request pairs, commit histories, build and CI logs, and agent trajectories.
We mid-trained the long-context checkpoint for 65.4B tokens at a sequence length of 256K.
67B tokens of software-engineering mid-training at 256K context
67B tokens of software-engineering mid-training at 256K context