Open Models Almanac

Multimodal · X2

Hearing, speaking and keeping time: audio, omni models and multimodal positions

A separate speaking model keeps a large model's steps and context for text at the price of a second stack; one backbone is a single model whose every speech frame takes a step and a position; and time can travel in position ids, which cost no tokens, or in text, which is read like any other token

Illustration of a workbench with a large brass horn at the left and a small metronome beside it. A long strip of paper with evenly spaced coral and teal marks runs out of the horn towards the right.

An hour of speech would cost Qwen2.5-Omni 90,000 tokens of audio, nearly three times the 32,768 its config accepts. Qwen3.5-Omni, working from the same spectrogram, spends 22,500. Written out as words, the hour would be about 11,700 tokens (assuming 150 words a minute and 1.3 tokens a word), and audio tokens pay for silence too. Qwen spent two generations cutting the first number to a quarter.

Then the model has to answer aloud. A speech codec is a network that compresses sound into a short list of integers per slice of time and rebuilds sound from them. One that runs at 12.5 slices (frames) a second with eight integers (codes) per frame needs 100 codes every second, from a different alphabet than the 3 to 4 text tokens the same speech would take (Défossez et al., 2024). Something has to produce those codes on time without making the language model that decides what to say worse at its job. And once sound and pictures share one sequence, the model also needs to know when each token happened, which its position in the sequence no longer tells it.

Getting audio in resembles the job X1 covers for images. Audio out and time have no counterpart there.

What a model sees when it hears

Sound reaches a computer as a list of air-pressure samples, 16,000 numbers a second at 16 kHz.

The usual first step is a spectrogram. Cut a short window of samples (typically 25 ms), measure its energy at each pitch, slide along by a hop (typically 10 ms) and repeat. The result is a picture with time on one axis and pitch on the other. A mel spectrogram spaces its bands as human hearing does, closely at low pitches and widely at high ones. “128 mel bands, 25 ms window, 10 ms hop” means 100 columns a second, of 128 numbers each.

A toy, with a synthetic signal: two chirps, not speech. A column is one 10 ms slice of the spectrogram.

columns per token

100 columns a second ÷ 8 = 12.5 tokens a second, 80 ms a token.

Waveformenvelope
Windows25 ms each, one every 10 ms; a few drawn
Spectrogram24 mel bands, low at the bottom; stronger is more opaque
Tokenseach groups the columns above it
00.25 s0.5 s0.75 s1 s
time
Figure 1.A toy signal turned into a spectrogram and then into tokens. At a 10 ms hop there are 100 columns a second, and the downsampling factor gives 25, 12.5 or 6.25 tokens a second.

A token is one position in the language model’s sequence. A frame is the time step of an encoder or codec, typically 40 or 80 ms, and a code is one integer chosen from a learned table. On the way in, one frame usually becomes one token. On the way out, one frame holds several codes.

A model generates one token at a time, a decode step, and stores each earlier token’s attention keys and values in memory, the KV cache, which every new step reads. An audio token therefore costs compute once when the prompt is first read (prefill), memory and read bandwidth at every later step, and room in a fixed-length context (00). A token rate is that cost; it does not set the model’s time resolution, since one wide token can encode what happened inside its 160 ms.

Audio in: three routes to a token

A mel spectrogram and a trained encoder

Most models this site tracks run a small network over the spectrogram, and its downsampling factor sets the token rate. All three Qwen Omni reports use the same front end (16 kHz, 128 mel bands, 25 ms window, 10 ms hop), so 100 columns a second go in. Qwen2.5-Omni reduces them to about 25 tokens a second and Qwen3-Omni to 12.5. Qwen3.5-Omni’s AuT encoder downsamples “16 times using 4 Conv2D blocks” to a “6.25Hz token rate”, one token per 160 ms. At that rate 256K tokens hold about 11.7 hours of audio alone (computed), consistent with the report’s “more than 10 hours”. No report isolates what the lower rate loses.

Gemma 4 E2B and E4B use a 305M encoder “based on the Universal Speech Model”, “consisting of two downsampling convolution layers followed by twelve Conformer layers”. A Conformer layer adds a convolution block to a transformer layer (Gulati et al., 2020). The two stride-2 convolutions give 25 tokens a second (computed from the processor’s audio_ms_per_token: 40), “we keep weights frozen during pre-training”, and a clip is capped at 750 tokens, 30 seconds. Google’s QAT blog notes that encoders can be left out of a deployment (“deploying only the modalities you need”). LFM2-Audio uses 17 FastConformer layers and 8× downsampling for 12.5 embeddings a second.

Each of these needs a separate network: 115M parameters in LFM2-Audio, 305M in Gemma 4, 650M in Qwen3-Omni (its Table 1). Those parameters are resident memory. Going from 25 tokens a second to 6.25 instead cuts prefill steps and KV bytes by 4 and leaves the encoder’s memory unchanged. In a live session few positions arrive each second, so the rate is mainly a KV-cache cost (our reading).

Raw samples and one layer

Gemma 4 12B has no audio encoder. “Raw audio is segmented into 40ms chunks at 16kHz, resulting in 640-dimensional vectors per chunk. These are projected directly into the LLM embedding space.” In the code the projection is an RMSNorm and one linear layer from 640 to 3,840 dimensions. Forty milliseconds at 16 kHz is 640 samples, so it spends the same 25 tokens a second as E2B.

Google’s stated reason is that split encoders “add latency and increase memory usage”. Without an encoder, the backbone has to do what a Conformer did. A linear layer can express a Fourier transform but cannot take a magnitude: shifting a waveform by one sample changes every raw input number and barely changes a spectrogram, and that invariance is left for the backbone to learn. The RMSNorm also removes the loudness contour across frames by setting each 40 ms frame to unit level (our reading of the code). The report says “competitive audio-text performance can be achieved without a dedicated audio encoder”, but 12B also has a far larger language model than E4B, so its table is no clean comparison of the routes.

Discrete input, which means two different things

Inkling and MiMo both turn audio into discrete values before the language model. A quantizer does that: it replaces a vector by the index of its nearest entry in a table, called a codebook.

Inkling’s “discrete token encoding” is dMel (Bai et al., 2024). Each of 80 mel bands is clamped and rounded to one of 16 fixed levels, and a frame is the sum of 80 embeddings from one table of 1,280 rows of 6,144 numbers, about 8M weights (computed). The levels are fixed, the table that embeds them is learned, and there is no encoder, so the blog calls the design encoder-free. One token covers 50 ms, 20 a second.

MiMo-V2.6 uses a learned codebook. A 24-layer, 308M audio tokenizer ends in a 20-level residual quantizer (explained below) that writes 20 codes for each 40 ms frame. A 6-layer, 127M transformer takes four consecutive frames, attends both ways inside that group, and projects them into one backbone token, so 25 frames a second become 6.25 tokens a second. MiMo-V2.5 has the same code and tokenizer config, though its card lists a 261M encoder “further finetuned” from MiMo-Audio-Tokenizer. So MiMo-V2.6 has 435M of encoder for 6.25 tokens a second where Inkling has 8M for 20, and Xiaomi’s report gives no reason for the discrete step.

context

1 hour of speech against a 256K context: 22,500 to 90,000 tokens by route; written as text, 11,700.

  • mel + encoder
  • raw waveform
  • discrete
  • text
  • past one clip's cap
  • selected context
  1. Qwen3.5-Omni (AuT)mel + encoder22,5008.6%
  2. MiMo-V2.5/2.6 (tokenize, patch of 4)discrete22,5008.6%
  3. LFM2-Audiomel + encoder45,00017.2%
  4. LFM3-Audiomel + encoder45,00017.2%
  5. Inkling (dMel)discrete72,00027.5%
    hatched: past the 10 min vLLM cap of one clip (12,000 tokens); needs splitting
  6. Gemma 4 E2B/E4B (mel + Conformer)mel + encoder90,00034.3%
    hatched: past the 30 s cap of one clip (750 tokens); needs splitting
  7. Gemma 4 12B (raw 640-sample frames)raw waveform90,00034.3%
    hatched: past the 30 s cap of one clip (750 tokens); needs splitting
  8. Qwen2.5-Omnimel + encoder90,00034.3%
  9. Same speech as texttext11,7004.5%
050K100K150K200K250K
tokens (linear, from 0) · percentages are of the selected context
Figure 2.What one stretch of speech costs in context under each input route. The default hour takes 22,500 tokens at 160 ms a token and 90,000 at 40 ms; written as text it would take about 11,700.

The models here that generate audio all take it in as continuous features and emit it as discrete codes. MiMo and Inkling read discrete values and generate only text.

Audio out: codecs and codebooks

A language model picks from a finite list at each step, so the usual way to make it talk is to give sound a finite alphabet. An encoder compresses a stretch of waveform into a vector, a quantizer picks the nearest codebook entry, and a decoder rebuilds the waveform: the VQ-VAE recipe (van den Oord et al., 2017). The language model then only has to predict table indices.

Why one codebook is not enough

A single table would be huge. Zeghidour et al. (2021) give the example for SoundStream: at 6,000 bits a second and 75 frames a second, each frame gets 80 bits, which “requires storing a codebook with N=280N=2^{80} vectors, which is obviously unfeasible”.

Residual vector quantization (RVQ) splits the job. Quantize the vector with a first small codebook, subtract the result, quantize what is left with a second codebook, and carry on for KK levels. With ff frames a second, KK levels and codebooks of size NN, the bitrate is

R=f⋅K⋅log⁡2N.R = f \cdot K \cdot \log_2 N .

SoundStream’s 80 bits become eight tables of 1,024 entries, 8,192 vectors in place of 2802^{80}. Storage and search shrink. The error is larger than a search over all the sums would give, because the encoder picks level by level. Multi-stage quantization for speech goes back at least to Juang and Gray (1982); SoundStream learned it end to end.

Mimi, the codec from Moshi (Défossez et al., 2024), runs at 12.5 frames a second with eight codebooks of 2,048 entries: 12.5×8×11=1,10012.5 \times 8 \times 11 = 1{,}100 bits a second. Those are the opening’s 100 codes a second.

A toy, with invented codebooks: 16 points per level, each level's scale half the last's. Not any real codec's tables.

The error never rises as levels are added, and falls on average: a level that picks its table's origin leaves it unchanged. The search is greedy, level by level, so it does not find the nearest of all combinations.

remaining error after each leveltoy units, linear from 0
  1. start0.713
  2. level 10.194
  3. level 20.124
  4. level 30.082
  5. level 40.021
  6. level 5not used
  7. level 6not used
  8. level 7not used
  9. level 8not used
levels K
4
bits per frame, K log₂ 16
16
table entries, 16K
64
flat table, 16K
65,536
  • target (drag it)
  • where the chain ends
  • the tables, drawn around the origin, smaller each level
  • the target stays inside
Figure 3.A toy residual quantizer. Each level picks the nearest entry for what is left. Four tables of 16 points store 64 points in all and can name 65,536 sums. The search is greedy, level by level, so it does not find the nearest of them, and each table holds the origin so the error never rises.

Mimi’s levels are not one cascade. The first is a plain vector quantizer holding “semantic” information, distilled from a speech model. The other seven form an RVQ that runs beside it on the same latent, and the outputs are added, so that acoustic detail need not survive in the first level’s residual.

The codec is trained first, on waveforms. In SoundStream and EnCodec the nearest-entry choice passes no gradient, so the encoder trains as if quantization were the identity (VQ-VAE’s straight-through trick), and the entries move by a running average of the vectors assigned to them, which is online k-means. Then the codec is frozen, and its codes are training data for the language model, which needs only a cross-entropy against the true next code.

Predicting the codes: time and depth

The simplest plan flattens a frame’s codes into the sequence and has the language model emit one at a time. But the next code has to be ready before the audio now playing runs out. At 100 codes a second that leaves 10 ms for a step of the largest model, per user; at 12.5 frames a second it leaves 80 ms. A frame’s codes are also an NKN^K-way joint choice, the flat codebook’s problem again.

The way out splits prediction along two axes. The large model steps once per frame, along time. A small model then predicts the frame’s codes along depth, meaning from the first codebook level to the last, each conditioned on the earlier ones and on the large model’s output: the chain rule. Lee et al. (2022) introduced the split for images as RQ-Transformer, and Moshi uses it with a 7B temporal and a 6-layer depth transformer. Moshi’s Table 5 measures the value: at a 240 ms delay setting, perplexity falls from 135 with independent heads to 37 with a depth model, about 5 bits a code against 11 nominal (computed; a rough average over levels).

Adding levels costs only small-model steps, fKfK of them, while the large model stays at ff.

  • LFM2-Audio feeds each backbone step into a 115M RQ-Transformer that “runs for 8 steps”. The eight codes are embedded, summed into one vector and fed back, so one frame takes one backbone position. Its loss weights each code’s cross-entropy from 100 for the first to 1 for the eighth, which “we found empirically to improve perceived audio quality”; Moshi likewise gives its semantic code a weight of 100.
  • Qwen’s Talker predicts the first codebook with its own head and uses an “ultra-lightweight” dense transformer for the residual codebooks of that frame (Qwen3-Omni: 80M). The report calls this a multi-token prediction (MTP) module.
  • LFM3’s LiquidCodec is a product code with no learned table: the latent is split into 12 groups and each is rounded on a fixed grid of 8×7×6×6 = 2,016 values (finite scalar quantization). In the style of MaskGIT all 12 slots start masked, and a small model fills 4 at a time over three passes, each seeing the earlier ones.
Figure 4.Flat prediction runs the large model once per code. The time-and-depth split runs it once per frame and hands the codes of each frame to a model a tenth its size, which runs eight times per frame (LFM2-Audio: 115M × 8 against 1.2B), so by parameter count its compute is close to one large-model step. The saving is in the large model's step rate and KV reads.

The small model’s steps are not free. LFM2-Audio’s runs 8 times per backbone step, close to one backbone step if compute follows parameter count (8 × 115M / 1.2B = 0.77, computed), and in Qwen3-Omni’s Table 2 the MTP takes 14 ms a token against about 7 ms for the Talker itself. The split saves the large model’s step rate and KV reads.

Streaming decoders

Decoding a frame must take less time than the frame lasts. A causal decoder uses no future samples, so it can produce sound as soon as a frame arrives. Qwen3-Omni replaced Qwen2.5-Omni’s block-wise diffusion decoder with a causal ConvNet, at 3 to 5 ms in Qwen3.5-Omni’s table. Liquid AI found “the Mimi decoder too slow for the target hardware” and trained a 35M detokenizer for LFM2-Audio.

Who speaks

Qwen and Liquid build speech output differently. In LFM2-Audio and Qwen3.5-Omni text and speech share one sequence. In LFM2-Audio one model writes the text itself, while Qwen3.5-Omni’s Talker receives it from the Thinker.

Thinker and Talker

Qwen2.5-Omni introduced the split: a Thinker, a language model producing text, and a Talker producing codes, “motivated by” the dual-track Mini-Omni (Xie and Wu, 2024). Qwen3.5-Omni “continues to adopt” it. Both stacks are mixture-of-experts (MoE) transformers, in which a router sends each token through a few of many parallel feed-forward blocks, so only some parameters are active per token.

What crosses between them has changed. Qwen2.5-Omni’s Talker read the Thinker’s hidden representations plus the sampled text. In Qwen3-Omni the “Talker no longer consumes the Thinker’s high-level text representations”. Qwen3.5-Omni’s Talker is conditioned on “historical text tokens, multimodal representations, and the streamed text of the current turn”.

The Qwen3.5-Omni report also merges the Talker’s text input and speech output, formerly two tracks, into one stream, with ARIA (Adaptive Rate Interleave Alignment). Its rule: “for any prefix of the generated sequence, the cumulative speech-to-text token ratio must not exceed the corresponding item-level global ratio”. In words, at every point speech so far may not run ahead of text so far, relative to the whole utterance’s ratio. The rule needs the utterance’s total counts, and the report does not say what replaces it during generation. The Talker also takes a system prompt for the voice, with “textual descriptions and codec sequences”.

The report gives no output frame rate. It inherits Qwen3-Omni’s RVQ representation (12.5 Hz), and its “Generation RTF” is within about 0.01 of Talker time per token divided by 80 ms, so we take 80 ms: 160 ms a token in, 80 ms a frame out. The RTF may simply be computed that way, so this remains an inference.

Sizes are public only for Qwen3-Omni: a Talker of 3B parameters with 0.3B active, plus the 80M MTP and a 200M codec, beside a Thinker of 30B with 3B active, about 11% more weights in memory (computed). In Qwen3.5-Omni the Talker has its own context, trained up to 64K tokens against the Thinker’s 256K, so history is held twice. Per second of speech the Talker does most of the stepping. With the report’s theoretical time per token, our 12.5 frames a second and an assumed 3.25 text tokens a second, Flash spends about 18 ms in the Thinker and 178 ms in the Talker (MTP included) for one user, and Plus 57 and 186 (computed). These are wall-clock times on unstated hardware. The large model’s steps and context stay free for text, and a scheduler has to run two dependent loops.

A backbone that must also learn speech might get worse at text. The report says Qwen3.5-Omni-Plus’s text is “on par” with its text-only sibling Qwen3.5-Plus-Instruct. Plus scores lower on 12 of 14 benchmarks, by up to 5 points, and the report does not credit the split for the result.

One backbone

LFM2-Audio, at 1.5B parameters, uses one model. In interleaved mode, “we generate 6 text tokens followed by 12 audio tokens and repeat until all text tokens are exhausted”, then finishes the remaining audio. The report calls the text “a kind of chain-of-thought”: the model writes what to say, then translates it to speech. The ratio exists “to keep text generation ahead of audio generation and to minimize time-to-first-audio token”. A sequential mode instead generates one modality at a time, switching on control tokens.

The report does not define “audio token” in that sentence. We read it as one 8-code frame; LFM3’s code, which has the same interleaved_n_text and interleaved_n_audio fields, calls the 12 “speech frames”. (The 12 codes in each LFM3 frame and the 12 frames in each block are different twelves.) A block is then 0.96 seconds of speech. While text remains, the backbone takes 18.75 steps per second of speech: 6.25 text and 12.5 audio (computed). That is the rate the hardware must sustain. Text runs out about halfway through the audio (at 3.25 tokens a second), and the rest is frames alone, so over a whole reply the context pays about 16 positions per second of speech.

Moshi, outside this site’s set, is neither design. Each step of its 7B model reads 17 values at once: one text token, 8 codes of the user’s speech and 8 of its own. So it listens while it talks (full duplex), and every 80 ms costs a step, silence included.

Figure 5.If one audio token is one frame, a block of LFM2-Audio's pattern is 0.96 seconds of speech and 18 backbone steps; when the text runs out, frames continue alone. In a Thinker–Talker model the large model handles only the text and a smaller model handles the frames, at 12.5 a second as inferred from the report's table. The placement of text inside the Talker's stream is schematic, since ARIA's rate is adaptive and unpublished.

We infer two costs for one backbone. A 32K context lasts about 34 minutes of speech at those 16 positions a second (computed), and training tokens spent on codes are not spent on text. Neither has been measured, and with LFM2-Audio at 1.5B parameters and Qwen3.5-Omni at “hundreds of billions”, public data cannot separate design from scale.

Keeping time

Text needs one number per token. A video frame has a row, a column and a moment, and its sound has the same moment.

RoPE (rotary position embedding, T4) rotates pairs of query and key dimensions by an angle proportional to the token’s position id. The two rotations combine, so as a function of position the attention score depends only on the difference between the ids. Each pair ii has a rotation speed θi=b−i/n\theta_i = b^{-i/n} for nn pairs and base bb; this “frequency” has nothing to do with pitch. A fast pair separates neighbours, but after a full turn two far-apart tokens look alike. A slow pair tells far-apart tokens apart and barely separates neighbours. A model needs both.

Most models here have no extra position axes in the backbone: every token gets its index in the sequence, as text does. That includes Gemma 4, MiMo, Kimi K3 and GLM-5.3-Flash; in MiniMax-M3 and DeepSeek the extra axes are inside the vision encoder. Inkling has no RoPE at all and uses a learned relative-position term in attention.

M-RoPE: one id per axis

M-RoPE (Wang et al., 2024) gives each pair one of three ids, time, height or width. A token pair’s score then depends on three id differences, one per axis. For fixed content the position part is a sum of one function per axis, and the softmax turns the sum into a product: M-RoPE assumes a separable kernel, as a separable filter does in image processing (our reading). Text tokens get the same id on all three axes, so a text-only sequence reduces to ordinary RoPE.

From config, for Qwen3.5-397B (head_dim 256, partial_rotary_factor 0.25), Qwen3.6-35B and Qwen3.8-Flash-Next: 64 rotated dimensions make 32 pairs, split mrope_section: [11, 11, 10]. The code assigns them in rotation: pair 0 to time, 1 to height, 2 to width, 3 to time, and so on. Each axis then holds every third speed, itself a shorter RoPE: neighbouring speeds on one axis differ by a factor of 4.5, where one axis with all 32 pairs would have 1.65 (computed). Interleaving gives up resolution to keep range. Earlier block allocations (Qwen2.5-Omni’s [16, 24, 24]) gave time only the fastest pairs. Huang et al. (2025) compare allocations on a Qwen2.5-based model and report that giving every axis the full range of speeds helps, long video especially.

Figure 6.How the rotary speeds are shared among time, height and width. With interleaving (top, Qwen3.5-397B) each axis has fast and slow pairs; with blocks (bottom, Qwen2.5-Omni) time has only the fastest, with wavelengths up to 160 ids.

Here is what the code assigns to a four-token prefix, an image whose patches merge into a 2×2 grid, and three more tokens:

sequence indextokentimeheightwidth
0 to 3text0, 1, 2, 30, 1, 2, 30, 1, 2, 3
4image (row 0, col 0)444
5image (row 0, col 1)445
6image (row 1, col 0)454
7image (row 1, col 1)455
8 to 10text6, 7, 86, 7, 86, 7, 8

Text resumes at the largest id used plus one, so an image of 32 × 32 merged tokens (1,024 positions) advances the running id by only 32. The Qwen2-VL report names this as a benefit: it “reduces the value of position IDs for images and videos”. Of the 39 models whose code this site has cached, one outside Qwen’s releases uses M-RoPE in the backbone: Xiaomi-Robotics-1, by embedding Qwen3-VL.

Time from the clock

Qwen2-VL’s temporal id counts frames, so one clip sampled at 2 fps and at 8 fps appears to last different lengths. The Qwen2.5-VL report ties the id to absolute time.

TM-RoPE (time-aligned multimodal RoPE; Qwen2.5-Omni) puts audio on the same clock. Audio tokens get one id each, from the same start. A video’s jj-th temporal patch gets

pj=start+⌊j⋅s⋅r⌋,p_j = \text{start} + \lfloor j \cdot s \cdot r \rfloor ,

where ss is seconds per temporal patch and rr is ids per second. With s=1s = 1 and r=25r = 25 the patches get 0, 25, 50 while audio goes 0, 1, 2, so an audio token and a video frame from the same moment share a temporal id, to within one id. One id is 40 ms in Qwen2.5-Omni, 80 ms by Qwen3-Omni’s report (about 77 in its shipped config) and 160 ms in Qwen3.5-Omni’s report.

The Qwen3-VL report names two limits of this design, and the Qwen3.5-Omni report repeats them almost word for word: ids that are “excessively large and sparse” for long videos, and the need for “extensive and uniformly distributed sampling across various frame rates”. Neither report measures them.

We can say what they might mean, as our own reading. A rotary score sees only id differences, so the size of an id cannot matter, while the set of differences that occur can. Spacing patches gg ids apart is the same as multiplying every speed by gg, the mirror image of position interpolation in T4. A rotation sampled every gg ids can be told from a slower one only if it turns less than half a circle between samples. A pair with a wavelength under 2g2g ids fails that test and behaves as a slower pair, a different one at each frame rate, which would explain the need for data at many frame rates. With Qwen2.5-Omni’s block of time pairs and one patch a second at 40 ms an id, 10 of the 16 pairs fail and the longest wavelength is 6.4 seconds of video; with Qwen3.5-397B’s interleaved allocation (base 1e7, and no clock ids in that model), 2 of 11 would fail (computed). Qwen3-VL replaced the ids and interleaved the speeds in one release, so its results cannot separate the two causes. Huang et al. test allocation with absolute time switched off, and Qwen3-Omni’s code keeps clock ids and interleaves, with no comparison reported.

Our reading, not a measurement: what a rotary pair can tell apart when patches sit g ids apart.

At 25 ids a patch the fast pair turns -0.133 rad from patch to patch. Time pairs under 50 ids: 10 of 16 in Qwen2.5-Omni (blocks), 2 of 11 in Qwen3.5-397B (interleaved).

  • a patch, every g ids
Fast pair1 rad per id
π0−π
Slow pairone turn in 160 ids
π0−π
050100150200
id difference · vertical: rotation angle in radians, wrapped to (−π, π]
Figure 7.What a rotary pair can see when ids are spaced g apart. At g = 25, one patch a second at 40 ms an id, the fastest pair moves by -0.13 radians per patch, and 10 of Qwen2.5-Omni's 16 time pairs are under 2g ids, against 2 of 11 in an interleaved allocation. This is our reading; no report measures it.

Time as text

The Qwen2.5-VL authors had rejected textual timestamps. The Qwen3-VL report puts a string such as <3.0 seconds> in front of each temporal patch, after TimeMarker (Chen et al., 2024). The code for Qwen3.5-397B, Qwen3.6-35B and Qwen3.8-Flash-Next uses such strings: each temporal patch becomes its own span, so the “time” axis of M-RoPE holds the sequence position when reading video, and the time is in the text in front of each patch.

The Qwen3.5-Omni report describes both mechanisms. It says “we apply TM-RoPE”, lists the two limits, and says “we prepend each video or audio-video temporal patch with an explicit timestamp”. Its next paragraph, close to Qwen3-Omni’s, describes a “temporal ID for every 160 ms”. For “audio sequences” timestamps go in “at random intervals”, perhaps only in training. No code is public to show how the ids are used.

Other labs write timestamps too. The LFM3-Omni card says “each pair of frames is prefixed with its timestamp”, and MiMo’s vLLM processor writes MM:SS per temporal grid over flat 1-D positions.

Timestamps take context. At about 9 tokens each (assuming one per digit) and one per patch, they add about 1% to the 880 vision tokens per patch at 720p (computed), consistent with the report’s “slightly increases the context length”. At 6.25 audio tokens a second, one stamp a second would more than double the audio’s context, which may be why stamps are sparse there. The report gives no rate and no before-and-after numbers.

1 min at 2 fps, 40 ms ticks: frame index up to 59, clock to 1,500, timestamp strings about 410 tokens.

Frame indexid per temporal patchlargest id 59
60 temporal patches at 2 fps; at another sampling rate the same moment gets another id
Clock ids40 ms a tick, with audioreaches 1,500
video ticks 25 ids apart; coral comb below: one audio token per 40 ms, 1 in 21 drawn
Timestamp stringsposition ids run on410 tokens
grey: the string in front of each patch; coral: the patch; assumed: one token per digit and per '.', plus 3; at most 9 each
020 s40 s1 min
time in the video
Figure 8.Three ways of saying when. Double the sampling rate and the frame count changes; the clock ids keep their spacing and reach about 90,000 in an hour at 40 ms; the timestamp row spends tokens instead of id range.

Streaming

In a spoken conversation the model has little time to answer. Qwen’s report calls first-packet latency, the wait until the first audio comes back, “a critical factor affecting user experience”. An encoder that needs the whole clip cannot meet that.

Chunked prefill is the usual remedy. The encoder works in blocks and each block is prefilled as it arrives, so when the user stops only the last piece is left: 2-second audio blocks in Qwen2.5-Omni. In Qwen3-Omni’s AuT the attention window ranges from 1 to 8 seconds, and the block length is not given. In Qwen3-Omni the Thinker’s representations of each input chunk also prefill the Talker, so one turn loads two KV caches.

A causal encoder uses nothing later than the end of the token it is producing, so its delay floor is one token’s duration: 40 ms at 25 tokens a second, 160 ms at 6.25. LFM3’s encoder is “fully causal” and mixes sliding-window attention with short convolutions, which keeps the encoder’s memory bounded; the backbone’s KV cache still grows with every token.

A session that reuses its cached prefix needs tokens that stay fixed once written, which a causal encoder gives and a dynamic window may not (our reading).

Qwen3.5-Omni’s report gives these numbers, labelled “theoretical”, on an internal vLLM build with no named hardware. Input length, turn number and end-of-speech detection are not stated. Its predecessor’s table starts from the last input chunk (its “tail packet”); this report counts “from receiving the input stream” and does not say which packet.

Flash, 1 userFlash, 8 usersPlus, 1 userPlus, 8 users
First packet, audio input235 ms352 ms435 ms955 ms
First packet, video input426 ms1,625 ms651 ms1,980 ms
Thinker first token, video255 ms765 ms377 ms1,243 ms
Talker first chunk, video61 ms116 ms56 ms116 ms
Generation RTF0.1780.2570.1870.334

First packet is how fast the model answers. The real-time factor (RTF, generation time per second of audio) says whether it keeps up, and must stay below 1. The report’s overall row is a critical path and not a sum of the rows, and it warns against comparing Flash with Plus. Qwen3-Omni’s 234 ms is a sum of parts for a cold start, and its RTF adds the Thinker’s and the codec’s time, which Qwen3.5-Omni’s appears to leave out (our inference), so the generations do not compare. Moshi reports 160 ms theoretical and 200 ms in practice.

Qwen3.5-Omni lists “semantic interruption through native turn-taking intent recognition” with no mechanism. An interruption leaves text and frames in context that the user never heard, and a Thinker–Talker model has two contexts to repair (our reading).

Training order

Labs use “native” loosely, as X1 showed. Pre-training here means learning from a large corpus, mid-training from curated data, and post-training means instruction tuning and reinforcement learning. Xiaomi’s blog describes MiMo-V2.5 as text pre-training, then “projector warmup”, then multimodal pre-training, with encoders “pretrained in-house”. Qwen3.5-Omni is “natively pretrained in an omnimodal manner” and starts by locking an LLM initialized from Qwen3.5.

ModelOrderAudio’s share and notes
Qwen3.5-OmniStage 1: LLM frozen, encoders aligned. Stage 2: everything unfrozen, sequences of 32,768 tokens. Stage 3: sequences of 262,144 tokens, more long audio and videoStage 2 lists 1.99T audio tokens of 4.29T (46%, computed; the report rounds the total to “approximately 4 trillion”)
LFM2-AudioAlignment: connector trained on speech-recognition data, backbone and encoder frozen. Then everything unfrozen in mid-training, then post-trainingMid-training about 100B tokens: 62% text, 25% audio output frames, 13% audio input embeddings
MiMo-V2.6Text stage, then omni stage with pretrained ViT and audio encoderTokens by stage: Flash 26T + 22T; Pro 27T + 3T
Gemma 4 E2B/E4BEncoders frozen during pre-training
Gemma 4 12B“Trained from scratch”, encoder-freeStages not stated
InklingMultimodal components “trained from scratch”; post-training is a bootstrap SFT, then RLPre-training stages not stated

The Qwen3.5-Omni report says what listening through the language model costs after pre-training. The Thinker gets three post-training stages, “designed to preserve the model’s capabilities across all modalities without degradation”. The second exists because “a substantial gap remains between the quality of responses conditioned on audio queries and that of responses conditioned on text queries”, and it trains the answer to the spoken query to match the answer to the same query typed. The Talker has its own four-stage pipeline.

Comparison

DecisionDesignWhat it buysWhat it costsSeen in
Audio inMel + trained encoder6.25 to 25 tokens/s115M to 650M in memoryQwen, Gemma E2B/E4B, LFM2, LFM3
Audio inRaw projectionNo encoderBackbone must learn phase invariance; 25 tokens/sGemma 4 12B
Audio indMel8M weights, no encoder20 tokens/s, fixed quantizationInkling
Audio inTokenize, then patch6.25 tokens/s435M of encoders (V2.6)MiMo-V2.5, V2.6
Who speaksThinker–TalkerLarge model’s steps and context stay for textSecond model and pipeline; Talker does most stepsQwen2.5/3/3.5-Omni
Who speaksOne backboneOne model, one set of weightsAbout 16 positions per second of speech; codes in the training mixLFM2-Audio, LFM3
Time1-D positionsNothing to buildLayout and time learned from order or textMost models here
TimeM-RoPERow and column structure, smaller idsEach axis gets a third of the speeds; three ids per tokenQwen3.5/3.6/3.8, Xiaomi-Robotics-1
TimeClock idsShared time at no token costPer Qwen: sparse ids, data across frame rates (unmeasured)Qwen2.5-VL, Qwen2.5/3-Omni
TimeText timestampsRead like any other tokenAbout 1% more tokens at 720p (computed)Qwen3-VL, 3.5-397B, 3.5-Omni (with ids), LFM3-Omni, MiMo

Where the models sit

Of the 39 models whose code this site has cached, four have a working audio path: Inkling, MiMo-V2.5, and MiMo-V2.6-Flash-RL and -Pro-RL. A config key proves nothing by itself: Gemma 4 31B has audio_config: null. The other rows come from public repositories or reports.

ModelEvidenceAudio inTokens/s1 h of audio vs contextAudio outPositions
Qwen3.5-Omnireport onlymel + AuT6.2522,500 (8.6% of 256K)RVQ codec, TalkerTM-RoPE with text timestamps (as reported)
Qwen3-Omni-30Breport, codemel + AuT12.5 (13 in config)45,000–46,800 (69–71% of the config’s 65,536)16 code groups (config), TalkerTM-RoPE
LFM2-Audio-1.5Breport, cardmel + 17 FastConformer12.545,000 (137% of the card’s 32,768)Mimi, 8 codebooks of 2,048, RQ-Transformernot stated
LFM3-Audio-2B, Omni-2Bcode, cardmel + causal LFM12.5 (computed)45,000 (141% of the card’s 32K)LiquidCodec, 12 codes per frame1-D; Omni has text timestamps
Gemma 4 E2B, E4Breport, codemel + Conformer2530 s per clipnone1-D
Gemma 4 12Breport, coderaw 40 ms2530 s per clipnone1-D
Inklingcode, blogdMel2010 min per clip in vLLM; card says 20 minnonerelative-position term, no RoPE
MiMo-V2.5, V2.6code, reporttokenize + patch6.2522,500 (2.1% of 1M)none1-D, MM:SS strings

Audio generation is confined to two labs in this table, Qwen and Liquid.

Easy to get wrong

  • MTP means two things. In P1 it predicts several future tokens. In Qwen’s Talker it predicts the residual codebooks of the same frame, the depth half of a time-and-depth split.
  • The Qwen3.5-Omni report does not say it dropped TM-RoPE: it describes 160 ms ids and text timestamps side by side, and no public code confirms which the model uses. The replacement of clock ids is documented for Qwen3-VL and Qwen’s non-Omni line.
  • A Thinker–Talker split does not make speech cheap. The Talker steps at frame rate and, in the report’s theoretical one-user timings, takes more wall-clock time per second of speech than the Thinker. What stays free is the large model.
  • Qwen3-Omni’s 234 ms is a sum of parts, and Qwen3.5-Omni’s table is a critical path. Both are labelled theoretical, and a 235 ms headline and a 1,980 ms loaded video case come from one table.

Check yourself

  1. Ten minutes of speech arrives at Qwen3.5-Omni’s rate and at Gemma 4 12B’s. How many tokens does each cost, and what share of a 256K context (262,144) is that?
    Answer6.25×600=3,7506.25 \times 600 = 3{,}750 tokens, 1.4%, and 25×600=15,00025 \times 600 = 15{,}000 tokens, 5.7%.
  2. A codec runs at 25 frames a second with four codebooks of 1,024 entries. What is its bitrate, how many codes a second must be predicted, and how many entries would one flat codebook for a frame need?
    Answer25×4×10=1,00025 \times 4 \times 10 = 1{,}000 bits a second and 100 codes a second. A flat table for 40 bits needs 2402^{40} entries, against 4×1,024=4,0964 \times 1{,}024 = 4{,}096 for the four tables.
  3. In Qwen3.5’s M-RoPE an image’s merged grid is 3×3 and starts at running position 10. What are the ids of its first and last tokens on the three axes, and what id does the next text token get?
    AnswerFirst token: time 10, height 10, width 10. Last token: time 10, height 12, width 12. The largest id used is 12, so the next text token has id 13 on all axes.
  4. One lab has a 300B text model and wants it to speak. Another has a 2B model for on-device use. Which speaking design suits each, and what does each give up?
    AnswerThe 300B model suits a Thinker–Talker split, whose expensive steps and context stay on text, at the price of a second model to train, hold and schedule. The 2B model suits one backbone, giving up about 16 context positions per second of speech and sharing its weights between text and codes.

In the map

Methods

Taxonomy nodes

Further reading

  • Wang et al., Qwen2-VL (2024), arXiv:2409.12191: M-RoPE’s definition.
  • Xu et al., Qwen2.5-Omni (2025), arXiv:2503.20215: Thinker–Talker and TMRoPE with 2-second interleaving.
  • Xu et al., Qwen3-Omni (2025), arXiv:2509.17765: multi-codebook Talker with MTP, Talker and codec sizes, and the 234 ms theoretical first packet.
  • Qwen3-VL (2025), arXiv:2511.21631: why text timestamps replaced clock ids.
  • Qwen3.5-Omni (2026), arXiv:2604.15804: AuT, ARIA, first-packet tables, the training stages.
  • Huang et al. (2025), arXiv:2510.23095: how M-RoPE’s speeds should be shared among axes.
  • Liquid AI, LFM2 (2025), arXiv:2511.23404, §6: the full one-backbone recipe, including the loss weighting and the alignment phase.
  • Zeghidour et al., SoundStream (2021), arXiv:2107.03312: RVQ learned end to end, and why a flat codebook cannot work.
  • Défossez et al., Moshi (2024), arXiv:2410.00037: Mimi, RQ-Transformers for audio, and the delay ablations.
↑