Liquid AI
LFM3-1.7B
What LFM3-1.7B’s own documents say it is built from — every method with the sentence that describes it.
- Parameters
- 1.73B dense
- Context
- 256K
- Attention
- conv hybrid13 gated short-conv (k=7) : 5 sliding-window (256) : 6 global GQA layers
- Experts
- dense
- Input
- text
- Licence
- LFM Open License v1.0
- Released
- 2026-10-27
Against the consensus recipe
Its documents state 4 of the 18 methods the field agrees on.
Data curation
- Benchmark decontaminationnot in its documents
- Synthetic data generationnot in its documents
- Corpus deduplicationnot in its documents
Training objective
- Multi-Token Predictionnot in its documents
- Knowledge insulationnot in its documents
- Quantization-aware distillationnot in its documents
Model architecture
- Mixture of expertscore
- Multi-Token Predictionnot in its documents
- Grouped-query attentioncore
Optimization
- Muonnot in its documents
- Expert parallelismnot in its documents
- Quantization-Aware Trainingnot in its documents
Post-training
- Supervised fine-tuningcore
- Asynchronous reinforcement learningnot in its documents
- Multi-Teacher On-Policy Distillationused
Inference & serving
- Thinking modenot in its documents
- Multi-Token Predictionnot in its documents
- FP8not in its documents
Checked in its code
Its code shows 7 of the 19 architecture features checked — 4 of them are not stated in its documents.
Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.
Read from Hugging Face Transformers (modeling_lfm3.py, configuration_lfm3.py) and config.json at revision 0dc4268b56, by anthropic/claude-sonnet-5 on 2026-11-07.
| Feature | Code | Its documents | Verdict |
|---|---|---|---|
| Grouped-query attention | in its code | adopted | 16 query heads share 4 KV heads via repeat_kv in every attention layer (full and sliding). show evidencenum_attention_heads = 16num_key_value_heads = 4 |
| Multi-head latent attention | not in its code | not stated | No low-rank KV latent projection exists; attention uses standard per-head q/k/v projections sized by num_key_value_heads. |
| Sliding-window attention | in its code | adopted | Layers typed sliding_attention set self.sliding_window = config.sliding_window = 256 and a sliding window mask is built for them. show evidencesliding_window = 256layer_types = ["conv", "conv", "full_attention", "conv", "sliding_attention", "conv", "full… |
| Interleaved sliding-window and global attention code only | in its code | not stated | layer_types interleaves full_attention and sliding_attention layers (plus conv layers) in a fixed pattern, with separate mask builders per… show evidencelayer_types interleaves full_attention and sliding_attention layers (plus conv layers) in a fixed pattern, with separate mask builders per type. layer_types = ["conv", "conv", "full_attention", "conv", "sliding_attention", "conv", "full… |
| Indexer-selected sparse attention (DeepSeek Sparse Attention) | not in its code | not stated | No indexer scoring module or top-k token selection mechanism exists in the provided code. |
| Linear-attention or state-space layers alongside full attention | not in its code | not stated | Non-attention layers use a short causal convolution (Lfm3ShortConv), not a linear-attention or SSM recurrent mixer, alongside full/sliding… show evidenceNon-attention layers use a short causal convolution (Lfm3ShortConv), not a linear-attention or SSM recurrent mixer, alongside full/sliding softmax attention. layer_types = ["conv", "conv", "full_attention", "conv", "sliding_attention", "conv", "full… |
| Gated DeltaNet layers | not in its code | not stated | No delta-rule recurrent state update with a decay gate is implemented; the non-attention layers are plain causal convolutions. |
| Mamba-2 layers | not in its code | not stated | No Mamba-2 SSD selective state-space block is implemented; non-attention layers are short causal convolutions (Lfm3ShortConv), not SSM. |
| Learnable attention sink | not in its code | not stated | No learned per-head sink logit is added to the attention softmax denominator anywhere in the code. |
| Gated attention output code only | in its code | not stated | use_attention_gate is true, so g_proj produces a per-head sigmoid gate that multiplies the attention output before the output projection. show evidencemodel_type = "lfm3" |
| QK normalization code only | in its code | not stated | Queries and keys are each passed through an RMSNorm over the head dimension before RoPE and the attention dot product. show evidence |
| Partial RoPE | in its code | adopted | full_attention layers use partial_rotary_factor 0.5, applying RoPE to only half of the head dimension. |
| YaRN RoPE scaling | not in its code | not stated | rope_parameters entries specify rope_type "default" for both layer types, not YaRN, and no YaRN scaling code path is exercised. |
| Layers without positional encoding (NoPE) | not in its code | not stated | All attention layer types (full_attention, sliding_attention) have an entry in rope_parameters and receive rotary embeddings; only… show evidenceAll attention layer types (full_attention, sliding_attention) have an entry in rope_parameters and receive rotary embeddings; only non-attention conv layers lack RoPE, which does not count. rope_parameters = {"full_attention": {"partial_rotary_factor": 0.5, "rope_theta": 1000000.0, "r… |
| Mixture of experts documents disagree | not in its code | adopted | No router or expert MLP modules exist; Lfm3MLP is a single dense SwiGLU feed-forward applied in every decoder layer. show evidence |
| Shared expert | not in its code | not stated | There is no MoE mechanism at all, so there is no shared-expert path either. |
| Auxiliary-loss-free load balancing (selection bias) | not in its code | not stated | No router or expert-selection bias exists since the model has no mixture-of-experts layers. |
| Multi-token prediction layers | not in its code | not stated | No extra next-next-token prediction modules are declared in config or built in the modeling code. |
| SwiGLU feed-forward code only | in its code | not stated | Lfm3MLP computes SiLU(w1(x)) * w3(x) then projects with w2, a SwiGLU feed-forward used by every layer. show evidence |
filed at the root 1
“Built on a new hybrid architecture”
token mixer 7
“uses gated short convolutions for most layers and a small number of GQA blocks for global context”
“each gated short convolution block applies input-dependent multiplicative gating around a depthwise short convolution”
“The convolutions also use a kernel size of 7, up from 3 in LFM2.”
softmax attention 4
global attention 1
“LFM3 combines three operators: gated short convolutions, sliding-window attention (SWA), and global attention.”
sliding window attention 1
“sliding-window attention (SWA) gave the best quality–speed tradeoff.”
grouped-query attention 2
“interleaved with a small minority of GQA blocks”
“a small number of GQA blocks for global context”
channel mixer 3
mixture of experts 3
“replace dense MLPs with sparse MoE MLPs in most layers”
“the first two layers remain dense while all subsequent layers include an MoE block”
expert routing 1
“selects the Top-k experts per token with a normalized sigmoid router and adaptive routing biases for load balancing”
positional encoding 2
“Global attention layers rotate only the highest-frequency half of each head and leave the low-frequency half unrotated”
“SWA layers keep full RoPE.”
normalization & residual 2
“A sigmoid gate scales each head’s output, so a head can switch off when it has nothing to add.”
“each block normalizes both its input and its output to keep activations well-scaled”
prediction head 1
“a small residual MLP maps the hidden states of eight action-query tokens to an eight-step chunk of continuous robot actions in a single…”
multimodal architecture 20
“is fully causal, so the model can prefill while you speak”
“it combines convolution and attention and uses 3x less compute than the SigLIP2 encoder”
“audio generation is implemented through a discrete Residual Vector Quantization (RVQ) code path”
“Proprioception is provided both as discretized text and through an embodiment-specific learned projection”
“Specifically, the model predicts 8-codebook Mimi RVQ codes”
“it embeds each pair of sampled frames into one temporal slice, halving the temporal sequence length”
“We train LFM2-VL in three stages, mirroring the language-only pipeline and introducing multimodal alignment at appropriate points.”
“All LFM2-VL variants retain the decoder-only LFM2 backbone and attach a SigLIP2 image encoder through a small connector that maps visual…”
“In the alignment phase, we use ASR data to train the MLP connector, while keeping the LFM2 backbone and encoder frozen.”
“The first stage focuses on aligning the visual and textual embedding spaces while keeping the image encoder and language backbone frozen”
“For higher-resolution inputs, we switch to dynamic tiling.”
“In our implementation, we generate 6 text tokens followed by 12 audio tokens and repeat until all text tokens are exhausted.”
“we adopt parallel action prediction inspired by OpenVLA-OFT and StarVLA-α”
“This connector first applies a PixelUnshuffle (Shi et al., 2016) operation that lowers the number of visual tokens”
“We only repurpose 2,068 vocabulary tokens; the architecture is otherwise unchanged.”
“Each decision consists of five tokens: two spatial/control tokens and three hierarchically quantized codes encoding a ten-waypoint path”
“adds a SigLIP2-based vision encoder with dynamic tiling and token reduction”
“The manipulation VLA maps images, language, and robot state to eight-step chunks of continuous actions in a single forward pass”
“The navigator follows route descriptions using recent RGB observations, predicting spatial targets and a waypoint trajectory in just five…”
“We optionally append a single thumbnail frame, processed through the same single-frame pipeline as above, to provide a coarse global…”
1 more mentioned or evaluated
multimodal architecture “LFM3-VL-2B reads a route description and a short history of RGB frames to predict navigation targets and residual-quantized trajectory…”
filed at the root 1
“The detokenizer is trained separately as a GAN”
language modelling objective 1
“LFM3-1.7B-Base was pre-trained on 22T tokens.”
distillation objective 6
“decomposing the KL via the chain rule into (i) a binary term that matches the total probability mass assigned to the Top-K set, and (ii) a…”
“We apply temperature only to the conditional Top-K distributions inside”
“A tempered, decoupled Top-K knowledge distillation objective reduces storage and bandwidth when distilling from a larger teacher”
“We train LFM2-ColBERT-350M using knowledge distillation through the PyLate framework (Chaffin and Sourty, 2024), leveraging pre-computed…”
“We employ the distillation loss implemented in PyLate, which minimizes the mean squared error between student and teacher relevance scores.”
“The PyLate framework applies min-max normalization to teacher scores to ensure numerical stability and consistent gradient magnitudes…”
auxiliary loss 1
“We combine the 8 audio codebook losses into a single via a weighted average.”
1 more mentioned or evaluated
distillation objective “we also consider a teacher-weighted surrogate”
filed at the root 1
“Pre-training: two stages build LFM3-1.7B-Base on 22T tokens in total.”
training stability 1
“we apply 10% input dropout on embeddings during training only, while avoiding internal dropout layers to preserve inference latency”
training parallelism 1
“The distributed training infrastructure leverages ZeRO-2 sharding with strided activation checkpointing across 8 H100-80GB GPUs per…”
training runtime 1
“the highest-performing checkpoints are combined with a simple linear merge in weight space”
3 more mentioned or evaluated
“For each layer, parameters are ranked by absolute magnitude, and dropout probabilities are assigned inversely proportional to rank”
“magnitude-based sparsification retains only the top of parameters by absolute value in each task vector”
“sign election resolves conflicts by computing the majority sign for each parameter position across all task vectors”
data sourcing 1
“across training stages we integrate vision-language data translated into several target languages”
data filtering 6
“We implement a comprehensive multi-stage filtering process to ensure dataset quality and prevent evaluation contamination.”
“Some large-scale image datasets used at this stage are filtered for safe-for-work content using automated filters and LLM-based screening”
“we perform decontamination through exact n-gram matching against evaluation benchmarks, with additional semantic similarity filtering to…”
“we perform human evaluation on a subset of each candidate dataset”
“Filtering combines rule-based validation (e.g., duration constraints, transcription alignment, and pattern-based filtering) with…”
“Stylistic filtering uses rule-based methods to eliminate samples with overused phrases, clichés, and undesirable patterns.”
deduplication 3
“We perform exact deduplication through direct matching”
“apply near-duplicate detection using CMinHash locality-sensitive hashing to catch similar but non-identical content”
“Semantic deduplication proved most impactful, using sentence embeddings with a high similarity threshold to remove semantically similar…”
synthetic data 9
“pipelines specialized for turning existing text SFT datasets into audio conversations”
“To improve mid-training data quality, we recaption existing open-source image datasets”
“The dataset construction focuses on in-distribution samples and combines on-policy generation with skill-specific refinement through…”
“A large focus on synthetic conversation data is employed to generate a high volume of multi-turn audio conversations”
“This involves the synthetic generation of speech cues and conversational behaviors such as intonation patterns, pauses, filler words”
“generate additional VQA samples specifically targeted at visual understanding, including fine-grained reasoning about objects, layouts…”
“pipelines that generate audio conversations following a pre-determined theme using randomly sampled seed specifications”
“We incorporate additional document OCR data with visual augmentations such as blur, noise, and geometric distortions to make the OCR…”
“Additionally, we sample user voices via voice-cloning techniques to teach the model a wide distribution of user voice characteristics”
data mixture & curriculum 5
“The final 3.5T tokens are a mid-training stage on curated data with a larger share of agentic traces and tool calls”
“We implement a data-driven curriculum learning strategy across the entire dataset to optimize the training progression from simpler to…”
“the mix adds more scientific papers, repository-level code, math, and a wider range of knowledge sources”
“We develop the final SFT composition through an iterative process targeting individual capabilities in isolation before combining them…”
“We apply a data annealing schedule that starts with 100% text-only data and, over the first 5% of steps, linearly decreases its fraction…”
tokenization 2
“LFM3 uses this ChatML format, generated from the checkpoint's chat template”
“LFM3 uses the same 128K vocabulary as LFM2.5-8B-A1B, built to tokenize non-Latin scripts more efficiently.”
1 more mentioned or evaluated
data mixture & curriculum “Ablations showed that the annealing data schedule removes the need for connector pre-training”
filed at the root 3
“Training proceeds in three phases: alignment, mid-training, and post-training.”
“A three-stage post-training recipe including supervised fine-tuning, length-normalized direct preference alignment, and parameter-space…”
“through continued pre-training or full post-training via TRL”
supervised fine-tuning 10
“Large general-purpose mixture to provide good capabilities on downstream tasks and teach the chat template.”
“the whole network is fine-tuned end to end.”
“we systematically integrate multilingual samples across most task categories”
“Paired with a fine-tuned d1-2B skill monitor, we can further double recovery success”
“The final stage performs supervised fine-tuning on multimodal instruction-following data”
“A rank-32 LoRA adapter was trained on about 40,000 samples”
“we fine-tuned LFM3-VL-2B to find and classify defects in drone photos of wind turbine blades.”
“adapt the model to your domains, formats, and tools via quick single-GPU fine-tuning”
“this can be achieved by continuing to train the projection layer while keeping the LFM2 backbone frozen”
“it can be fine-tuned locally”
reinforcement learning algorithm 2
“Post-training from this checkpoint (SFT, mixed RL, multi-domain on-policy distillation, and agentic RL) produces LFM3-1.7B.”
“Training used about 3,000 public photos (WTBs2025) with SFT and GRPO for a few hours on one A100.”
reward modelling 1
“A length penalty in the reward keeps answers concise.”
preference optimization 3
“we combine a family of length-normalized direct alignment objectives with an offline dataset”
“Enabling efficient preference optimization through direct alignment on offline data including on-policy samples generated from the SFT…”
“For instruction following data, we further refine the chosen responses via Contrastive Learning from AI Revisions (CLAIR).”
policy distillation 1
“six specialist RL teachers (code, math, tool use, long context, agentic work, and enterprise workflows) are distilled back into a single…”
agentic post-training 2
“Post-training from this checkpoint (SFT, mixed RL, multi-domain on-policy distillation, and agentic RL) produces LFM3-1.7B.”
“Agentic training: post-trained for agentic harnesses such as Hermes Agent, Pi, and OpenClaw”
mid-training & continual pretraining 6
“This stage also extends the context window to 256K tokens.”
“The last 3.5T tokens use curated, higher-quality data with a larger share of agentic traces and tool calls.”
“Here, we unfreeze the entire model and train on a large-scale mixture of text and audio datasets.”
“we move these datasets to the mid-training stage instead of using them in SFT”
“After language mid-training, we introduce a multimodal mid-training stage that jointly optimizes the language backbone, pre-trained…”
“a starting point for continued pre-training and your own post-training”
filed at the root 1
“In interleaved generation mode, LFM2-Audio generates the assistant response in both text and audio, with a predetermined interleaved…”
decoding strategy 2
“Generation: Greedy decoding, max 8192 tokens.”
“General (chat, tool use, agents, coding): temperature=0.1,top_k=50, repetition_penalty=1.1”
reasoning control 1
“Reasoning is enabled by default. You can set enable_thinking=False to disable thinking traces”
context management 3
“Reasoning from turns before the latest user message is removed unless preserve_thinking=True.”
“Automatically truncates prompts exceeding model’s max context length.”
“Reducing the token budget introduces minimal degradation on general vision benchmarks”
agentic scaffolding 5
“it grounds objects by pointing, measures them with depth tools, invokes motion planners, and checks outcomes in subsequent camera images”
“Execute the call in your application. Add the parsed assistant call to history as a tool_calls entry, then append the result as a tool…”
“In the system prompt, tell the model when to search:”
“The template formats tool definitions and results automatically.”
“Use retrieval or search for knowledge-heavy tasks.”
evaluation 10
filed at the root 5
“Implementation: 5-shot by default (uses first 5 examples from dev set per subject).”
“5-shot by default using Chain-of-Thought (CoT) examples from validation set”
“Loose score is used because of low performance with strict scoring with Gemma 3 models.”
“5-shot examples randomly sampled from train set.”
“Each question evaluated 10 times with different answer orderings to reduce position bias.”
evaluation harness 3
“The evaluation scores for the text-only models come from a custom internal harness.”
“Custom parser with priority order:”
“we build a custom evaluation harness with robust parsing logic”
judge 2
“We then score individual responses via an LLM jury, selecting the highest scored sample as the chosen response and the lowest scored…”
“we favor on-policy responses over instruction / preference samples in case of a tie to mitigate distribution shift”
other 15
filed at the root 9
“We explicitly search this design space with a hardware-in-the-loop search procedure”
“Following the ColBERT framework (Khattab and Zaharia, 2020), we compute similarity between a query and document using the MaxSim operator”
“Systematic evaluation and combination of candidate models to increase robustness and optimize performance.”
“separates continuous audio input from discrete audio output”
“The remaining candidates are ranked by hypervolume improvement on the quality–latency–memory Pareto frontier.”
“we instead train a compact, LFM2-style detokenizer optimized for edge deployment”
“We frame architecture selection as a Pareto optimization over three axes”
“using both a multi-period discriminator (Kong et al., 2020) and a multi-resolution discriminator (Jang et al., 2021)”
“using both a multi-period discriminator (Kong et al., 2020) and a multi-resolution discriminator (Jang et al., 2021)”
6 more mentioned or evaluated
“The techniques we test (model soup, task arithmetic, TIES-Merging, DARE, and DELLA) are described in detail in Appendix B.”
“The techniques we test (model soup, task arithmetic, TIES-Merging, DARE, and DELLA) are described in detail in Appendix B.”
“the disjoint merge averages only those parameters that agree with the elected majority sign”
“The techniques we test (model soup, task arithmetic, TIES-Merging, DARE, and DELLA) are described in detail in Appendix B.”
“The techniques we test (model soup, task arithmetic, TIES-Merging, DARE, and DELLA) are described in detail in Appendix B.”
“The techniques we test (model soup, task arithmetic, TIES-Merging, DARE, and DELLA) are described in detail in Appendix B.”
unfiled 18
“based on our new tri-hybrid architecture”
“We extend the LFM2 backbone to information retrieval through LFM2-ColBERT-350M, a late interaction retriever optimized for multilingual…”
“LiquidCodec, our new audio tokenizer, turns generated tokens into speech at about 87x real time on a single CPU thread on Apple M4 Max and…”
“a minimal hybrid suffices: gated short convolutions for most layers plus a small minority of grouped-query attention (GQA) layers”
“LFM2 follows this general principle of hybridization, but adopts a minimal hybrid tuned for devices”
“We pre-train LFM2 using a multi-stage pipeline designed for small, on-device models.”
“Instead, we use a much smaller RQ-Transformer (Lee et al., 2022), similar to the one used in Moshi (Défossez et al., 2024), as a decoder…”
“a new tri-hybrid architecture and training recipe allow LFM3-1.7B to outperform LFM2.5-2.6B”
“Of its 24 layers, 13 are convolutions, 5 are SWA with a 256-token window, and 6 are global attention.”
“13 gated short convolutions, 5 sliding-window attention (SWA) layers with a 256-token window, and 6 global attention layers”
“Your harness implements and runs the search.”
“Jointly trained across 15 embodiments”
“All multilingual benchmarks were translated from English into Arabic, Chinese, French, German, Italian, Japanese, Korean, Portuguese, and…”
“For images in the single-frame regime, we first resize the input so that the SigLIP2 encoder sees a target range of patch counts.”
“In sequential generation mode, LFM2-Audio can change its own generation state by generating special control tokens.”
2 more mentioned or evaluated
“We implement DARE in combination with both standard task arithmetic (DARE-linear)”
“with TIES-style consensus (DARE-TIES)”
Not used 1
“we do not employ explicit grounding supervision such as bounding boxes or point coordinates”
Sources
The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.
| Source | Kind | Publisher | Status |
|---|---|---|---|
| liquid.ai/blog/lfm3 | official blog | Liquid AI | read |
| huggingface.co/blog/LiquidAI/lfm3 | official blog | Liquid AI | read |
| huggingface.co/LiquidAI/LFM3-1.7B | model card | Liquid AI | read |
| huggingface.co/LiquidAI/LFM3-1.7B-Base | model card | Liquid AI | read |
| arxiv.org/abs/2511.23404 | technical report | Liquid AI | read |