vLLM

10 documents read from this publisher.

Qwen3.8-Flash-Next on vLLM — Serve command for H200, H100, GB200 NVL4, GB300 NVL4, MI355X, RTX Pro 6000 4x, DGX Station (GB300)

software documentation · vendor docs · 2026-09-17 · 15 techniques

Mixture of Experts Multi-Token Prediction YaRN Gated DeltaNet Gated Residual N-gram embedding host-memory offload and prefetch N-gram embedding lookup Qwen Sparse Attention Tensor parallelism (degree 4)Expert parallelism Gated DeltaNet and Qwen Sparse Attention hybrid Limit maximum concurrent sequences to 256 PLE CPU offload for N-gram embeddings Tensor and Expert Parallelism with Degree Eight Tiered CPU and filesystem offloading

zai-org/GLM-5.3 — 743B / 39B active · MOE · 1024K ctx

software documentation · vendor docs · 2026-09-10 · 9 techniques

Configurable reasoning effort Multi-Token Prediction FP8 KV-cache quantization FP8 Expert Parallelism NVFP4 ModelOpt re-quantization AITER Disable prefix caching for benchmarking Max sequences tuning

zai-org/GLM-5.3-Flash — 321B / 18B active · MOE · 1024K ctx

software documentation · vendor docs · 2026-09-05 · 16 techniques

Mixture of Experts Configurable reasoning effort Multi-Token Prediction FP8 KV-cache quantization Kimi Delta Attention Prefill-decode disaggregation Token-level expert routing Always-on thinking mode Chat-template image/video placeholder expansion Identical cache-layout pinning across prefill and decode pools MXFP8 block-scale regrouping at load NoPE sparse multi-head latent attention Reasoning-effort resolution and system-prompt injection Reusing the image token for video frames ROCm AITER sparse MLA attention backend Round-robin routing for prefill-decode disaggregation

tencent/Hy4-preview — 770B / 49B active · MOE · 1024K ctx

software documentation · vendor docs · 2026-08-28 · 14 techniques

Mixture of Experts Configurable reasoning effort Multi-Token Prediction FP8 Top-8 expert routing Automatic tool choice Gated DeepSeek Sparse Attention Identity Hyper-Connections IndexCache MoE with routed and shared experts Tensor parallelism (degree 8)FLASHMLA_SPARSE Hy v4 tool-call parser Reasoning parser hy_v4

thinkingmachines/Inkling — 975B / 41B active · MOE · 1024K ctx

software documentation · vendor docs · 2026-07-14 · 8 techniques

Multi-Token Prediction NVFP4 quantization Relative attention FA4 sheared-bias attention kernel Fused reduce-scatter/all-gather collectives MoE with Expert Sinks SConv-Aware Tensor-Parallel Sharding Short convolution (sconv) modules

tencent/Hy3 — 295B / 21B active · MOE · 256K ctx

software documentation · vendor docs · 2026-07-06 · 15 techniques

Mixture of Experts Multi-Token Prediction Speculative decoding FP8 KV-cache quantization Grouped-query attention Shared Experts AITER fused-MoE backend AITER Linear backend AITER MHA backend AITER RMSNorm backend HPC-Ops attention backend HPC-Ops fused MoE backend Routed experts TensorRT-LLM all-reduce backend Triton MoE kernel

Qwen/Qwen3.6-27B — 27B · DENSE · 256K ctx

software documentation · vendor docs · 2026-07-02 · 6 techniques

Multi-Token Prediction YaRN Gated DeltaNet Disabling reasoning via chat-template configuration CUDA Graph capture size reduction NVFP4 ModelOpt re-quantization

Qwen/Qwen3.6-35B-A3B — 35B / 3B active · MOE · 256K ctx

software documentation · vendor docs · 2026-06-03 · 18 techniques

Multi-Token Prediction YaRN FP8 KV-cache quantization FP8 Chunked prefill Tensor Parallelism Disabling reasoning via chat-template configuration NVFP4 quantization Prefix caching Automatic tool choice Asynchronous scheduling CUDA Graph capture size reduction Gated DeltaNet MoE fastsafetensors Mamba prefix caching in align mode Marlin MoE backend Qwen3 reasoning parser Qwen3 XML tool-call parser

XiaomiMiMo/MiMo-V2.5 — 311B / 15B active · MOE · 1024K ctx

software documentation · vendor docs · 2026-04-27 · 8 techniques

Mixture of Experts Multi-Token Prediction Hybrid Attention Speculative decoding Top-8 expert routing Full attention Block-wise E4M3 FP8 weight quantization SWA-128

Qwen/Qwen3.5-397B-A17B — 397B / 17B active · MOE · 256K ctx

software documentation · vendor docs · 2026-04-16 · 12 techniques

YaRN FP8 Expert Parallelism Disabling reasoning via chat-template configuration NVFP4 quantization Prefix caching Language-model-only serving mode Data-parallel vision encoding MTP-1 speculative decoding reducing CUDA graph capture size to fit Mamba cache shared-memory caching of preprocessed multimodal inputs Tool calling