SourcesOpenVLA
publisher · 3 sources
OpenVLA
3 sources read from this publisher, newest first, with the methods each one describes.
Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success
research paper · technical report · 2025-02-27 · 22 methods
Low-Rank AdaptationAction chunkingBidirectional attentionEnd-to-end fine-tuningFeature-wise Linear Modulation (FiLM)Imitation learningParallel action decodingSpatially shared FiLM modulation across vision-transformer patch embeddingsBlock-specific FiLM projectorsConditional denoising diffusion for action predictionContinuous action representationEmpty action embeddings with distinct positional encodingsFiLMFine-tuning termination at a normalized mean L1 loss thresholdL1 regression for continuous action predictionL1 regression objective for continuous actionsL1 regression's median-mode action fittingLearning-rate decayMulti-view image inputOptimized Fine-TuningRobot proprioceptive-state projection into language embedding spaceShared-sequence multimodal input projection
openvla/openvla-7b · Hugging Face
first party release · model card · 2024-06-13 · 7 methods
OpenVLA: An Open-Source Vision-Language-Action Model
research paper · technical report · 2024-06-13 · 47 methods
Speculative decodingLow-Rank AdaptationFully Sharded Data ParallelAction chunkingQuantizationAction-token-only cross-entropy lossFull-model fine-tuningBF16 inferenceFiltering no-op actions and unsuccessful demonstrationsFlashAttentionFreeze the vision encoder during fine-tuningParameter-efficient fine-tuningPer-dimension 256-bin robot-action discretization4-bit quantization8-bit and 4-bit quantization8-bit quantizationAutomatic mixed precisionBlocking control that waits for each action to finish before predicting the nextDropping a poorly fitting dataset from the training mixtureDynamic action queryingFine-tuning a vision-language model to predict robot actionsFine-tuning all model parameters including the vision backboneFine-tuning on BridgeData V2 onlyFirst-transition removal from demonstrationsFixed learning rateFused SigLIP-DINOv2 vision encoderLast-layer and token-embedding fine-tuningLearning-rate warmupMulti-epoch dataset trainingMulti-resolution visual feature fusionOcto dataset-mixture weightingOpen X-Embodiment dataset mixturePatch-as-token vision-language architectureQuantile-bounded action discretizationQuantized and reduced-precision inferenceRemote VLA inference serverRepresenting robot actions as language-model tokensReusing least-used tokenizer vocabulary entries for action tokensRobot dataset filtering by camera view and control typeSandwich fine-tuning