Skip to content
Conference Open access

Aligning Flow-Matching Vision-Language-Action Models via Advantage-Guided Action Preference Optimization

Oct 2026 · International Conference on Information and Software Technologies · pp. 252-257 · 0 citations · 35 references

Abstract

Deploying robust manipulation policies on robotic arms with lightweight Vision-Language-Action (VLA) models remains challenging. Teleoperated demonstrations often contain human hesitation and suboptimal corrections, which Supervised Fine-Tuning (SFT) inadvertently mimics rather than learning decisive control. In this paper, we propose the Advantage-Guided Flow-Matching Action Preference Optimization (AFAPO), an online fine-tuning algorithm that iteratively refines a continuous VLA using successful human-in-loop trajectories. First, we train a distributional value function on mixed interaction data to extract advantage estimates. This dynamically assigns binary preference labels to state-action pairs, filtering out teleoperation noise without manual annotation. Second, we introduce a tractable preference objective for continuous flow-matching policies by approximating the intractable log-likelihood ratio via the Evidence Lower Bound (ELBO), which reduces the implicit reward to the difference in L2 denoising errors between the current and reference models. Combined with the Kahneman-Tversky Optimization (KTO), this yields a single-sample preference loss. Validated on a task of placing earplugs into the box using a SO-101 robotic arm, AFAPO achieves 85% success rate which shows a significant improvement and reduces completion time compared to plain SFT and heuristic baselines.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.