RefineSVG, a single-step closed-loop visual feedback framework that enables multimodal large language models (MLLMs) to perform high-fidelity image-to-SVG generation through self-correction, consistently outperforms existing baselines in reconstruction fidelity, structural accuracy, and code efficiency.
Abstract
We propose RefineSVG, a single-step closed-loop visual feedback framework that enables multimodal large language models (MLLMs) to perform high-fidelity image-to-SVG generation through self-correction. Existing MLLM-based approaches rely on single-pass open-loop inference, where the model receives visual input only once and must generate thousands of SVG code tokens without intermediate verification. This paradigm inevitably leads to geometric drift, error accumulation, and visual hallucination on complex images. RefineSVG overcomes this limitation by invoking an external rendering engine after an initial SVG generation pass to compare the rendered output against the target image. The comparison yields a multi-dimensional visual residual map (Diff-Map) that is fed back to the model as a ReAct-style correction signal, driving a targeted correction step. To support this render-observe-correct interaction, we further introduce an SVG-oriented semantic vocabulary that compresses token sequences by over 52%. A progressive training pipeline spanning supervised fine-tuning, rejection-sampling cold-start data construction, and end-to-end agentic reinforcement learning aligns the model with closed-loop visual correction. Extensive experiments show that RefineSVG consistently outperforms existing baselines in reconstruction fidelity, structural accuracy, and code efficiency.Code is available at https://github.com/liuxiaobo66/RefineSVG.
Unigen-AR, a framework that pairs a general-purpose multi-modal language model (MLLM) with an efficient next-scale visual auto-regressive (VAR) decoder, is proposed, establishing visual auto-regressive modeling as a compelling and efficient backbone for unified visual generation.
A training-free, interpretable mid-generation correction framework that integrates multimodal large language model (MLLM) feedback directly into the diffusion sampling loop and achieves diffusion trajectory correction by injecting semantic evaluation signals during video synthesis, enabling the model to optimize the generated content through continuous self-reflection.
Junhao Chen, Zheqi Lv, Keting Yin et al.· 0 citations
Transformer-based models have significantly advanced image captioning through self-attention mechanisms and parallel computation. However, existing methods typically adopt teacher-forcing strategies during training by conditioning the decoder exclusively on ground-truth tokens, whereas at inference, captions are generated autoregressively based solely on previously predicted tokens. Such discrepancy between training and inference conditions leads to a progressive accumulation of prediction errors, resulting in captions that deviate significantly from the visual content. While scheduled sampling strategies mitigate this issue, directly integrating them disrupts Transformer parallelism and overlooks visual saliency differences. To tackle these limitations, we present VMFormer, a multi-stage decoding Transformer featuring a Visual-Aware Scheduled Sampling (VASS) module that bridges this gap through two key innovations: 1) A two-stage decoding scheme where an initial self-study stage generates candidate tokens, followed by a hybrid stage dynamically blending ground-truth references with predictions via a visual clues controlled gate. 2) A cognitive-inspired prioritization mechanism that retains visual keywords (nouns/verbs/attributes) in early training phases before transitioning to linguistic refinements, mirroring human captioning patterns. Crucially, the VASS module preserves the parallel computational strengths of Transformer architectures and is designed as a plug-and-play component, readily adaptable to Transformer-based captioning models. Experiments on challenging MSCOCO achieve 142.2 CIDEr. We extend VMFormer to video captioning and demonstrate consistent improvements on MSRVTT and MSVD datasets.
Yuchen Ren, Xin Chen, Hongrui Yuan et al.· IEEE Transactions on Image P...· 0 citations
Swift-Image achieves leading aggregate performance among evaluated open-source models with only 6B parameters and 243K GPU training hours; the compressed 3B model incurs nearly no loss, while few-step distillation further improves aggregate editing performance with substantially fewer sampling steps.
Taihang Hu, Zhaowen Wang, Zuan Gao et al.· 0 citations
This work proposes Visual In-context Editing, a new paradigm elevating video editing from textual instructions to multi-modal visual guidance encompassing single image, image pair, and video pair, and curates VicEdit-400K, the first large-scale dataset for visual in-context video editing.
Yu-Ji Wang, Teng Hu, Yuheng Chen et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
