A VLM-as-a-Judge metric for SVG generation, validated through human correlation studies, and an evaluation of frontier VLMs reveals significant performance gaps, positioning VectorGym as a rigorous framework for advancing visual code generation.
Juan A. Rodriguez, Haotian Zhang, Abhay Puri et al.· arXiv.org· 5 citations· ⚡1
The results suggest that skills and workflow memory can be useful in specific domains, but their apparent gains often vanish against a budget-matched actor, and that run-to-run variance materially affects outcomes and should be reported as a core evaluation criterion for online web agents.
Sina Hajimiri, Masih Aminbeidokhti, J. Dolz et al.· arXiv.org· 1 citation
PiSAs (Privacy in Shared Agentic systems), a benchmark for assessing unintentional leaks with dual CI annotations, enables direct measurement of cross-user spillage across agentic system components and interfaces, such as outputs, inter-agent communication, and memory.
Shubham Gupta, N. Sepahvand, Abhinav Kumar et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.