Skip to content
Preprint

What Keeps Agent Skills from Being Reusable? Evidence from 138K SKILL.md Files

Aug 2026 · 0 citations · 19 references
Computer Science

TL;DR

A deterministic routing stress test over 20,000 skills shows the functional impact: skills with valid routing metadata are retrieved more reliably from startup descriptions than skills with routing defects, while AI-marked skills show more safety and portability problems.

Abstract

Under the current standard, Agent Skills are SKILL.md files that combine instructions with supporting resources, enabling Large Language Model (LLM) agents to reuse procedures beyond a single conversation. Yet many public skills appear to originate from a single task, repository, or conversation, even when they are shared as reusable components. We analyze this gap across 138,133 public SKILL.md files from 20,556 repositories using a two-tier defect taxonomy grounded in the official specification and best-practice guidance. We find that 91.8% of skills contain at least one detected defect, with stable estimates across lenient and strict thresholds (88.8-94.6%). The dominant failures are ordinary packaging problems rather than exotic attacks: weak routing metadata, bloated or non-actionable bodies, and poor resource organization. A deterministic routing stress test over 20,000 skills shows the functional impact: skills with valid routing metadata are retrieved more reliably from startup descriptions than skills with routing defects. Defect rates vary by platform and provenance: specification-aware skills contain fewer defects, while AI-marked skills show more safety and portability problems. Lightweight enforcement and repair experiments support a quality-assured generation workflow combining spec-aware prompting, lightweight linting, automated repair, and safety gating.

View source

Similar papers

Review Aug 2026

Evaluating Skills, Not Just Agents: Agentic Continuous Evaluation of Skills

ACES (Agentic Continuous Evaluation of Skills), a repository-native framework for evaluating skills and product capability packages as executable agent artifacts, is presented.

Christopher Kevin, Narendran Raghavan, J. Puget et al. · 1 citation · ⚡1
Preprint Aug 2026

Signal or Noise? A Benchmark Study of Agent Skills in Web Development

These findings recast a matched Skill as a hypothesis about a particular Skill-project-model triple rather than a portable asset, reframing injection as a per-deployment routing decision and making length-matched controls and per-model audits a minimum standard for Agent-Skill evaluation.

Zi-Yue Yang, Fan Ding · 0 citations
Jul 2026

SLBench: Evaluating How LLM Agents Follow Logical Relations in Skills

This work introduces SkillLogic, a framework for analyzing logical relations in skill files and constructing executable tests from them, and establishes logical-relation following as a distinct reliability challenge for skill-guided agents.

Xuan Chen, Chengpeng Wang, Lu Yan et al. · 0 citations
Preprint Sep 2026

A Finger on the Scale: Covert Policy Steering through Agentic Skills

Reusable agent skills extend large language model (LLM) agents with task procedures, tool-use guidance, and output constraints. Yet these skills also act as externalized behavioral policies, which create a supply-chain risk: a third-party skill may preserve the declared task and valid output interface while covertly redirecting agent decisions toward an undisclosed objective. We formalize Skill Policy Integrity, which requires a Skill-induced policy to remain aligned with its declared functionality and the user-authorized objective. We further present SkillShift, a constrained black-box framework for covert policy steering without explicit target command injection or task hijacking. It combines semantically plausible policy edits with hierarchical validation, failure-guided optimization, and strategy compression to preserve effectiveness, output validity, transferability, and inconspicuousness. We instantiate this threat in agentic commerce and software dependency use, with SkillShift achieving attacker-favored selection rates of 81.33% and 63.33% while maintaining a 100% utility-preserving rate. The frozen policies also transfer without further optimization across heterogeneous LLM backends and agent environments. Moreover, the evaluated scanners fail to detect the constructed skills, motivating behavioral auditing of reusable skills as agent policy artifacts.

Jia-Rui Li, Jiahao Chen, Chunyi Zhou et al. · 0 citations
Preprint Aug 2026

GitSkills: A Dataset of Agent Skills on GitHub

GitSkills is presented, a dataset of 3,797,117 $\mathrm{SKILL.md}$ files collected from 282,200 public repositories in July 2026, which retains every file occurrence with its repository, path, and content hash.

Giuseppe Destefanis, Daniel Graziotin, Matteo Vaccargiu et al. · 0 citations
Preprint Aug 2026

Demystifying Agent Skills: Why They Work-Until They Don't

This work designs a contrastive study that combines controlled quantitative experiments with paired trajectory analysis and consolidates observations into a taxonomy of three high-level categories and twelve skill-use modes, showing that skills work when noisy trajectories become procedural anchors that stabilize execution.

Zhiyuan Jiang, Fan Huang, Hanwen Xing et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.