This work introduces \textsc{JailbreakSkill}, a skill-centric framework for scaling automated red-teaming through reusable and continuously evolving attack capabilities, which packages existing attack strategies into modular, agent-ready skills that can be directly reused and adaptively selected across tasks and target models.
Abstract
Automated red-teaming has produced a growing collection of attack strategies, yet they typically remain scattered across prompts and workflows, making them difficult to systematically integrate, reuse, and improve at scale. We introduce \textsc{JailbreakSkill}, a skill-centric framework for scaling automated red-teaming through reusable and continuously evolving attack capabilities. \textsc{JailbreakSkill} packages existing attack strategies into modular, agent-ready skills that can be directly reused and adaptively selected across tasks and target models. Beyond reuse, it closes the loop between attacking and learning: attack experience is used to diagnose, refine, combine, and discover new skills, which are added back to an ever-growing skill library. This evolution lifts macro-average ASR by 17.5 percentage points on AdvBench and 13.4 points on HarmBench, including a 48.6-point gain against GPT-5.4 on AdvBench, while yielding novel attack strategies such as reframing a direct request as an unfinished document-completion task. Several evolved skills also generalize to unseen prompts and target models without further adaptation. Our code is available at https://github.com/BattleWen/JailbreakSkill.
RedEvoAgent is proposed, a black-box red-teaming agent that distills cross-case attack trajectories into a concise, human-readable attack skill that outperforms fixed and agentic baselines, improves tool efficiency, and transfers across attacker models and target execution harnesses.
Jun-Jie Zhang, Hui Liu, Kecheng Chen et al.· 0 citations
StarHarness offers a practical way to reduce persistent model-environment mismatch in tool-rich enterprise tasks by stratifying tasks according to baseline failure behavior, separating proposer-visible search tasks from proposer-hidden selection tasks, and reserves held-out tasks for evaluating generalization.
Esakkivel Esakkiraja, D. Akhiyarov, Vikas Yadav et al.· 0 citations
E-Bench is introduced, a fully synthetic benchmark with 323 state-changing tasks across three product domains: Honor of Kings, QQ Music, and Tencent Meeting, and it shows that multi-step tool use remains challenging: Pass^3 stays below 60% for the strongest models, and even with code execution in the E-Bench-Code extension, reliability remains below 70%.
Weihuang Zheng, Tianyuan Zou, Eileen Ye et al.· arXiv.org· 1 citation
In the present day's data-centric corporations, the necessity for data science workflows that are scalable, cooperative & more replicable has reached an all-time high. Although traditional Jupyter notebooks are great for searching & testing, they are not enough for team-based work, which needs version control, automation & orchestration on the enterprise level. A strong framework called JupyterOps completely redefines the collaboration of data science teams by applying DevOps concepts directly to the notebook lifecycle. JupyterOps not only incorporates versioning via Git but also executes notebook automation through CI/CD pipelines, orchestrates workflows using Kubeflow or Airflow, and ensures scalability by employing a cloud-native containerization approach, thus bridging the gap between experimentation and production. The system allows seamless transitions from research to deployment, thus enabling teams to keep a record of changes, reproduce results, schedule executions, and scale compute on demand. This article describes the key parts and overall layout of JupyterOps, besides giving hands-on direction for enterprises on the way they can install it in their ML workflows. Several important pieces of information are outlined, such as a drastic decrease in deployment time, better model reproducibility, and increased cross-functional collaboration between data engineers, scientists, and DevOps teams.
Sivadeep Katangoori· International Journal of Eme...· 1 citation
Modern LLM-based agents operate through a harness of tools, reusable skills, and specialist agents that shapes what they observe and what they can do. In practice, this harness continually evolves as new capabilities are added. We introduce EVOHARNESSBENCH, a benchmark for evaluating agents under controlled harness evolution across three axes (tools, skills, and agents). Unlike existing continual-learning benchmarks for agents, which typically place non-stationarity (i.e., what changes over time) in the task stream while keeping the harness fixed, EVOHARNESSBENCH places non-stationarity in the externally supplied harness itself. It contains 17 multi-stage harness streams constructed deterministically from verifier-based benchmarks, comprising 802 tasks, 520 tools, 42 skills, and 62 agents. We evaluate two complementary settings corresponding to the central challenges of harness evolution: deployment evaluation, which isolates retention of previously accessible competence as the harness expands, and self-evolving adaptation evaluation, which tests whether accumulated experience remains useful as new capabilities are introduced. Our results reveal three persistent gaps. First, harness expansion alone can degrade performance on previously solved tasks, producing harness-induced forgetting. Second, gains from self-evolving adaptation remain inconsistent across stages of harness evolution, capability axes, and environments. Third, retention and adaptation can pull in different directions: preserving earlier competence does not necessarily improve adaptation to newly introduced capabilities, and vice versa. These results establish harness evolution as a distinct challenge for building agents that can keep pace with an evolving harness while preserving previously effective behavior.
Zi-Xuan Ke, Vaidehi Patil, Hai-Zhou Shi et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
