Back to #small language model

StrategyBench: Evaluating Explicit Strategy Induction in Large Language Models

Jinghan Tan Yuanzheng Wang Lu Chen Zijun Chen Yuqian Wang Maosong Sun
Aug 2026 · 0 citations · 24 references
Computer Science

Abstract

As large language models are increasingly used in data-scarce and evolving task scenarios, few-shot in-context learning (ICL) has become a key paradigm for task adaptation. However, direct ICL often uses a small set of examples without explicitly abstracting task rules, making it sensitive to example construction. In contrast, human learners often reduce such sensitivity by first summarizing task rules from examples and then applying them to new instances. To evaluate this ability, we propose StrategyBench, which selects strategy-inducible tasks from BIG-Bench, constructs reference strategies, and defines evaluation metrics along two dimensions: strategy quality and downstream utility. We further analyze strategy induction from three perspectives: task variation, model configuration, and adaptation setting, covering category-wise differences, generator-executor choices, demonstration design, and SFT-based adaptation. Experiments show that explicit strategy utility differs substantially across task categories and depends on both strategy generation and execution conditions. The benchmark is released at: https://anonymous.4open.science/r/StrategyBench-D53C.

View source

Similar papers

#small language model Open access Aug 2026

LifeSciBench: Evaluating Language Models on Realistic, Expert-Level Tasks in the Life Sciences

LifeSciBench is introduced, a benchmark of 750 expert-authored tasks designed to evaluate whether language models can handle realistic life science research work, with each constituent task paired with a human expert-written rubric.

Amelia Liu, Andrew Ho, Anne Marie Droste et al. · 2 citations
#small language model Review Open access Sep 2026

SYNGAP1-related disorder: pathophysiology, epilepsy, cognitive and behavioral phenotypes, and precision therapeutic approaches.

A rapidly advancing precision-therapy pipeline-including antisense oligonucleotides to upregulate the intact allele, AAV-based gene replacement, CRISPR-mediated transcriptional activation, epigenetic modulators, and rational pathway-targeted small molecules-offers realistic prospects for disease modification.

Debopam Samanta · 1 citation
#small language model Open access Aug 2026

Co-designed yoga nidra targeting anxiety in autistic children: A mixed methods feasibility study.

Autistic children experience disproportionately high rates of anxiety, yet common interventions such as cognitive behavioural therapy and traditional mindfulness practices may be less effective due to their cognitive and abstract demands. Yoga nidra, a form of guided meditation using concrete visualisations and breath awareness, may offer a more accessible alternative. This mixed methods study evaluated the feasibility and pilot efficacy of a co-designed six-week online yoga nidra intervention targeting anxiety in autistic children aged 8-14 years. Neurophysiological and psychological data were collected from 13 participants using parent- and self-reported measures of anxiety (ASC-ASD), intolerance of uncertainty (IUSC), and emotion dysregulation (EDI), alongside heart rate variability (HRV). Notably, the cohort included a minimally speaking child, which demonstrates the potential for the intervention to extend to autistic children with language support needs, a group who are often underrepresented in similar research. Results indicated no statistically significant changes in anxiety over time; however, medium effect sizes were observed in self- (Hedge's g = 0.55) and parent-reported anxiety (Hedge's g = 0.45), with three participants moving from clinically significant to non-significant anxiety levels post-intervention. Additionally, intolerance of uncertainty and emotion dysregulation demonstrated small to medium effect size reductions (IUSC, Hedge's g = 0.50; EDI-Reactivity, Hedge's g = 0.55; EDI-Dysphoria, Hedge's g = 0.25), suggesting potential benefits of yoga nidra in these areas. Greater attendance was significantly associated with self-reported reductions in anxiety (ASC-ASD-SR, r = .757, p < .05), but not the other outcome measures. Unexpectedly, HRV outcomes indicated reduced autonomic functioning post-intervention. Additionally, qualitative data from semi-structured interviews with three child participants and their mothers from the study were analysed using reflexive thematic analysis, revealing five key themes of their experience: Thinking About the Body is Hard; We Want More; Homework Sucks; Making Mindfulness Concrete; and On-Screen and At Home is Convenient. While the online format was appreciated for its accessibility, challenges with adherence and scheduling highlighted the need for more flexible delivery models. This study advances upon prior research by being the first to develop a co-designed yoga nidra intervention specifically for autistic children. Findings support the feasibility of yoga nidra as a complementary intervention for autistic children and suggest directions for future research, including larger trials and further co-design with the autistic community.

Tundi Loftus, Shu H Yau, Sophia Soares et al. · 1 citation
#artificial intelligence Preprint Aug 2026

TestifAI: Tomography-Based Testing for Deep Learning Systems

TestifAI, a deep learning testing framework for efficient and accurate estimation of robustness against combinations of perturbations, is proposed and partial model tomography is introduced, a novel approach to reconstructing model behaviour in a multi-perturbation space from tests that apply only a small number of perturbations.

Arooj Arif, T. Hartung, E. Botoeva et al. · 1 citation

Stacking Approaches for Multi-Label Emotion Recognition in Persian Using Large and Small Transformer Models

A hybrid model, termed SE_LLM_ST, is introduced, which leverages recent advancements in large language models (LLMs) and transformer-based architectures to effectively capture both contextual and sequential information vital for Persian emotion recognition.

Toktam Khatibi, Elham Farahani · 0 citations
#small language model Review Aug 2026

Large Language Models in Oral and Maxillofacial Surgery Triage: A Scoping Review

Large Language Models show potential in their diagnostic accuracy and consequent ability to reduce clinician burden, and may provide the greatest benefit when used to optimise referral quality at source, improving both clinician and potentially LLM triage downstream.

K. Surendran, I. Aziz, Glyndwr Jenkins · 0 citations

Related blog posts