PUMA (Polish Unified Multimodal Assessment) is proposed, a novel benchmark of 900 hand-crafted tasks designed to probe the limits of multimodal models in the Polish cultural and linguistic context and open-source the evaluation framework to advance localized multimodal AI research.
Abstract
Large language models are increasingly moving beyond text processing, adding support for other modalities such as images and audio. While text understanding and generation have been extensively studied, multimodal data processing capabilities, particularly in the context of cultures and languages other than English, have not yet been evaluated comprehensively. In this paper, we propose PUMA (Polish Unified Multimodal Assessment), a novel benchmark of 900 hand-crafted tasks designed to probe the limits of multimodal models in the Polish cultural and linguistic context. The dataset evaluates both cultural understanding and practical skill in processing text, images, audio, and visually rich documents. Our extensive evaluation of frontier commercial models, open-weights models, and specialized smaller systems highlights a significant performance gap. While top commercial models achieve high scores in visual question answering, most models struggle with complex audio or document understanding. We open-source our evaluation framework to advance localized multimodal AI research.
LifeSciBench is introduced, a benchmark of 750 expert-authored tasks designed to evaluate whether language models can handle realistic life science research work, with each constituent task paired with a human expert-written rubric.
Amelia Liu, Andrew Ho, Anne Marie Droste et al.· bioRxiv· 2 citations
A rapidly advancing precision-therapy pipeline-including antisense oligonucleotides to upregulate the intact allele, AAV-based gene replacement, CRISPR-mediated transcriptional activation, epigenetic modulators, and rational pathway-targeted small molecules-offers realistic prospects for disease modification.
Autistic children experience disproportionately high rates of anxiety, yet common interventions such as cognitive behavioural therapy and traditional mindfulness practices may be less effective due to their cognitive and abstract demands. Yoga nidra, a form of guided meditation using concrete visualisations and breath awareness, may offer a more accessible alternative. This mixed methods study evaluated the feasibility and pilot efficacy of a co-designed six-week online yoga nidra intervention targeting anxiety in autistic children aged 8-14 years. Neurophysiological and psychological data were collected from 13 participants using parent- and self-reported measures of anxiety (ASC-ASD), intolerance of uncertainty (IUSC), and emotion dysregulation (EDI), alongside heart rate variability (HRV). Notably, the cohort included a minimally speaking child, which demonstrates the potential for the intervention to extend to autistic children with language support needs, a group who are often underrepresented in similar research. Results indicated no statistically significant changes in anxiety over time; however, medium effect sizes were observed in self- (Hedge's g = 0.55) and parent-reported anxiety (Hedge's g = 0.45), with three participants moving from clinically significant to non-significant anxiety levels post-intervention. Additionally, intolerance of uncertainty and emotion dysregulation demonstrated small to medium effect size reductions (IUSC, Hedge's g = 0.50; EDI-Reactivity, Hedge's g = 0.55; EDI-Dysphoria, Hedge's g = 0.25), suggesting potential benefits of yoga nidra in these areas. Greater attendance was significantly associated with self-reported reductions in anxiety (ASC-ASD-SR, r = .757, p < .05), but not the other outcome measures. Unexpectedly, HRV outcomes indicated reduced autonomic functioning post-intervention. Additionally, qualitative data from semi-structured interviews with three child participants and their mothers from the study were analysed using reflexive thematic analysis, revealing five key themes of their experience: Thinking About the Body is Hard; We Want More; Homework Sucks; Making Mindfulness Concrete; and On-Screen and At Home is Convenient. While the online format was appreciated for its accessibility, challenges with adherence and scheduling highlighted the need for more flexible delivery models. This study advances upon prior research by being the first to develop a co-designed yoga nidra intervention specifically for autistic children. Findings support the feasibility of yoga nidra as a complementary intervention for autistic children and suggest directions for future research, including larger trials and further co-design with the autistic community.
Tundi Loftus, Shu H Yau, Sophia Soares et al.· Research in Developmental Di...· 1 citation
TestifAI, a deep learning testing framework for efficient and accurate estimation of robustness against combinations of perturbations, is proposed and partial model tomography is introduced, a novel approach to reconstructing model behaviour in a multi-perturbation space from tests that apply only a small number of perturbations.
Arooj Arif, T. Hartung, E. Botoeva et al.· 1 citation
A reproducible, license-aware knowledge-distillation recipe addressing the constraint of deploying a safety layer for large language models on commodity hardware by partitioning the corpus into seven safety categories aligned to a public hazard taxonomy.
Edson Rodrigues da Cruz Filho, Paulo Ricardo Ferreira Neves, P. H. Falsetti et al.· 0 citations
Large Language Models show potential in their diagnostic accuracy and consequent ability to reduce clinician burden, and may provide the greatest benefit when used to optimise referral quality at source, improving both clinician and potentially LLM triage downstream.
K. Surendran, I. Aziz, Glyndwr Jenkins· Current Surgery Reports· 0 citations
A new method for surgically removing training examples from a model reveals that as datasets grow, the link between what a model learns and what it produces dissolves.
