This work introduces circuits that organize LLM classifiers into an adaptive sequence of queries and decisions that characterize the optimal circuit, obtain the fundamental limits of achievable accuracy, and provide practical insights on realizing the optimal circuit.
Abstract
Language models make it easy to build classifiers by simply prompting a model with the sample to classify. However, their predictions are neither deterministic nor uniformly reliable across samples. How can we combine predictions from different models to improve reliability? We view LLM classifiers as noisy measurement instruments of an underlying target label. Our goal is to maximize average classification accuracy at a given average query cost. To this end, we introduce circuits that organize LLM classifiers into an adaptive sequence of queries and decisions. We then characterize the optimal circuit, obtain the fundamental limits of achievable accuracy, and provide practical insights on realizing the optimal circuit. Across seven classification tasks spanning safety, health, political science, human preference, agent evaluation, decision-making and reasoning, our method improves the cost-accuracy frontier over routing and cascading baselines.
Large language models (LLMs) are increasingly used as classifiers, yet they operate as opaque systems whose decisions are difficult to interpret, which complicates their use in regulated domains such as credit scoring or medical diagnosis. We propose an evolutionary framework that iteratively discovers natural language...
Jack Butler, Zainab Afolabi, Nikita Kozodoi· 0 citations
Since compact, interpretable classifiers correspond to simple programs, it is natural to ask if LLM-based coding agents can be used to learn interpretable models from data. In this paper we present a simple harness that repeatedly prompts a coding LLM to produce the next rule in a rule set, given a sample of labeled ex...
Aditya Kumar, Cassandra A. Cohen, William W. Cohen· 0 citations
Large Language Models excel at reasoning and generation but their autoregressive, token-by-token inference makes them unnecessarily slow, expensive, and poorly calibrated for the many routine decisions (routing, gating, scoring, classification) that modern AI systems require at sub-second latency. A growing class of mo...
This work evaluates how accurately large Language Models predict the impact of self-generated ML experiment ideas before these are put into training runs to show that current LLMs have useful but limited intuition about which ML ideas will work.
Ayush Bansal, Aldo Pareja, S. Klein et al.· 0 citations
The point at which a linear classifier stops being "sufficient" lacks a quantitative definition. To address this, we evaluated 1,687 records from 298 OpenML datasets across three model families. Using a practical significance framework, we defined linear sufficiency as an accuracy drop within a tolerance threshold δ. A...
Lu-Juan Zhang· Frontiers in Computing and I...· 0 citations