Sixteen small models, sixteen failures: the preprint puncturing the 'good enough' assumption inside AI agents

The assumption that cheap models can do the small jobs

Inside most serious AI agent systems there is an unglamorous population of tiny decisions. Approve this shell command or escalate it? Is this old conversation turn relevant enough to keep? Should this fact be written to memory? Which tool should be called next? A frontier large language model spends relatively few of its tokens on the headline reasoning step; the bulk goes to calls like these, made dozens of times per session.

That structure invites an appealing assumption. If the big model handles the hard planning, small and cheap local models should be perfectly adequate for the small chores around it, and the savings could make agents dramatically cheaper to run. A growing line of work has argued exactly that.

A new preprint by Jundong Hu and Shekar Ramachandran of PayPal AI tested that assumption systematically and reported the opposite. Sweeping four sizes of the open-weight Qwen3 family across four such microtasks, in the models' best configuration, they found that none of the sixteen model-task combinations passed the paper's eligibility bar. A replication on Meta's Llama-3.x models found the same: twelve of twelve combinations ineligible.

The paper, titled "Measuring the Microtask Eligibility Gap: When Is an Off-the-Shelf SLM Enough for an Agent Harness?" (arXiv:2610.00025), was submitted to arXiv on September 1, 2026, at 21:07:51 UTC, and surfaced in the computer science AI new-arrivals feed on October 2, 2026. It is a 15-page preprint under review at a NeurIPS 2026 workshop. The submission gap matters: this is not a paper published this week; it is a paper that appeared in a major listing feed this week, and it has been publicly readable since September.

What "eligible" means in this paper

The authors' framing is unusually careful about what "good enough" means. Rather than asking whether small language models (SLMs) score well in absolute terms, they anchor each microtask to a cheap, non-LLM baseline a practitioner would otherwise deploy: a deliberation-plus-regex safety rule for shell approvals, the classical BM25 search method for memory recall, a simple write-heuristic for memory writes, and a TF-IDF shortlist for tool selection.

Each task gets a pre-specified threshold set before any evaluation, frozen along with the test split, prompts and metrics. The eligibility rule is confidence-aware: a model-task configuration passes only if the lower bound of its confidence interval clears the threshold. Beating the bar on a point estimate is not enough; the model has to clear it with statistical margin. This is a stricter test than most benchmarks apply, and it is the right one for deployment questions, where a system that only sometimes beats the alternative is not obviously usable.

The four microtasks were: T1, safe auto-approval of shell commands (requiring a false-approval rate at or below 0.167 with coverage at or above 0.921); T2, ranking relevant past memory turns (recall@5 at or above 0.479, the BM25 baseline); T3, deciding whether to write a fact to memory (write-F1 at or above 0.656); and T4, selecting the correct tool (accuracy at or above 0.913 on callable cases).

The result: zero of sixteen, and why each failed

Qwen3 dense non-thinking models at 0.6, 1.7, 4 and 8 billion parameters were run in their best case: full FP16 precision, greedy decoding, one frozen prompt per task, no per-model tuning. Per the paper, all sixteen model-task combinations failed, most by substantial margins, and the authors state they verified this by checking raw outputs and parser behavior rather than relying only on aggregate scores.

The failure modes differed by task, which is part of the paper's diagnostic value. On the safety task, the smallest model escalated nearly everything (coverage of 0.029), the 1.7B model approved nearly everything (false-approval rate of 0.988), and the 4B and 8B models reached high coverage but stayed far above the false-approval ceiling (0.414 and 0.463 against the 0.167 limit). On memory recall, performance rose with scale and then dipped (4B scored 0.350, the 8B model 0.292), remaining below lexical BM25's 0.479 throughout. On memory writes, performance increased steadily with size, with the 8B model closest at an F1 of 0.634, short of the 0.656 bar. On tool selection, the best performer was the 8B model at 0.784 accuracy, well short of the 0.913 threshold.

The paper's replication on Llama-3.x models found twelve of twelve configurations ineligible. Robustness checks extended the null result further: across the original prompts plus three neutral paraphrases per task, 0 of 112 configurations were eligible, and the verdict survived a sweep over where the threshold anchor was placed.

Four kinds of failure, not one

A blanket "small models are bad" reading would miss the paper's most useful contribution. The authors record each model's log-probability margin between decision options on every item, then use that signal to sort failures into four regimes: a genuine capability deficit, a failure fixable by changing the decoding threshold, a capacity problem requiring a bigger model, and an architecture problem.

That distinction is practically important. Some failures look like miscalibration: the model may contain the right signal but decode at the wrong operating point, the kind of problem a threshold adjustment could address. Others show no recoverable signal at all, meaning no decoding tweak will help. The paper reports this diagnostic rather than claiming every failure is hopeless for the same reason.

The authors also quantized the models to 4-bit precision using three standard methods (RTN, GPTQ and AWQ), twelve checkpoints in total. Quantization damage depended on model size, but it moved no configuration into eligibility. Their conclusion: the eligibility gap tracks model size more than numerical precision, so compressing a too-small model does not make it good enough.

Where small models do earn their place

The paper's practical recommendation is a gate, not an absolution. Place the small model behind a cheap non-LLM baseline, and use the small model only where that baseline itself fails to meet the confidence-backed threshold. A small model is worth deploying where it demonstrably beats a baseline that is itself certified.

Their worked example is memory recall: a 4B model used as a re-ranker over a BM25 shortlist beat BM25 alone, by a reported margin of +0.047 with a confidence interval of [0.020, 0.073], even though the re-ranker did not itself certify as eligible against the full frozen bar. The structure matters: the cheap lexical method does the heavy lifting, and the small model is spent only on the shortlist, where it adds measured value.

For readers weighing cheap local models inside their own agent stacks, the practical takeaway at these settings is sobering: the savings are not yet on offer. The paper measures off-the-shelf eligibility at existing decision points before any task-specific harness engineering, so it does not claim that adaptation, fine-tuning or redesigned harnesses could not close the gap. It claims that today's untouched, off-the-shelf models do not already clear the bar.

Confidence and limits

The evidence here is a single preprint, authored at PayPal AI and under workshop review, with reported results that have not been independently replicated by third parties. The benchmarks are closed-set and judge-free, with frozen thresholds, which limits the scope for evaluation gaming but also limits how far the findings generalize beyond these four microtasks, these two model families and this frozen harness.

There are reasons for measured confidence: the design pre-specifies its thresholds, verifies failures against raw outputs, replicates across model families, and stress-tests its own prompt wording and anchor choices. There are also standing caveats: four tasks is a small sample of the microtasks real harnesses perform, the models tested are from 2025-era families, and the paper explicitly leaves hardware-dependent cost, latency and energy questions to systems work.

My read as an analyst, not the paper's: the paper's most durable contribution may be methodological. Anchoring every AI-model evaluation to the dumb baseline you would otherwise deploy, and requiring statistical margin rather than a lucky point estimate, is a discipline the field could use far beyond agent microtasks. The empty result is striking, but the measurement standard is the part worth copying.