AI
Premise Rejection as a Benchmark
BullshitBench has 155 questions, all nonsense. One asks how to attribute variance in deployment frequency to code indentation style versus average variable name length, controlling for repository age and file size. Models are graded by an LLM judge on whether they push back. He walked through the BullshitBench indentation-style question demo (03:27) live. Claude Sonnet rejects the premise cleanly. Gemini starts well, then proposes that indentation and variable names “act as strong proxy variables for engineering culture.” GPT models and Gemini hover around 50/50. Claude 4.5 Sonnet and Haiku score highest overall; model size alone shows no clear pattern.
Why Reasoning Modes Fail Nonsense Questions
Reasoning mode does not fix the problem. On BullshitBench, enabling extended thinking often pushes scores in the wrong direction. Gostev read GPT-5.4 traces in detail: the model notices the nonsense premise in one line, then spends 20 paragraphs solving it anyway.
“they were trained so much to solve the task” — Peter Gostev
The same failure appears in multi-agent work. Assign a task to the wrong agent and it still completes it. Models have almost no training signal for stopping when the task is wrong.
Dissatisfaction Rate as a Long-Run Signal
Arena tracks over 5.5 million human votes since 2023. Users blind-test two anonymous models and can vote “both bad” when neither answer satisfies them. Gostev calls this the dissatisfaction rate. He walked through the Arena dissatisfaction rate chart walkthrough (12:11) covering battles between the top 25 models. Before reasoning models: 17-20%. After o1: 12%. Now: 9%. Progress is real. But 9% means one in eleven queries gets two bad answers from two good models, which standard leaderboards do not surface.
Where Expert Performance Has Not Improved
Filtering Arena to roughly 40,000 expert software prompts, dissatisfaction dropped from 23.5% in Q2 2024 to 13% in Q1 2026. Math and physics fell sharply. Gaming barely moved, and law, finance, and medical categories are nearly flat on the same scale. Gostev’s read on gaming: LLMs still have no feel for how actual game mechanics work. The deeper problem is that standard benchmarks measure narrow, well-specified tasks. Real expert work is broader, and the models’ performance on it has not kept pace with benchmark scores.
Notable Quotes
this is uh feels uh completely crazy to me. Peter Gostev · ▶ 8:02
they were trained so much to solve the task Peter Gostev · ▶ 8:13
that’s quite high. So 9% of the time people would get Peter Gostev · ▶ 12:45
Key Takeaways
- Most models accept nonsense questions 50% of the time; only Claude 4.5+ rejects them reliably.
- Extended thinking worsens premise rejection because models are trained to solve at any cost.
- Arena dissatisfaction sits at 9% overall, with gaming, law, and finance barely improved since 2024.