Ai2's XYBench tests whether chatbots spot the real problem behind a question
Ai2 and partners released XYBench, an 8,115-question benchmark that checks whether AI assistants notice when a user asks about the wrong fix, and found models trail human answers.
The Allen Institute for AI (Ai2) released XYBench on November 13, 2026, a benchmark that measures how well AI assistants handle questions built on a misconception. Ai2 says it built the test with researchers at Carnegie Mellon, NVIDIA and Johns Hopkins, and that the paper won the Best Resource Paper Award at EMNLP 2026.
What's new
- The problem it targets: the benchmark is named after the "XY problem", where someone wants outcome X, assumes method Y will get them there, and asks only about Y. Ai2's example is a person asking how to thicken frosting when the cupcakes are simply too warm.
- The data: according to Ai2, XYBench has 8,115 questions drawn from Stack Overflow, Stack Exchange, hand-picked examples and questions generated from WikiHow guides, covering programming and everyday tasks such as cooking and home repair.
- What it scores: whether an answer solves the stated request, the underlying goal, or both; which of the two it focuses on; and whether it steers the user away from the mistaken approach. Claude Sonnet 4.5 acts as the automated judge, and Ai2 says it checked the judge against two human reviewers.
- Results: Ai2 ran five models through the test: GPT-5, Claude Sonnet 4.6, Gemini 3.1 Pro, DeepSeek-V4 Pro and Qwen3.5-9B. On Stack Overflow questions, the models made the underlying goal the main focus in only 35–41% of answers, and all five were less likely than human reference answers to redirect users.
- Knowing versus doing: when the models were asked to pick between a literal answer and a goal-focused one, they chose the goal-focused answer 71–84% of the time on Stack Overflow, which Ai2 describes as a gap between recognising good advice and producing it.
The dataset, paper and code are public; Ai2 notes the benchmark currently covers English-language, mostly procedural questions.
Why it matters
Analysis: many people ask AI tools about the fix they have already settled on. These results suggest assistants often answer that literal question instead of questioning it, so it can pay to describe your end goal, not just the step you are stuck on.
Source: Ai2 (Hugging Face blog)