
OFFICIAL Written by GPN · Nov 14, 2026, 8:45 PM ET · · AI
AllenAI has released XYBench, a benchmark of 8,115 questions that tests whether AI assistants help with the problem a user actually has, rather than the one they asked about. It was developed with researchers at Carnegie Mellon, NVIDIA and Johns Hopkins, and won the Best Resource Paper Award at EMNLP 2026.
The benchmark is built around the XY problem: a person wants to achieve X, assumes a particular approach Y will work, and asks for help with Y. The post gives the example of someone asking how to thicken frosting that keeps sliding off cupcakes, when the real problem is that the cupcakes are still too warm to frost. A useful assistant should spot the misconception and redirect the user, not just answer the question as asked.
XYBench draws its questions from four sources: 1,557 from Stack Overflow, 184 from Stack Exchange, 102 manually curated examples, and 6,272 generated from WikiHow guides. For each question, the benchmark records both the stated request and the underlying goal that can be inferred from context.
The team evaluated five frontier models: Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B and DeepSeek-V4 Pro. On Stack Overflow questions, the models included a solution to the stated request in 75 to 86 percent of responses, but a solution to the underlying goal in only 59 to 71 percent. The focus was the problem too: models focused primarily on the underlying goal in only 35 to 41 percent of responses, against 47 to 57 percent focused on the stated request. All five trailed human reference answers at steering users away from the mistaken approach.
The gap is not just about recognition. When the same models were asked to pick between two answers, one focused on the literal request and one on the underlying goal, they chose the goal-focused answer in 71 to 84 percent of cases on Stack Overflow. But when writing freely, only 35 to 41 percent of their answers focused on the underlying goal. Knowing the better answer and producing it are two different things.
The paper, data and code are all released, so researchers can use XYBench to test whether changes to prompting or training help assistants recognise mistaken assumptions and make the real goal the focus of their answers.