FDE-Gym: a 95-task benchmark shows even top models resolve only ~15% of realistic forward-deployed delivery work
DeepLethe and academic collaborators released FDE-Gym on October 16, a benchmark of 95 end-to-end customer delivery tasks adapted from real FDE projects across 18 sectors. In the realistic setting, the best of 13 model configs scores 0.56 and fully resolves only 14 of 95 tasks—finding unstated constraints is the main bottleneck.
Researchers from DeepLethe and a broad academic–industry group published FDE-Gym (arXiv:2610.22350) on October 16: a benchmark that grades agents the way a customer would—by uplift on the customer’s own business metric, not by ticket completion.
The suite has 95 end-to-end delivery tasks drawn from 20 scenarios across 18 sectors, each adapted by industry experts from real customer projects. Agents work in simulated environments with scripted stakeholders and production-like systems. Scores run from 0 (do-nothing baseline) to 1 (expert reference solution), with penalties for stakeholder messages, trial runs, and waiting time. Literal compliance with a misdirected sponsor request is capped near 0.2.
In the primary L3 setting—where the sponsor’s request may miss the real problem and constraints must be discovered—the best of 13 model configurations (Claude Opus 5.5) reaches a mean score of 0.56 and resolves only 14 of 95 tasks (score ≥0.8, no incident). Discovering unstated constraints drives the largest drop; even when a leading model obtains every key fact, it still hits high uplift on only a minority of those runs. Weaker models often score below the do-nothing baseline.
For FDEs, the paper is a rare quantitative map of the last mile: clarifying messy requirements, operating in live systems, and shipping measurable business change—not just passing a well-specified coding benchmark. An online practice platform is at fde-gym.com; code and sample data are on GitHub and Hugging Face.
Source: arxiv.org