FDE-Bench: best AI agents reach only 0.55 of a human FDE's delivery score on real underspecified customer projects

Researchers have released FDE-Bench (arXiv:2610.14539), a benchmark scraped from real forward-deployed engineer engagements to test whether AI agents can do the same end-to-end work: clarify an incomplete business request with a simulated customer, then iteratively build a solution that meets customer acceptance criteria.

The suite covers 49 de-identified cases (29 combinatorial optimization, 20 machine learning) drawn from projects that needed more than a month of FDE work. Clarification targets come from actual practitioner–FDE discussions. Agents get the original underspecified brief and data, up to 30 clarification rounds and a 12-hour solving budget; case-specific evaluators score the final artifact against the customer-accepted FDE solution as the 1.0 reference.

Across 18 agent configurations spanning seven model families and six scaffolds, the highest mean delivery quality was 0.5510. The highest delivery success rate—runs that actually hit the FDE reference—was only 6.12%. Feasibility and hard-constraint violations dominate failures. The authors conclude that reliable FDE-level delivery on underspecified real-world business tasks remains out of reach for the agents they evaluated.

For practicing FDEs, the paper is useful evidence that the hard part of the job—eliciting unstated rules and shipping something a customer will accept—is still poorly automated, and it gives teams a public harness to measure agent progress on that loop.