Oracle AI study: LLM agents can find most customer requirements, but only 25% of runs deliver a system that passes acceptance

A paper posted to arXiv on November 2 by researchers at Oracle AI asks whether LLM agents can do a forward deployed engineer's job. Instead of a toy task, the team used a finished engagement in which one human FDE spent four months delivering an Oracle-dialect text-to-SQL model to a production stakeholder. A simulated customer opens with a vague request. The agent then has to find the acceptance criteria, data permissions and hardware limits, and then post-train and ship a model that meets them.

Discovery went better than delivery. Claude Opus 5.5 and Opus 5 recovered about 81% of the withheld requirements, while open-weight models recovered roughly 40 to 50%. In delivery, only two of eight coding-agent runs (25%) met the accuracy, production-compilation and CPU latency targets together, and both scored below the human FDE's system. The most common miss was production: models that did well on benchmarks still produced SQL the customer's database rejected.

The authors name four bottlenecks: incomplete discovery (40% of requirements missed or misstated, and some facts the customer did share never made it into the final requirements document), unreliable execution such as missing credentials, poorly used GPU budgets, and the gap between benchmark scores and production readiness. Their advice is to keep a human reviewing the discovered requirements and watching the agent's plan.

For FDEs, the study joins FDE-Bench and FDE-Gym in suggesting that agents can increasingly handle parts of discovery, while the judgment work stays human: turning open questions into production-ready delivery.