FDE Times: Shadow mode for agents — 90% agreement can still mean kappa near zero

Before an agent is allowed to touch live decisions, FDE Times argues for a shadow period: production requests are copied to the agent, users still see only the existing process, and both sides are logged. Raw agreement misleads when one label dominates—an agent that always clicks approve can match operators 90% of the time with kappa exactly zero.

The walkthrough fixes one discrete decision and label set, uses a shared table for agent and operator decisions (keeping agent_reason for disagreement review), wraps the agent so exceptions become ERROR rows without breaking the operator UI, and records the human click only after finalisation so operators never see the agent’s suggestion first. Kappa subtracts chance agreement; a taught 100-case example with 89% agreement yields kappa ≈ 0.36 (“fair” on Landis–Koch orientation bands).

The shadow period ends with a promote / hold / rollback report that a human process owner signs—n, agreement, kappa, error rate, disagreement matrix and sample agent_reason rows. Cost matters too: shadow traffic can roughly double peak load, so say that before the invoice. For FDE CVs, “built shadow mode, reported agreement and kappa, customer signed promotion” beats “built AI agents.”