FDE Times: Temperature 0 still isn’t deterministic — one eval run can be noise, so report a range
A 5 November FDE Times guide cites Thinking Machines’ experiment (same prompt, temperature 0, Qwen3-235B, 1,000 runs → 80 distinct outputs) and arXiv work showing accuracy swings of up to 15% between ordinary runs. For FDEs deciding keep-or-drop on a prompt, it recommends five runs, logging system_fingerprint, and reporting min–max or confidence intervals instead of a single score.
On 5 November FDE Times published a practical guide on why a single LLM eval run is a poor basis for a client keep-or-drop decision. Thinking Machines ran the same prompt 1,000 times at temperature 0 on Qwen3-235B and got 80 different completions; the most common answer appeared only 78 times. An arXiv study (2408.04667) found accuracy varying by up to 15% across ordinary “deterministic” configurations.
The guide follows Thinking Machines and Simon Willison in blaming batch non-invariance under changing server load more than GPU floating-point reordering. OpenAI and Azure document seeds as best-effort only; even matching seed and system_fingerprint does not guarantee identical answers, and larger max_tokens makes outputs less stable.
The recommended harness runs each prompt five times at fixed temperature, seed and max_tokens, records system_fingerprint per call, and reports mean plus min–max (or a confidence interval) rather than a lone percentage. When two ranges overlap, the honest answer is “no difference detected yet.” Snapshot-style exact-match tests on long outputs are called out as a trap that teaches teams to ignore failures. The skill maps directly to FDE job ads that mention evals or reproducibility.
Source: fdetimes.net