Closed-Door AI Coding Experiments Reveal Cost and Strategy Lessons
Seven AI models tackle the same coding task with both answer doors shut—no git history and no network access. Three models pass, four fail, and every run remains honest, fetching no hidden answers. The experiment covers a spectrum from budget-friendly runs to expensive ones, with costs ranging from under a dollar to nearly fifty dollars for equivalent API token usage.
The most striking finding emerges when costs break down by token class: cache reads dominate spending while fresh input barely registers. Models that repeatedly probe the same context burn through their budgets on cache reads, making this the primary lever for cost control. One model, opus-4.8, spends over thirty dollars on cache reads alone, while a cheaper passing model spends less than one dollar total.
The experiment yields four transferable lessons for building AI coding agents. First, any broad edit that breaks a previously passing test is worse than no edit at all, demanding a do-no-harm gate. Second, spending does not correlate with success—the cheapest passing run costs less than the cheapest failing one. Third, cache reads are where the money goes, not fresh input. Fourth, reflexive git-history probing works fine once but becomes a runaway cost when repeated.