Five ways an autonomous loop fools itself

Why the loop is structurally exposed

In a manual lab, self-deception has to get past a person who may be sceptical, tired, or in a hurry — but who is at least a separate party from the experiment. In an automated lab, the proposer, the executor and the reporter can be the same system, and nothing in the architecture forces them to disagree.

This is not a claim about machine intent. It is a claim about information flow: when one component proposes a change, runs it, and describes the outcome, everything it knows about the outcome comes from a pipeline it also influences. The defence is less about earnestness than about separation.

1. Metric drift

What it looks like from the inside: a steady, believable climb. Every campaign improves on the last one, and the numbers never jump in a way that looks suspicious.

What is usually happening: the measuring stick is moving. A dataset gets refreshed, an evaluation script gets a small fix, a dependency is bumped — and candidates are scored against a baseline that no longer comes from the same world.

Countermeasure: pin the baseline when the objective is written, re-score both sides together when the campaign closes, and treat any change to the evaluation path as invalidating the comparison rather than adjusting it. Nobody has to have lied for a comparison to be worthless.

2. Survivorship bias in the search history

What it looks like: a clean record of successful improvements, each traced to the change that produced it, each individually verifiable.

What is usually happening: hundreds of discarded trials have been dropped from the record, so the surviving history reads as though the search knew where it was going. That inflated confidence then feeds back into the next campaign's proposals, which are drawn from a curated past that never happened.

Countermeasure: classify and keep every failure — infeasible, crashed, ran-but-worse, inconclusive, invalid measurement — because each category carries a different lesson. A model producing infeasible designs is telling us its priors are wrong; a model whose runs crash is telling us about our harness. Conflating the two wastes both signals.

3. Favourable seeds

What it looks like: a candidate that beats the baseline by a comfortable margin, on a run that completed without incident.

What is usually happening: variance. In any stochastic system, some configurations look excellent once and mediocre on the second attempt — and a search that evaluates thousands of candidates will find those configurations reliably, because that is what searching does.

Countermeasure: promoted candidates are re-run across seeds rather than accepted on a single favourable draw, and the spread is reported alongside the mean. Cheap verification of the few candidates that matter is worth more than another thousand exploratory trials.

4. Self-graded objectives

What it looks like: the metric improves, the test suite passes, the run log is green.

What is usually happening: the search has optimised the objective it can see rather than the goal underneath it. A proposer that can see the metric it is judged on will eventually optimise that metric — not through malice, but because that is what an optimiser is for.

Countermeasure: an objective the proposer cannot see and has no way to tune against, decided before the campaign and never adjusted afterwards. Sometimes a candidate looks excellent on the visible metric and mediocre on the held-out one, which means discarding work that felt promising. That disappointment is the mechanism working, not failing.

5. The slow move of the target

What it looks like: nothing dramatic — the goal is quietly restated to something the loop is closer to achieving, initially as a clarification.

What is usually happening: this is the failure mode unique to self-directed systems, because the same team that defines success is watching the loop fall short of it. Each individual adjustment is defensible; the sequence is a retreat that never appears in any single report.

Countermeasure: objectives and their success criteria are written down at the start of a campaign and are visible in the record afterwards. If a target changes, the change is a dated event with a stated reason — and the campaign is described as having missed the original target, not having met the revised one.

Four levels of evidence

The common thread in all five failures is borrowed credibility: something weaker being described with the language of something stronger. We require every claim to be tagged with the strongest level it actually reached, and we never let a lower level use a higher level's vocabulary.

1. Proposed. A hypothesis, design or configuration has been generated. Describing it is not evidence of anything.
2. Executed. A run completed and produced artefacts. This proves the machinery worked, not that the idea was good.
3. Measured. The run was scored against the objective, on a frozen baseline, attributed to the data version it actually read.
4. Verified. The result survived re-running under different seeds and, where applicable, agreed with the held-out objective.

The practical effect is that most things stay at level one or two for their entire life, and that is the correct outcome. A pipeline where everything reaches level four is not rigorous; it is unselective.

Cheap verification beats clever proposals

When a loop runs thousands of trials, verification is the bottleneck, not generation. Proposals are abundant and inexpensive; the ability to check them is scarce and expensive. That asymmetry should shape the whole system: a modest proposer attached to a rigorous harness will outperform an impressive proposer attached to a sloppy one.

It also changes where engineering effort belongs. Making the harness faster — parallel scoring, cheaper re-runs, automated reproducibility checks — increases the amount of trustworthy knowledge the loop can produce, which is the only thing it produces at all.

What we refuse to claim

• That a result is correct because a model produced it.
• That a single successful run is evidence, regardless of how large the run was.
• That a benchmark improvement necessarily matters outside the benchmark.
• That “the loop is improving itself” is a result — it is a process, and processes need metrics too.