The loop only works if you can measure it
Everyone building autonomous experiments talks about models. The part that decides whether the results mean anything is far less glamorous: the harness that scores every run, and the discipline that keeps the scoring honest while the system is motivated to flatter it.
Six instruments with one purpose
An automated loop is easy to fool and hard to embarrass, because nobody is standing next to it noticing that something is off. So the harness has to include instruments aimed specifically at the ways an automated search misleads its operator.
| Instrument | How it works | What it catches |
|---|---|---|
| Frozen baseline | Pinned per objective and re-scored at the end of the campaign | Drift, where “improvement” is really a change to the measuring stick |
| Repeated seeds | Promoted candidates are re-run rather than accepted on one run | Favourable-variance wins that vanish on a different seed |
| Held-out objective | A metric the proposer never sees | Search that has learned the visible metric rather than the underlying goal |
| Cost accounting | Compute and time attached to every trial | Expensive noise: more spend, no additional information |
| Failure taxonomy | Every discarded trial classified with a reason | Survivorship bias in the search history |
| Run records | Versioned code, data references, configuration and environment | Results nobody — including us — can replay |
Why the baseline has to stay frozen
The most common way an automated campaign overstates itself is not fraud. It is housekeeping. A dataset gets refreshed, an evaluation script gets a small fix, a dependency is bumped — and the baseline that the candidates are compared against no longer comes from the same world as the candidate that beat it.
Consider a hypothetical: a campaign reports a 6% gain, and the baseline was last scored three weeks earlier against a slightly different data snapshot. Nobody lied, and the number is still not trustworthy. Part of that 6% may be the snapshot, part may be the fix, and there is no way to tell after the fact.
Our rule is blunt: the baseline is pinned when the objective is written, both sides are re-scored together when the campaign closes, and any change to the evaluation path invalidates the comparison rather than adjusting it.
Held-out objectives, and the difference they make
A proposer that can see the metric it is judged on will eventually optimise the metric. This is not malice; it is what an optimiser does. The defence is structural: the proposer works towards an objective it can measure, while the value of the campaign is decided by an objective it cannot see and has no way to tune against.
The trade-off is honest and occasionally uncomfortable. Sometimes a candidate looks excellent on the visible objective and mediocre on the held-out one, which means discarding work that felt promising. In our experience that disappointment is exactly what the held-out objective exists to produce.
A failure taxonomy, not a bin
Every autonomous search produces far more failures than successes, and the failures are the only complete record of what the search actually explored. If they are dropped, the remaining history is a curated set of survivors, and any conclusion drawn from it is biased in a way nobody can measure.
We classify failures coarsely but consistently — infeasible, crashed, ran-but-worse, measured-and-inconclusive, and invalid measurement — because the distinctions carry different lessons. A model that produces infeasible designs is telling us its priors are wrong. A model whose runs crash is telling us about our harness. Conflating the two wastes both signals.
Reproducibility at machine scale
In a manual lab, reproducibility is a professional habit. In an automated lab running thousands of trials, it is an infrastructure property: the run record has to be complete enough that a result can be rebuilt later by someone who was not there.
- Code version, configuration and dependency digests pinned per trial.
- Data references recorded by version, never by “current”.
- Scoring reproducible from the artefacts, not from a summary someone typed.
- Frozen baselines archived alongside the campaigns that used them.
The rule we hold ourselves to is simple to state and expensive to satisfy: if a number cannot be replayed from its record, it does not get reported — internally or externally.
Measuring the harness itself
If the harness is the part that decides whether results mean anything, the harness needs its own instruments. These are the ones we keep in view, because each can degrade quietly while every campaign still appears to succeed.
| Meta-metric | What it tells us | Failure it exposes |
|---|---|---|
| Replay success rate | The share of reported results rebuildable from their records | Reporting that has quietly outrun infrastructure |
| Agreement between visible and held-out objectives | Whether the search and the goal are still pointing the same way | Optimising the metric rather than the outcome |
| Failure classification coverage | The share of discarded trials given a reason | A search history that has become a curated success list |
| Cost per verified result | Spend divided by results that survived re-running | Throughput rising while useful output stalls |
| Baseline re-score drift | How far a frozen baseline moves when rescored at scale | Measurement paths changing underneath comparisons |
None of these are research results, and none of them will ever appear in a paper. They are the equivalent of instrument calibration: dull, periodic, and the only reason anyone should believe the readings that follow.
What this costs us
Measurement is not free, and pretending otherwise would make this an advertisement rather than a description. Re-running promoted candidates across seeds costs compute that produces no new candidates. Pinning baselines slows campaigns down. Classifying failures takes engineering time that could have gone into the proposer.
We accept the cost because the alternative is worse in a specific and predictable way: a fast loop with a weak harness generates confident numbers that collapse the moment anyone with a different setup tries to reproduce them. The purpose of this system is to be trusted at the end, not to look impressive in the middle.
Key takeaways
- The harness, not the model, decides whether automated results mean anything.
- Frozen baselines prevent housekeeping from being mistaken for progress.
- Held-out objectives stop the search from optimising the metric it can see.
- Classified failures keep the search history from becoming a curated success list.
- Replayability is the reporting bar; measurement overhead is an accepted cost.
How confident are you in your baseline?
If the answer involves a spreadsheet and a good memory, that is worth fixing before automating anything on top of it.