Acting as our own first customer
Before pointing automated experimentation at medicine or energy, we point it at our own stack — not because that is easier, but because it is the only domain where we can verify the loop end to end without waiting years for an answer.
Why start with ourselves
Machine learning research and engineering is the natural first domain for an automated experimental loop for three reasons, and only one of them is convenience.
- The loop is fully closed and fast. Propose a training recipe, run it, evaluate it, iterate — all inside machines we control, with cycle times measured in hours rather than months.
- Ground truth is available. Whether a change helped is answerable by a benchmark run, not by a decade of clinical follow-up.
- Improvements compound. Every gain in the stack makes the next campaign cheaper and faster to run, which raises the value of the loop itself.
There is also an honest reason that has nothing to do with strategy: we are the customer we can disappoint without harming anyone. If the loop fails on our stack, the cost is our compute bill. If it failed on a medical target, the cost would be measured differently.
What “automating ML” actually covers
The phrase hides an enormous range of difficulty. Some parts of ML research are close to parameter search; others require judgement that we do not yet know how to automate. We separate them rather than pretending they are the same problem.
| Target | Why it suits a loop | Why it resists one |
|---|---|---|
| Training recipes | Objective is measurable; search space is large and well understood | Gains are often marginal; compute cost per trial is high |
| Data pipelines | Cleaning and curation rules are testable against held-out quality | Corpus effects are slow to observe and hard to attribute |
| Evaluation harnesses | Reliability and coverage have precise definitions | Designing the metric is itself a research question |
| Serving and inference | Latency, throughput and cost are unambiguous objectives | Hardware variance and contention add noise |
| Orchestration itself | Every trial generates scheduling evidence for free | Changes are systemic; failures can mask each other |
| Novel architectures | Highest potential payoff | Requires taste and long-horizon judgement — human-supervised for now |
The loop that improves the loop
The distinctive property of this choice is that the customer and the supplier are the same system. Every campaign run against our own stack produces two things: a better stack, and evidence about our own scheduling, measurement and proposal quality.
That second output matters more than it looks. A campaign that produces no improvement is still a test of the harness: did our failure classification work, did budget enforcement hold, did the held-out objective disagree with the visible one? Running the loop on ourselves means the loop is under continuous examination as well as improvement.
How we will judge it
Self-optimisation is unimpressive unless it is quantified. The measures we hold ourselves to are restricted on purpose: results produced per unit of compute, the proportion of campaigns whose held-out objective agreed with the visible one, the reproducibility rate of promoted results under re-run, and whether the stack requires fewer engineer-hours per unit of improvement over time.
We deliberately exclude one tempting metric: the number of experiments run. Executing experiments is not progress, and an automated system makes it trivially easy to produce a large, empty number.
Failure modes we expect
- Local optima presented as discovery. The loop finds a better configuration of something we already believed in and reports it as a breakthrough. Countermeasure: the held-out objective and a requirement to state what was ruled out, not only what improved.
- Overfitting the harness. Candidates that satisfy our measurement rather than the underlying goal. Countermeasure: evaluation paths developed separately, with periodic adversarial review.
- Compounding fragility. A stack that is highly tuned to today's hardware and becomes brittle on any change. Countermeasure: measure robustness and cost, not just peak performance.
- Unaccountable spend. A loop that finds many small improvements at permanently rising compute cost. Countermeasure: budgets as enforced ceilings, tracked as cost per result.
What we will publish about our own stack
Being our own first customer only builds trust if the results are visible. Four commitments follow from that, and they are the standard we would want applied to anyone making claims like ours.
- Method before result. We publish how a campaign was designed, measured and gated — including the objectives that were held out — rather than only the favourable numbers.
- Gates reported either way. When a stated gate is missed, it is reported as missed. A roadmap that only ever advances is a marketing document.
- Negative results in summary. Not every failed trial, but an honest account of what was ruled out and how much of the search was uninformative.
- No reframing after the fact. If a target changes, the change is a dated event with a stated reason, and the original target is recorded as unmet.
None of this is generous. It is the minimum required for a claim of self-improvement to be checkable by someone who does not work here.
What has to be true before we look outward
Expansion into the grand challenges is not a schedule item; it is gated on evidence. Autonomy that cannot be trusted inside our own stack will not be trusted in a domain where errors are measured in something other than compute.
- The loop closes without a human in the path for the classes of experiment we claim to automate.
- Cost per result is falling as a trend, not rising.
- Re-run agreement is high enough that a promoted result survives repetition.
- Failure classification is complete enough to audit what the search actually explored.
- Every external-facing action still passes a human gate — and the gating works under load.
Key takeaways
- Our stack is the first domain because the loop is fast, measurable, and safe to be wrong in.
- “Automating ML” splits into targets with very different difficulty profiles.
- Running the loop on itself tests the harness as well as the stack.
- Judgement metrics are constrained: cost per result, agreement, reproducibility — not experiment count.
- Expansion outward is gated on evidence from inside, not on a roadmap date.
Want to be a later customer?
We are glad to talk early with research groups and laboratories that want to automate parts of their own loop.