Running a thousand experiments without losing one

Three questions that decide everything

Before discussing schedulers, it is worth naming the only questions that really matter for a parallel experiment platform. Everything else is implementation.

1. Where does a trial run when it starts? On hardware we control, or somewhere it can be interrupted by someone else's priorities?
2. What happens when it dies halfway? Whether a trial can resume, and what it costs to resume, sets the entire shape of the loop.
3. Who pays, and how is that decided before launch? Budget is not a report you read afterwards; it is a constraint the scheduler enforces continuously.

Scheduling: utilisation is not the goal

The obvious optimisation is to keep hardware as busy as possible. It is also the wrong target. Two trials, identical in cost, can differ enormously in what they teach: a wide sweep that rules out a region of design space is worth more than a narrow refinement near a point we already believe is good.

So the scheduler is fair-share rather than strictly first-come. Long trials are guaranteed a reservation so that they cannot be starved by an endless supply of short ones, and short trials are packed into the gaps. Neither the largest nor the smallest job is allowed to define the queue.

Preemption and recovery

At any real scale, trials are interrupted. Nodes are drained, jobs are preempted, a process is killed by a memory limit. The question is not whether this happens but whether it is visible.

A retry policy that is too generous is worse than no retry policy, because it converts a systematic problem into background noise. If the same configuration dies twice, that is not a flake; it is a measurement. Our approach is to distinguish clearly between infrastructure interruption (retry, same trial, no new information) and reproducible failure (record, classify, and treat as a result about the design).

Checkpointing exists to make the first category cheap. The cadence is a genuine trade-off — frequent checkpoints make recovery fast and the trial slower; infrequent checkpoints do the reverse — and the right answer differs by workload, so it is a per-campaign setting rather than a global default.

Budgets as first-class objects

In an autonomous loop, nothing external stops a campaign from spending more. A human-run experiment ends when the week ends. An automated one ends when something tells it to. That something has to be the budget, and it has to be enforced rather than monitored.

• Every trial has a compute, time and monetary ceiling attached before it starts.
• Campaigns have aggregate ceilings that the scheduler can refuse to exceed.
• Exhausting a ceiling is a normal, expected outcome — not an alarm.
• Spend is attributed per campaign and per objective, so cost per result is always answerable.

Result integrity

Parallelism introduces a new class of correctness problem that single experiments never face. Two trials of the same thing run at once and both get promoted. A run completes against a cached dataset that has since been replaced. A partially finished trial is scored by an eager report and quietly counted as a result.

The defence is unglamorous: deduplicate trials by configuration hash before they are dispatched, stamp every run with the data version it actually read, refuse to score incomplete runs, and never mix results produced against different versions in a single comparison. And when something does slip through, the fix is to mark the affected comparisons invalid — not to recompute them quietly and hope the number stays similar.

The metric that matters: effective trials per hour

Raw trial throughput is a vanity metric. What matters is effective trials per hour: trials that produced a usable, correctly attributed result, divided by wall-clock time. If a configuration doubles throughput and also doubles the proportion of unusable results, nothing was gained.

We track three numbers together — throughput, effective rate, and cost per effective result — because optimising any one of them in isolation reliably damages the other two.

What every run record must answer

A campaign produces thousands of runs, and months later someone will need to trust one of them. The test we apply is simple: the record has to answer five questions without anyone reconstructing the story from memory.

1. What exactly was run? Configuration, code version, dependency digests, and the identity of the candidate being tested.
2. Against what data? The dataset version actually read — never a floating reference to “current”.
3. How did it end? Completed, interrupted and resumed, or terminated — and if terminated, under which classification.
4. What did it cost? Compute, wall-clock and currency, attributed to the trial and rolled up to the campaign.
5. What was concluded, and by which scorer? The score, the version of the scoring path that produced it, and the baseline it was compared against.

Attribution is where this usually breaks first. The moment two trials share a pooled resource that is not tracked separately, cost per result becomes an estimate, and an estimate is a poor basis for the only question that matters commercially: is this loop getting more efficient or just more expensive?

Three things we got wrong

1. We treated trials as uniform. They are not. A handful of long trials determine when a campaign can conclude, and a scheduler that ignores that will happily starve them.
2. We underestimated queue latency. For short trials, time waiting for a slot dominated time spent computing. The fix was batching and reservations, not faster hardware.
3. We assumed hardware instability was exceptional. It is the normal operating condition at scale. Once treated as normal, it stops being an incident and becomes a design input.