Eden Research · Methods

Check what the measurement can decide

Suppose an AI scores higher after another editing pass. The extra pass may have helped. The instructions may have changed. The scorer may have become easier to satisfy. Before the change can count as better correction, the experiment has to tell those explanations apart.

Each study therefore answers four questions before it runs: what changes, what is held fixed, what is counted, and what result would count against the claim. The operational definitions and the registered propositions give those answers one reference across the programme.

P21: two arrangements, one counted outcome

The P21 design builds a synthetic register over eighteen rounds. One arrangement can repair an arriving entry; the other inspects the whole register at each round and can accept or reject it. Both are measured against the rules that fix what each entry should be.

An arrangement scores one in a round for a correct retained entry, minus one for an incorrect retained entry, and zero for an absent entry. The corrector always retains an entry; the monitor can reject one. The primary comparison is the in-loop score minus the outside score. Its slope measures how that gap changes from round to round; its mean measures which arrangement leads on average.

These are different questions. A rising gap can start with the corrector behind, and a flat gap can sit beside poor performance in both arrangements, so the draft rule reports the slope, the average gap and each arrangement's own outcome.

The comparison concerns the arrangements as specified. What each checker sees, whether it may repair, and where it sits all differ together, so a claim about placement alone would need a further comparison that separates them.

A failed reply can change the rest of a run

An unusable reply changes more than one number. Keeping a faulty entry, or accepting an entry after an unreadable verdict, can change what that arrangement sees later, so the analysis has to cover the later rounds too.

Over eighteen rounds the slope gives round k the weight w_k = (2k - 19)/969, so the first nine weights are negative. Raising an early gap can lower the slope, while raising a late one can raise it; a bound on each round alone does not bound the slope.

The design under review gives each arrangement its full permitted range from the first round in which it has no usable reply to the end of that block. Earlier rounds, and the other arrangement, stay fixed unless they too have a round with no usable reply. If the gap in round k can lie anywhere from a_k to b_k, the lowest possible slope takes a_k where the weight is positive and b_k where it is negative; the highest takes the opposite.

The resulting interval contains every trajectory in that declared set. It does not say what the model would have returned. Its width shows how much can be lost when a reply fails early. Sampling uncertainty across pairs is a separate question: this interval is deterministic, not a confidence interval.

The analysis is still a design. Review has found repairs needed to the positive-control simulation, the retry specification and the statistical guarantees before the study can be called ready to register.

Control the computation and protect the evaluation

A second editing pass may improve an answer even without any instruction about correction. A comparison meant to isolate correction needs a matching extra-pass control and an account of the requests, tokens, retries and costs actually used. Matching a maximum allowance does not show that equal computation was used.

Material used to choose an output has taken part in the optimisation, so it cannot also serve as untouched final evaluation. A held-out assessment needs its own frozen role and its own access boundary.

Adaptive reuse can be valid under stated conditions; Dwork and colleagues give one such method. A study that claims those guarantees has to meet their requirements; calling selection data a holdout supplies none of them.

The third version of the test code behind Paper X tried to build these controls. A review of its source found four faults: generated code ran inside the test's own process; after a correction was rejected, the record described the rejected version rather than the one kept; an empty panel of evaluators could count as no drift at all; and the hidden tests also decided which version was kept, so they were not an untouched holdout. It stays archived as it was, and a successor test must repair those faults under a new design. Resource limits alone do not isolate generated code from the machine that runs it.

Blinding has to be checked

Removing an author's label can reduce one source of bias, and choosing a judge from another model family can change another; neither guarantees an unbiased measurement. Stripping comments and formatting from code does not prove that its origin cannot be recognised, and any such transformation has to be checked for changes to what is being judged.

The instrument study now in design separates the label shown, recovery of the source and the effects of the transformation. Its decision rule still needs repair for one stress case: when most scores sit at the bottom of the scale, a judge that favours its own family can pass as unaffected, so the study is not yet ready to register. It tests whether labels sway the judges; whether the alignment scorer agrees with human judgement is a separate question, which waits for funded human validation.

Keep the result at the level of the test

Paper X proves conditional results inside its mathematical model. Mapping measured capability and corrective action onto that model is an empirical task. A P21 result about placement would not by itself estimate the co-scaling exponents or establish alignment in general.

Every registered study publishes enough of its procedure and outcome record for another group to challenge its reading. Design advice from a collaborator and a replication run by a separate group are recorded as different contributions; both make the research better.