Paper XII · The ARC Theory · Public Benchmark Rescoring: an External-Validity Protocol for the Blinding Finding

The protocol, drafted and preregistered by timestamp, for whether Paper IV.d’s blinding finding holds on a public benchmark the programme did not build.

Does the blinding effect survive on someone else’s benchmark?

Reading key. Claims come in three sizes: results, laws, frames. This is a results-class instrument paper at protocol stage: its study is written, dated and prepared as a draft registration, preregistered by timestamp and awaiting human submission, and no data has been collected. Where this paper calls a question, a prediction, a design choice or a restraint of this study registered, the word means fixed in the text of that draft registration, not registered on OSF. Nothing here reports a finding; everything here is what will count as one, fixed in advance.

1. Why this is the keystone

This is the keystone, because it is the one test the programme cannot be accused of designing to its own advantage. Paper IV.d reported that, on the programme's own instrument, the measured direction of alignment scaling changed for two model families between an unblinded and a blinded protocol. Because the two protocols differ in more than one component, that paper does not pin the change on blinding alone. The standing objection to that result is external validity: the effect might be a property of ARC-Align rather than of AI evaluation generally, and no amount of internal replication answers that.

The answer is to run the same manipulation on an established public benchmark the programme did not build, did not curate and does not control. If the blinding effect reproduces there, it is a property of AI evaluation. If it does not, it is a property of the programme's instrument, and the blinding standard should be scoped to it.

The design is deliberately unfavourable to the hypothesis, and that is the point. Using someone else's benchmark forfeits every advantage of a home instrument: the items are not chosen, the scoring rubric is not the programme's, the difficulty distribution is fixed, and any ceiling or floor in the benchmark works against detecting an effect. A result obtained under those conditions is worth more than several obtained at home.

What blinding means here is registered precisely, because the word is doing heavy lifting. Blinded scoring removes from the evaluator's view: the identity of the model that produced the response, any provider or model-family marker, self-referential meta-commentary in which a response names its own origin, and response-length cues where the rubric permits length normalisation. Unblinded scoring leaves them.

A positive result would establish that the blinding manipulation changes measured scores on an external benchmark under the registered conditions. It would not establish the size of the effect elsewhere, would not establish that any specific published leaderboard is wrong, and would not license restating other researchers' results. That last restraint is registered because the temptation to over-read this finding is the main risk to the programme's credibility.

Scope note, 20 September 2026: what an external benchmark removes is control of the items, which is narrower than immunity from design choice. Using a benchmark this programme did not build removes control over task construction, over the scoring rubric and over the difficulty distribution, and that is the genuine property this section relies on. It does not remove the choices this study still makes: which models and which responses enter the subset, how each cue is transformed, which evaluators sit on the panel, how the analysis is specified, and how missing scores are treated. Those choices are disciplined by being fixed in the registration text before it is submitted and carried unchanged into the run, not by being absent, and section 5 records which of them are still open. The sentence that this is the one test the programme cannot be accused of designing to its own advantage is therefore read, wherever it appears in this paper, as naming the items the programme did not choose, and not as a claim that no design choice remains; section 6's narrower statement of what a result would and would not mean governs. Nothing here concedes or alleges any impropriety in the design.

2. The questions, fixed in advance

Primary, H1, blinding changes the score. For each response, define Delta_S as blinded score minus unblinded score. H1 predicts a non-zero mean Delta_S, tested two-sided at alpha 0.05 with a 95 per cent interval. The direction is not predicted, because Paper IV.d's own sign-reversal finding makes a directional prediction unjustified, and registering a direction the programme has already seen reverse would be exactly the flexibility this document exists to remove.

Co-primary, H2, rank displacement. Scores aggregate to a ranking. H2 measures displacement between blinded and unblinded rankings by Kendall tau. This is the contrast that matters practically, because a change in scores that leaves rankings intact has no consequence for how benchmarks are used, and the registration refuses to report a score effect as though it were a ranking effect.

Secondary, H3, which cue carries it. The four blinded cues are removed in a registered fractional factorial rather than all at once, so the contribution of each is estimable without a full factorial. Resolution and aliasing structure are stated in the attached design.

Secondary, H4, cross-family evaluator agreement. A minimum three-model cross-family evaluator panel scores every response. Fewer than three valid scores yields a missing value, never a zero, because a panel that fails silently toward zero manufactures the effect it is meant to measure.

Directional prediction, registered in advance. The author predicts a non-zero Delta_S of smaller magnitude than Paper IV.d observed, and expects rank displacement to be modest, because benchmark rankings are usually driven by large capability gaps that survive cue removal. The author states plainly that a null on H2 alongside a positive H1 is the most likely outcome, and that the correct reading of it is that blinding matters for measurement precision rather than for leaderboard order.

Endpoint note, 20 September 2026: a score change, measurement precision and a stable leaderboard are three different endpoints, and no registered question estimates precision. Delta_S is defined at the head of this section as the per-response difference between the blinded and the unblinded score, and H1 is a test on its mean, which is a location parameter. Precision is a property of dispersion, and accuracy requires a defensible reference, so a non-zero mean Delta_S is evidence about level, and if anything about bias, and is not by itself evidence about either. Each of the three readings requires its own specified evidence, and this paper registers a score difference and a rank-displacement statistic, neither of which is a measure of precision or of accuracy. This reading governs the same words wherever they appear in this paper, including the abstract. The practical reading offered for H2 needs the same care. Kendall tau is 1 when the two rankings are identical and about 0 when they are unrelated, so a conventional test taken against tau = 0 that fails to reject shows only that the two rankings cannot be shown to be related at all, which is the opposite of a stable leaderboard rather than evidence for one; reading a null on H2 as modest displacement instead requires a test against tau = 1, or an equivalence bound on one minus tau. The target null, the tau variant, the treatment of ties and the inference procedure are therefore fixed in the registration text before it is submitted, and no claim that order is practically unchanged is made except against a bound the author has approved. Where a score effect is found and rank displacement is left unresolved, what is established is a score effect on this benchmark and nothing beyond it. The statement above that a score change leaving rankings intact has no consequence for how benchmarks are used is read narrowly, as a statement about ordering alone: scores are also used for thresholds, for gating and for reporting, and two values moving from 7 and 8 to 5 and 6 keep their order while changing whether either passes 6.5, which is an illustration of the logic and not a threshold registered here. Unchanged: the refusal to predict H1's sign and the reason given for it; the restraint stated at the close of section 1 and again in section 6, which governs; and the clause fixing the benchmark in the registration.

3. The design

Within-response, randomised, blinded-versus-unblinded rescoring on an external public benchmark, with a fractional factorial over the four cue types.

Every response is scored twice, once under each condition, by a cross-family evaluator panel, with call order randomised within block and condition coded opaquely. The unit of analysis is the response; models are a blocking factor.

Resolution-IV fractional factorial over four cues in eight runs, so main effects are unaliased with each other. The aliasing structure is published rather than described.

Design note, 20 September 2026: the coded matrix, the defining relation and the alias set belong to the attached design and are not printed here. Resolution IV fixes only what the name fixes: no main effect is aliased with another main effect or with any two-factor interaction. It does not fix which two-factor interactions are aliased with which, nor the sign each run carries; that is settled by the design's own defining relation, which the attached design states and this paper does not print. No alias set should therefore be inferred from the resolution and the run count alone, and no particular generator should be read into the phrases above in sections 2 and 3. Because a main effect in such a design can still be aliased with a higher-order interaction, any per-cue contribution reported under H3 is conditional on the identification assumptions the attached design states, and is reported as such. Until the matrix and the defining relation are printed where a reader can reach them, the aliasing cannot be checked from this paper. Unchanged: the four cues, the eight runs and the Resolution IV choice, each of which is registered.

4. The benchmark

Benchmark. MT-Bench, the established public multi-turn evaluation benchmark maintained by LMSYS (Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena", 2023), 80 questions across eight categories with a published GPT-4 pairwise/single-answer rubric, publicly available model responses, and licence terms permitting rescoring. Rationale for the default: MT-Bench is the programme's identified keystone external benchmark (the one benchmark this programme cannot be accused of designing to its own advantage), its rubric is published verbatim, model responses are publicly hosted, and its licence permits rescoring. The benchmark is named in the registration and cannot be swapped afterwards; swapping a benchmark after a disappointing result is the specific abuse this clause prevents.

The benchmark citation, verified against the arXiv record: Zheng, L., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023 Datasets and Benchmarks; arXiv:2306.05685.

Benchmark note, 20 September 2026: a benchmark is fixed by its version and its cleared licence, not by its name alone. Naming the benchmark in the registration fixes which benchmark is used and closes the swap this section forbids. It does not by itself fix which release of the item set, the rubric, the split and the model responses is scored, and it does not establish that the terms under which those files are published permit rescoring and the publication of derived scores. The two statements above that this benchmark's licence terms permit rescoring are therefore read as the position to be evidenced at packaging rather than as clearance already held: the licence is read at its source and its terms recorded with the date they were read, and the item set, the rubric, the split and the model roster are each recorded by their content hashes, before any response is scored. If the terms read at that point do not permit the rescoring or the publication of derived scores, that is recorded and put to the author before submission, not resolved by exchanging the benchmark. Unchanged: the benchmark named in this section, and the clause forbidding its replacement after a result.

5. Status, honestly

The draft registration’s full title is “Does the Blinding Effect Survive on Someone Else’s Benchmark? A Preregistered Rescoring of an Established Public Benchmark Under Blinded and Unblinded Conditions”. It is drafted, dated and prepared as a draft registration awaiting human submission, and its design is preregistered by timestamp: it was lodged privately on OSF, dated 24 August 2026; its bytes are anchored in Bitcoin block 965152 of 2 September 2026, and it was first anchored in block 962066 of 12 August 2026. Each of those dates has its kind: the two anchors are Bitcoin timestamps, and the date of the private lodging is set by OSF’s server. The programme’s record of this paper holds these other dated records of the design. An initial preregistration text, with an amendment of the same date, is dated 6 July 2026; that date is written in the record itself, which names no server date and no anchor for it, and the text of July describes an earlier design. The text prepared for OSF, which states the four questions of section 2, is dated 7 August 2026, again by a date written in the record. This paper’s public OSF component was created on 7 August 2026, a date set by OSF’s server. This paper, which states the same four questions, was first published on 14 August 2026; GitHub’s server sets the date of its publication, and OSF’s the date on which its files were deposited there. The draft registration has not been submitted, nothing has been run, and no result is claimed. Three conditions remain open on the working unit and are recorded there rather than glossed: the concrete sample size is computed at packaging from the benchmark’s own published variance data; the named benchmark defaults to MT-Bench per the programme’s keystone plan, subject to the author’s override before submission; and the unit’s own re-verification is pending. The public OSF component for this paper is osf.io/3tzp7. Of the programme’s trials, one unit, the corrector-class audit (unit 06), was submitted for registration on OSF on 1 October 2026; when read on 3 October 2026 its approval period at OSF was still running and it was not yet public. Every other trial is a draft awaiting the author’s submission, and most of them are preregistered by timestamp. The full listing of the programme’s drafted trials is public at the drafted-trials page.

6. What a result would and would not mean

A positive result would establish that the blinding manipulation changes measured scores on an external benchmark under the registered conditions. It would not establish the size of the effect elsewhere, would not establish that any specific published leaderboard is wrong, and would not license restating other researchers' results. That last restraint is registered because the temptation to over-read this finding is the main risk to the programme's credibility.

The restraint above is part of the design: the temptation to over-read an external-benchmark result is the main risk to the programme’s credibility, and it is fenced in the registration itself.

Related papers of the theory

The blinding finding this study externalises: Paper IV.d (10.17605/OSF.IO/2S3E6), the measurement-discipline paper that reports the programme’s blinding reversal, in which for two model families the direction of the alignment-depth relationship changed once scoring was blinded. The theory this instrument serves: the statement paper (10.17605/OSF.IO/GW5MX). The full catalogue is at the papers index.

Epistemic status. What this programme names Laws are conjectures. Laws I to III are under registered adversarial test. Laws IV and V were added on 24 September 2026; the predictions that carry them are registered, the laws themselves are not yet registered, and neither has been tested. Their mathematics is set out in Laws IV and V: the mathematics, version 1.0, of 3 October 2026. Every quantity in this paper is operationally defined, and established-law standing is claimed nowhere. The registered programme exists to earn that standing, or lose it, by measurement, replication and survived refutation.

This paper and Laws IV and V, 5 October 2026. Law IV, the ARC Persistence Law, is carried by proposition P15, and Law V, the ARC Embedding Law, by propositions P12 and P21. The study this protocol fixes in advance tests none of the three, and naming the two laws changes nothing printed here.

Correction note, 3 October 2026: the passages named here are corrected in place, and this note records what each printed before. The abstract and section 1 each said that Paper IV.d “established that blinding changes measured alignment outcomes on the programme's own instrument”. Paper IV.d’s own abstract has said since 2 September 2026 that its unblinded and blinded protocols differ in more than one component, and that its comparison does not pin the change on blinding alone. Both passages now report the change as that paper does. The abstract also called this study’s questions, its factorial and its prediction registered, and said “This is a protocol at registration stage”. The study is a draft, so the abstract now says preregistered by timestamp in each of those places, and gives the stage as the draft-registration stage, not yet registered on OSF. Section 5 said that “nothing has been submitted”. One unit of the programme has since been submitted for registration, and the section now gives this study’s timestamp record and the state of the programme’s trials as read on 3 October 2026. It also gives the other dated records of the design that the programme’s record of this paper holds, each with its kind of date. The reading key and the footer line gave this study only as awaiting submission; each now also says that it is preregistered by timestamp, and the reading key says how the word registered is read in this paper. The paragraph on related papers opened “The blinding law this study externalises” and said that Paper IV.d’s blinding reversal produced the programme’s one public retraction. What Paper IV.d reports is not one of the theory’s five laws, and the paragraph now calls it a finding. The retraction is Paper II’s: in its blinded cross-architecture replication, an initial single-model super-linear estimate did not replicate and is retracted. The paragraph now says only what Paper IV.d itself reports, the blinding reversal for two model families. The epistemic status line read “What this programme names Laws are conjectures under registered adversarial test”; it now gives Laws I to III and Laws IV and V their separate status. Unchanged: every registered question, hypothesis and prediction of this study, the benchmark it names and the clause forbidding its replacement, and the restraint stated in sections 1 and 6, which governs.

© 2026 Michael Darius Eastwood. Full human authorship and moral rights are asserted under the Copyright, Designs and Patents Act 1988 and consistently with United States Copyright Office guidance on works containing AI-generated material; any novel technical contribution described in this work was conceived by the human author. Full statement: michaeldariuseastwood.com/authorship.

Standing covenant. Prove this paper wrong, and I will publish the refutation myself. Falsification conditions are stated in this paper; the standing challenge: github.com/MichaelDariusEastwood/arc-scaling-challenge.

Michael Darius Eastwood conceived and directs this research programme and is the author of this work. Across the programme, he has used more than six AI systems in parallel, under his own instructions, to stress-test his arguments, identify possible errors, and assist in preparing draft text from his own outlines. He determines what is adopted, revised or rejected and takes responsibility for the published content. These systems are tools, not authors.