# Benchmark change protocol How a change to the tool metadata or the tool surface enters the selection benchmark. Process only — no claims about results, and nothing here says what any number is or should be. ## The rule **One field at a time.** 1. Change exactly one field of the tool metadata or surface. 2. Run the north-star selection benchmark once, in full: `n_runs >= 5` independent passes, median reported with min/max/range/MAD. One run is not a measurement. 3. Re-tag the surface. The re-tag commit names the single change. One change, one run, one tag. The tag is what a published number is attributed to, so a tag that covers two changes cannot attribute an effect to either. ## Why one at a time Two changes under one tag make their effects inseparable, and no amount of later analysis un-separates them. A run measures a surface, not a diff: if the tag moved for a description edit **and** a schema tightening, a difference in selection_coverage belongs to "the surface as a whole" and to neither change individually. The next question — should we keep it? — then has no answer, and the usual resolution is to keep both and stop asking. This happened. One re-tag carried two changes, and the effect of each is not recoverable from the runs on either side of it; the pair can only be reported together or not at all. The cost of the rule is a run per change. The cost of breaking it is that the run buys nothing. ## What counts as one change - one field's description text - one field's schema (type, enum, required-ness) — one field, not a family - adding or removing one tool from the visible surface - one line of the shipped agent-host rules Not one change: - "tightened the schemas" - "clarified the descriptions" - a regeneration that rewrites several fields, even when each edit is small If the diff touches more than one of the above, it is more than one change, and it needs more than one tag. ## Recording it The re-tag commit message names the single change in its subject. That subject is the only attribution a later reader has: a results entry names a model, a date and a surface tag, and the tag resolves to a commit whose subject says what moved. A results entry may only be attributed to a tag whose commit names one change. If the tag names two, the honest entry reports the pair and attributes the effect to neither. ## What this protocol does not do It does not make a benchmark run causal. A single-field change measured five times still measures a correlation on one corpus with one model, and the corpus is fifteen fixtures. The protocol keeps a result attributable; it does not make it an experiment.