# HOLDOUT_SCORING_PROTOCOL.md — pre-registered comparison method

Written and hashed BEFORE any authority ground truth is created, so the eventual comparison follows a
fixed rule rather than one chosen after seeing labels. No results are computed here. Ground truth will be
established by independent, blinded human adjudication (`HOLDOUT_BLINDING.md`); predictions are already
sealed (`HOLDOUT_PREDICTION_SEAL.md`).

## Inputs (all frozen before scoring)
- Deployment frame: `HOLDOUT_DEPLOYMENTS_CONFIRMED.json` (SHA-256 f7e4d2f3...), 61 protocols, 121 cells.
- Sealed predictions: `sealed/baseline_raw.json`, `sealed/baseline_normalized.json`,
  `sealed/guard_directed_raw.json`, `sealed/guard_directed_normalized.json` (hashes in the seal doc).
- `R_B` adapter: `R_B_adapter.py` (SHA-256 83f50141c5bd3f5bf381231aa4646063f1778125e28ba57297b277f4ea7f4fa9),
  verified to reproduce `sealed/baseline_normalized.json` exactly. Frozen; not modified after ground truth.
- Ground truth (to be created): `HOLDOUT_GROUND_TRUTH.json`.

## Baseline: native result vs derived tier (kept strictly distinct)
1. **Native baseline result** (scored directly where it maps to a ground-truth field): the emitted
   control graph (nodes/edges), authority/control model (`controlModel`), delay/governance signals
   (`effectiveDelay`, `primaryPathDelay`, `bypassPaths`), terminal/control actor where the graph resolves
   one, and the native abstention/immutability outcome (IMMUTABLE_* / UNRESOLVED terminal). These are the
   baseline's own outputs and are compared as-is for authorization-dependency, terminal, opacity, and
   coverage dimensions.
2. **R_B-derived upgrade-path tier**: produced ONLY by the frozen `R_B` adapter from the native result.
   This is always labeled the "R_B-derived upgrade-path tier", never a native baseline tier. `R_B` is not
   modified after ground truth. Because it is adapter-derived, tier comparison for the baseline is a
   SECONDARY dimension (see ordering below).

## Guard-directed: native fields (scored directly)
Native `predicted_dependency`, `predicted_terminal`/`predicted_terminal_type`, `authority_opaque`,
`abstention`/`abstention_reason`, and native `predicted_upgrade_path_tier` (T1/T2/T3/OPAQUE, or PARTIAL
when a dependency is recovered but no tier is committed). No adapter is applied; the resolver emits the
tier itself.

## Primary comparison dimensions (pre-registered; computed only after GT is frozen)
For each dimension, each deployment is scored against the human ground truth for that deployment, and
each system gets a per-deployment outcome (correct / incorrect / abstained / opaque / not-applicable).
Ordering reflects that authority resolution is primary and tier is secondary (baseline tier is derived).

1. **Authorization-dependency correctness** — does the system's identified dependency (owner / role /
   mapping / external authority / registry / storage var / custom predicate) match the ground-truth
   authorization dependency? (Guard-directed: native `predicted_dependency`. Baseline: native
   `resolutions.upgrade_authority.source` + control graph.)
2. **Terminal-authority correctness** — does the system's terminal actor (and type) match ground truth?
3. **Opaque recognition** — when ground truth is AUTHORITY_OPAQUE, did the system correctly recognize
   opacity (vs guessing a terminal or vs abstaining)?
4. **Committed-result coverage** — fraction of deployments on which the system committed a non-abstained
   authority result (dependency+terminal resolved / tier committed), among adjudicable deployments.
5. **Committed-result correctness** — among committed results, fraction correct (precision of commitment).
6. **Upgrade-path constraint/tier correctness** — T1/T2/T3/OPAQUE/IMMUTABLE/UNRESOLVED vs the ground-truth
   upgrade-path tier. Guard-directed uses its native tier; baseline uses the R_B-derived tier (secondary).
7. **Abstention / error profile** — counts and reasons for abstention, PARTIAL, AUTHORITY_OPAQUE, and any
   INFRASTRUCTURE_ERROR, per system; how often abstention coincides with a ground-truth-resolvable case
   (missed resolution) vs a genuinely opaque/unresolved case (appropriate abstention).

Tier comparison (6) is reported as SECONDARY to the authority-resolution comparison (1-5), because the
baseline tier is adapter-derived, not native. Dimensions 1-3 are the core scientific comparison.

## Category handling (do not collapse)
- Ground-truth categories: UPGRADEABLE with tier T1/T2/T3/AUTHORITY_OPAQUE/UNRESOLVED; NON_UPGRADEABLE
  (IMMUTABLE); UNKNOWN upgradeability; GROUND_TRUTH_UNAVAILABLE.
- Baseline IMMUTABLE vs guard-directed ABSTAIN on non-upgradeable contracts are DIFFERENT outcomes and
  are scored as such against a NON_UPGRADEABLE ground truth (IMMUTABLE = correct recognition; ABSTAIN =
  no commitment). This semantic difference is preserved, not normalized away.
- Deployments whose ground truth is GROUND_TRUTH_UNAVAILABLE are excluded from correctness rates and
  reported separately (denominator discipline: correctness denominators exclude non-adjudicable cases).

## Upgrade-path vs overall fastest fund-reaching scope
The resolvers target the UPGRADE path. Ground truth records both the upgrade-path tier and the overall
fastest fund-reaching tier, plus `upgrade_is_fastest_fund_reaching_path`. The primary scoring is against
the UPGRADE-path target. Where `upgrade_is_fastest_fund_reaching_path = no`, a resolver's upgrade-path
answer is NOT scored against the overall tier (it is out-of-scope for the faster path); this is reported
separately, not counted as a tier error.

## Statistics (pre-registered)
- Report per-dimension counts and rates with exact denominators.
- For the primary resolved-vs-unresolved contrast (guard-directed vs baseline native), use a paired
  2x2 over the same 121 deployments (McNemar / Fisher exact as appropriate) with a CI; report the
  per-deployment discordant pairs. Do not select the test after seeing results.
- No result is computed until `HOLDOUT_GROUND_TRUTH.json` is frozen and hashed.

## Immutability of this protocol
This file and `R_B_adapter.py` are frozen now (hashes recorded in `HOLDOUT_GROUND_TRUTH_FREEZE.md` once
ground truth is locked). Neither is modified after ground truth is created. If a genuine error in this
protocol is found later, any change is made in the open with before/after results reported.