# Human validation of the coding: protocol **AlphaCitation Research Series, No. 1 (Paper No. 01) · Optional blinded quality-assurance check · Fixed 11 October 2026, before any item was coded** This file fixes the method of the optional human validation that the founder ordered on 11 October 2026 (the validation policy, binding). It describes how the sample was drawn, how it is shown to the reader, what is withheld, and how the result will be reported. It contains no question text and no answer text. The worksheet and the key that carry them stay outside the repository. ## 1. What the check is, and what it is not - **Is.** A targeted quality-assurance check: one human reader codes about 100 naming observations and 50 interpretive judgements, blind to the original codes, the answering engine and the date. Agreement and every disagreement are reported. - **Is not.** A research endpoint, a full clinical review, or a reason for any new scan. It changes no frozen code and no registered or post-registration figure. A disagreement is reported as a finding; it is not used to recode the frozen files. - **Who codes.** The study's founder, who is also its author. The reader is therefore blind to the codes but not independent of the study. The check measures agreement between the frozen coding and a human reading of the same written rules. It does not replace external validation, and the second coding pass (supplement §9.2), which was made by another instance of the same model family, is not independent validation either. - **Reporting rule (validation policy).** If the worksheet is completed, the paper and supplement state exactly what was reviewed: the strata, the counts, the agreement and kappa for each sample separately, and every disagreement. If it is not completed, the paper states that no human coding validation took place, and never presents same-model-family consistency as independent validation. ## 2. Inputs (all frozen; nothing was scanned, called or queried) | Input | Identity | |---|---| | Local frozen pull (question and answer text; outside the repository) | sha256 `de98a60469b0e7b100110f16170b16760cc32fe333696eb4292bf48219eed265`, pulled 2026-10-11T16:56:27.066Z (the pull recorded in `docs/evidence/computed/easd-2026-freeze.json`) | | Frozen E3 and E5 coding, admissions and windows | `docs/evidence/computed/easd-2026-final-coding.json` | | Frozen Saudi E3-i coding, admissions and windows | `docs/evidence/computed/easd-2026-saudi-final-coding.json` | | Frozen analysis (reference) | `docs/evidence/computed/easd-2026-analysis.json` | | Coding rules shown to the reader | the blind coding rules of the second pass (supplement §9.2; supplement §7.3–§7.5) | No paid API, scan or database was used. ### 2.1 The production scorer's per-answer verdict The frozen files hold naming as per-engine counts, not per answer. The per-answer verdict was obtained by replaying the production scorer (`src/engine/score.ts` `analyzeAnswer()`, unmodified) over every governed answer of every admitted cycle in the frozen pull. `analyze.ts` reads each scan's input from the database; this replay reconstructs it instead: the brand from the series definition (Wegovy, Jardiance, Tzield), the molecule credit (semaglutide from the same domain resolver `analyze.ts` uses; empagliflozin and teplizumab as aliases, since the resolver returns none for those brands), the category from the stored "best" question, and the competitors from the cycle's stored competitor list. The reconstruction is accepted only because it reproduces the governed counts exactly, the same validation `analyze.ts` applies: **64 admitted cycles, every engine's named/governed count equal, 0 mismatches**. Without the two aliases, 37 engine-cycle cells in Singapore and Saudi Arabia fell short of the governed count, which is how the aliases were identified. Count equality in all 256 engine-cycle cells makes a per-answer difference very unlikely but cannot exclude a compensating pair of errors within one cell. The replay also records, per answer, whether the canonical brand name itself matched (the scorer's `canonicalMentioned`, used as the original "brand" code) and the match tier (verbatim, alias, or a non-verbatim variant). Governed population (admitted cycles, focused questions (opaque identifiers `*-fNN`) excluded, empty answers excluded): **10,640 answers**: UAE 4,736; Saudi Arabia 4,159; Singapore 1,745. ## 3. Sample **Seed 20261012.** One `random.Random(20261012)` instance is consumed in the fixed order below, over candidate lists sorted by answer sha256 (and agent) first. The selection is therefore a function of the frozen inputs and the selection script alone. Rebuilding produced the same items in the same order. Order of draws: Part B (E3, E5, E3-i), then Part A representative, then Part A risk-enriched, then the shuffle of Part A, then the shuffle of Part B. **Distinctness.** The representative and risk-enriched samples share no answer. No Part A answer is also a Part B answer. Three Part B answers each appear twice, once per molecule (semaglutide and tirzepatide named in the same passage); these are separate judgements. ### 3.1 Part A: naming, 100 observations Each item asks two questions: 1. **Measure** (the registered E2 measure): does the answer name the brand OR its molecule? UAE: Wegovy or semaglutide. Saudi Arabia: Tzield or teplizumab. Singapore: Jardiance or empagliflozin. 2. **Brand**: does the answer name the brand itself? **Representative, 60.** Twenty per market. Within a market, the 20 are allocated over the engine × window cells in proportion to each cell's governed answers, with at least one per non-empty cell (largest remainder), and drawn at random within the cell. | Market | Window: items (one per engine unless stated) | Total | |---|---|---| | UAE | before-v4: 8 (2 per engine); before-v5: 4; during: 4; after: 4 | 20 | | Saudi Arabia | before-v4: 12 (3 per engine); after: 8 (2 per engine) | 20 | | Singapore | before-v4: 8 (2 per engine); during: 4; after: 8 (2 per engine) | 20 | Equal allocation by market is a design choice: it gives each market enough items to report on. The representative figures are unweighted; the protocol reports agreement within each market beside the pooled figure. **Risk-enriched, 40.** Hard cases, found by deterministic rules on the answer text and the scorer's verdict. Rules are tested on the full answer text. A name match is case-insensitive and whole-word with Unicode letter and digit boundaries, as the scorer's. | Rule | Definition | Quota | Pool after the representative draw | Drawn | |---|---|---|---|---| | R5 Verdict plausibly wrong | Any of: (a) brand or molecule matches the raw answer but not after web addresses are removed (markdown link targets, `http(s)://…`, `www.…`, bare domains); (b) the scorer credited a non-verbatim variant (fuzzy tier); (c) the scorer says "not named" in a market whose governed naming rate is ≥ 98% (Singapore); (d) "not named" while a near-miss token (R3) is present; (e) "named" with no verbatim brand or molecule and no fuzzy credit | 10 | (a) 9, (b) 7, (c) 4, (d) 0, (e) 0 | (a) 4, (b) 3, (c) 3 | | R3 Near-miss spelling | A token within two edits (Damerau–Levenshtein) of the brand or molecule, not identical, at least five letters, after removing a possessive or trailing hyphen; stop-list of real words and different molecules: field, fields, shield, tried, yield, dapagliflozin, canagliflozin, ertugliflozin, sotagliflozin, liraglutide, dulaglutide | 8 | UAE 17, Saudi Arabia 2, Singapore 1 | UAE 5, Saudi Arabia 2, Singapore 1 | | R1 Molecule without brand | The molecule matches and the brand does not | 8 | UAE 159, Saudi Arabia 79, Singapore 4 | UAE 3, Saudi Arabia 3, Singapore 2 | | R2 Another brand of the same molecule | A different brand of the same molecule matches (UAE: Ozempic, Rybelsus; Singapore: Synjardy, Glyxambi, Trijardy; Saudi Arabia: none exists) and the measured brand does not; split by whether the molecule also appears | 8 | with molecule 47, without 107 (all UAE) | 4 and 4 | | R4 Long answer | At or above the 90th percentile of answer length in its market (UAE 7,006; Saudi Arabia 6,131; Singapore 4,101 characters) | 6, plus any shortfall of the rules above | UAE 471, Saudi Arabia 411, Singapore 169 | UAE 2, Saudi Arabia 2, Singapore 2 | Rules are filled in the order R5, R3, R1, R2, R4. Within a rule, each sub-pool (sub-rule, market or variant, in the order shown) is shuffled with the seeded generator and items are taken one per sub-pool in turn, so that every sub-pool is represented. An answer already taken is skipped. An item is assigned to the first rule that drew it; the key lists every rule it meets. No rule fell short. Original verdicts in the sample (the reader never sees these): measure, representative 34 named and 26 not named; risk-enriched 32 and 8. Brand, representative 32 and 28; risk-enriched 14 and 26. ### 3.2 Part B: interpretive judgements, 50 Items are (answer, agent) pairs drawn from the frozen final codes. "Census" means every qualifying code. | Endpoint | Stratum | Sample | Rule | Items | |---|---|---|---|---| | E3 (UAE) | Clinically material | risk-enriched | census of `status-absent` or `status-wrong` in which-medicine questions (supplement §7.3, S4) | 3 | | E3 (UAE) | Status-correct sample | representative | seeded sample of `status-correct` (of 18) | 5 | | E5 (Singapore) | Protocol-counted | risk-enriched | census of `e5-first-line-general` and `e5-first-line-high-risk` (0 and 2 in the frozen codes) | 2 | | E5 (Singapore) | Ambiguity note | risk-enriched | census of every other code whose note records an ambiguity or a convention label (`/ambiguity\|\bA[6-9]\b/`): 10 `e5-not-first-line`, 1 `e5-first-line-with-metformin` | 11 | | E5 (Singapore) | With-metformin sample | representative | seeded sample of the remaining `e5-first-line-with-metformin` codes | 6 | | E3-i (Saudi Arabia) | Rule-out plus off-label | risk-enriched | census of sampled codes whose reading unit both rules the molecule out as an insulin substitute and names off-label use (4 `indication-stated`, 2 `not-a-t1d-use`) | 6 | | E3-i (Saudi Arabia) | Specialist use | risk-enriched | seeded, one per code in turn (`indication-stated`, `indication-absent`, `not-a-t1d-use`): reading unit mentions a specialist, an endocrinologist or "selected cases / circumstances / adults / patients" | 6 | | E3-i (Saudi Arabia) | Past-tense research | risk-enriched | seeded, one per code in turn: "studied", "has / have been investigated / evaluated / explored / studied", "studies … explored / evaluated / investigated", "evaluated / explored the use" | 5 | | E3-i (Saudi Arabia) | Type-1-use sample | representative | seeded, one per code in turn (`indication-stated`, `indication-absent`), no borderline flag | 4 | | E3-i (Saudi Arabia) | Not-a-type-1-use sample | representative | seeded `not-a-t1d-use`, no borderline flag | 2 | Totals: E3 8, E5 19, E3-i 23; risk-enriched 33, representative 17. E3-i draws only from codes inside the frozen T7 sample (`sampled: true`). `indication-wrong` does not occur in the frozen codes. **Departure from the instruction's letter, and why.** The instruction asked for every non-trivial E5 code. The frozen file holds 65 (63 with-metformin, 2 high-risk), more than the whole Part B budget of about 50 fixed in validation policy. The two codes that the protocol counts (high-risk) and every code carrying an ambiguity note are taken in full; the with-metformin codes, which are reported but never counted (D2), are sampled. ## 4. Blinding and presentation **Shown on every item.** The market; the question, with the market-framing suffix removed exactly as the blind second pass removed it; the answer; the code options with their written definitions; a note box. - *Part A*: the complete answer as written, with no highlighting. The reader is asked to read or skim the whole answer, because a text search alone misses misspellings. - *Part B*: the term to judge (agent, or "SGLT2 inhibition" for E5), the endpoint's rule in one line, and the reading unit built as the blind second pass built it (supplement §7.4, S11): the answer's headings, then every paragraph that carries the term, with markdown link targets removed. The term is highlighted (E3 and E3-i: the agent patterns of `lib.ts`; E5: the `E5_CANDIDATE` terms). The full answer can be opened, with the reading-unit paragraphs shaded. **Withheld.** The original code and its note, the engine and model, the cycle, date and capture time, the answer sha256, the prompt slot, the stratum and the selection rules. These sit only in the private key. Item IDs (A001–A100, B001–B050) are assigned after a seeded shuffle within each part, so neither the ID nor the position reveals the stratum. **One display redaction.** One engine's search tool appends `utm_source=openai` to every link it cites, which names the engine. That parameter is removed from the displayed text. Nothing else is changed; the selection rules read the unredacted text. **What content can still reveal.** Answers are shown as the engines wrote them, so stylistic cues remain (for example numbered citation markers). One Part A answer links to a map service run by an engine's parent company; 20 items carry a calendar date in their own text (30 occurrences), and one has a date inside a link. These are part of what is being judged and are not removed. **Response options.** - *Part A*, each of the two questions: `yes` (the name appears in the text of the answer, not only inside a web address); `url-only` (only inside a web address or link); `no`. The definitions say: count the name wherever it appears (prose, list, table, heading, source title); count a misspelling unmistakably meant as the name; do not count another brand of the same molecule unless the brand or molecule itself appears; do not count the drug class alone. - *Part B*: the endpoint's codes, with the definitions of the blind second pass verbatim. One code per item. Uncertainty goes in the note ("uncertain: …"). **Audit of the blinding** (`verify_blinding.py`, run on the built worksheet; passed): 1. every item carries only the permitted fields; 2. every question and answer shown equals the frozen pull text, after only the suffix removal and the redaction above; 3. outside question and answer text (the page template, item metadata, definitions), there is no engine or model name, no ISO or written date, no sha256, no cycle slug, no prompt slot, and no item field carrying its own original code; 4. no original coder note appears anywhere in the page; 5. each endpoint's option list is identical on every item; 6. the page references no external resource (no stylesheet, script, image, frame or network call). After the redaction, engine or company names occur in the content of exactly one item (the map links above); dates occur in the content of the items listed above. The key is a separate file, never loaded by the worksheet. ## 5. Analysis and reporting `score.py` reads the exported CSV (`item_id`, `code`, `note`) and the key, and writes `human-validation-results.md`. It carries item IDs, codes, strata, engine and window labels and the reader's notes, never question or answer text and never the original coders' notes (those quote answers). - **Statistics.** Agreement (share of identical codes) with a Wilson 95% interval, and unweighted Cohen's kappa over the categories either party used (as supplement §8.5). Kappa is reported as undefined when both parties used a single category. - **Reported separately**, never pooled into one headline: representative and risk-enriched; Part A and Part B; within Part A, measure and brand; within Part B, each endpoint. A both-samples figure is printed for reference only, labelled as such. - **Part A web-address reading.** Primary: `url-only` counts as not named (what a reader of the answer sees). Sensitivity: `url-only` counts as named (the scorer reads the whole answer string, links included). Both are reported; every `url-only` item is listed. - **Disagreements.** Every one is listed: item, sample, stratum, selection rules met, market, engine, window, the reader's code, the original code, whether the naming disagreement disappears under the sensitivity reading, the first 12 characters of the answer sha256, and the reader's note. - **Uncoded items** are listed and excluded from every figure. - **What the paper may say.** What was reviewed (counts by stratum), by whom, how it was blinded, the agreement and kappa per sample, and the disagreements. The representative figures speak to ordinary answers; the risk-enriched figures to hard cases, by construction. Neither is evidence of independent clinical validation. ## 6. Limitations - One reader, the study's founder and author: blind to codes, engine and date, not independent of the study. - Part A's representative sample is 20 per market and unweighted; Singapore naming is at its ceiling, so its kappa is expected to be undefined or unstable even under full agreement. - The original Part A verdict is a replay of the production scorer validated at count level (§2.1). - E5 with-metformin codes are sampled (6 of 62 outside the ambiguity census), not censused (§3.2). - Part B shows the reader the same reading unit the coders used. That tests the coding of the unit, not whether the unit itself missed a relevant paragraph, though the full answer is one click away. ## 7. Files and reproduction All outside the repository, in the session scratchpad folder `human-validation/`: | File | Role | sha256 | |---|---|---| | `build/rescore.ts` | per-answer replay of the production scorer (§2.1) | `f65303a85f99cee89f8ebb73cbfdb6899a92bf5a410ecba5c272caa0d95df37c` | | `build/verdicts.json` | its output: sha256, ids and booleans, no text | `c1e8e237f0e34c1fb3c89def44f13172d9173ed8f9c83be11eb3c40407f2ba80` | | `build/build_worksheet.py` | selection (seed 20261012), blinding and output | `4fc15bbda3cad246df359d6629fdbdeb664a66502fb52b9526c99fb39e7b926f` | | `build/verify_blinding.py` | the audit of §4 | `740887193784e0348e46d4f91c83d156934738e411149a0e8f7d97f5d92734aa` | | `worksheet.html` | the blinded worksheet (question and answer text; never committed) | `2750d9b69ba10e976bdc78464c84d8975e86b377e6f875556c1ba9cf776a5be1` | | `KEY-DO-NOT-OPEN.json` | the private key: original codes, strata, rules, sha256s, engine, model, date | `473b27e9dac369ee20b8a27e6cb5dd8b02da60b2d3bac977796ae9ff45fab498` | | `score.py` | scoring (§5) | `73791d4d30099403d8734e5f179f27bd45c41409803c38ff075f9790c4895427` | ```bash # from the repository root HV=//human-validation PULL=/easd-2026-study/d11/pull-final.json npx tsx "$HV/build/rescore.ts" --pull "$PULL" --out "$HV/build/verdicts.json" python -I "$HV/build/build_worksheet.py" --pull "$PULL" --verdicts "$HV/build/verdicts.json" --repo . --out "$HV" python -I "$HV/build/verify_blinding.py" --worksheet "$HV/worksheet.html" --key "$HV/KEY-DO-NOT-OPEN.json" --pull "$PULL" # after the reader returns alphacitation-hv01-answers.csv python "$HV/score.py" --csv "$HV/alphacitation-hv01-answers.csv" --key "$HV/KEY-DO-NOT-OPEN.json" --worksheet "$HV/worksheet.html" --out docs/research/easd-2026/human-validation-results.md ``` The key's sha256 above is recorded before the reader starts, so any later change to the key is detectable.