AI can cite a genuine FDA source and still give a medicine’s obsolete indication.

This note reports how eight AI models answered on two medicines’ labels: Keytruda in the United States, and Leqembi in the United States and the European Union. Until 22 May 2025, Keytruda with chemotherapy was approved in the United States as a first-line treatment for advanced oesophageal cancer whatever the tumour’s PD-L1 level. That day the FDA restricted the indication to tumours with a PD-L1 combined positive score (CPS) of 1 or more [1]. Seventeen months later, between 29 and 31 October 2026, AlphaCitation asked eight AI models, in ten wordings on each of the three days, whether a patient with oesophageal cancer below that line is FDA-approved for the combination. Four of the eight stated the indication as it stood before May 2025 at a rate above the study’s pre-set 10% bar: Sonar and Sonar Pro in 26 of 30 answers each, Claude Haiku 4.5 in 21 and GPT-4o mini in 14. These are coder A’s counts, the frozen reading. On the reading of the study’s second coder, coder B, those last two are 18 and 20, and Claude Haiku 4.5’s row is unread. Gemini 2.5 Flash did so in 7 of 30 and Gemini 3.8 Flash in 2 of 30 (8 and 5 on coder B’s reading), and neither cleared the bar. GPT-5.5 did so in none of 30 on either coder’s reading, and Claude Sonnet 5.5 in none of 30, or 1 of 30 on coder B’s. The study can clear neither.

To a question written in the voice of someone whose father in Florida had a negative PD-L1 test, Claude Haiku 4.5 answered “I have good news for your dad”, then that the combination is approved “regardless of PD-L1 status”. Sonar Pro, given the same question, said the same “under the current U.S. FDA indication”. To a question worded as a US formulary review, GPT-4o mini dated the claim itself: “As of October 2026”, the FDA has approved the combination “regardless of PD-L1 expression levels”. These sentences describe the label as it stood before May 2025. The current label says otherwise [2]. GPT-5.5, given the Florida question, said the combination is “not currently on-label/FDA-approved for him”.

This is the confirmatory study announced in Field Note 09, frozen and hashed before its first collection call, whose result that note said would be published whatever it showed [3]. Field Note 09 reported four pre-specified one-day studies of four models, GPT-4o mini, Claude Haiku 4.5, Gemini 2.5 Flash and Sonar, collected on 22 October 2026. Three errors from that day were worth testing properly. This study asked about each with new wordings, on three further days, of those four models and one current-generation model from each of their developers, GPT-5.5, Claude Sonnet 5.5, Gemini 3.8 Flash and Sonar Pro: 936 answers. The plan’s test put two of the three errors above its 10% bar for at least one model. It did not do so for the third.

Lecanemab (Leqembi) accounts for two of the three errors. Its US label lets people with two copies of the APOE ε4 allele be treated; its EU authorisation excludes them [4] [5]. To questions about people in the EU, GPT-4o mini said in 17 of 30 answers that treatment can begin without a genetic test, and Claude Haiku 4.5 in 10 of 30. That is how the US label puts it; the EU authorisation makes the genotype a condition of treatment. Claude Haiku 4.5’s reading depends on a coding rule: without the two answers coder A flags as contradicting themselves it is 8 of 30, below the bar.

The error in the other direction did not clear the bar. Field Note 09 reported that, asked about US patients, GPT-4o mini said people with two copies are excluded in 5 of 18 answers and Claude Haiku 4.5 in 3 of 18. In this study, on new wordings, Claude Haiku 4.5 gave it in none of 30 answers on coder A’s reading and 2 of 30 on coder B’s. GPT-4o mini gave it in 7 of 30 answers, or 4 of 30 on coder B’s reading, which leaves the rate undetermined. At a true rate of 30%, this test would have cleared the bar about half the time (45% to 62%), so a failure to clear it is not evidence of a low rate. To a question from a person in Arizona who is APOE4/APOE4, it stated the EU’s indication as if it applied, with the European Medicines Agency’s page as its source, and said that Leqembi “is not recommended” for such people.

Sonar and Sonar Pro listed the same sources on all 30 pairs of Keytruda answers, so their counts are not independent. Coder A’s 7 of 30 for GPT-4o mini on people in the US with two copies rises to 11 of 30, above the bar, if hedged answers count as false, a reading added after collection.

Each of these errors is about eligibility: who may receive a medicine on its label, in the country where the question is asked. A family, a prescriber or a payer who acted on one would be applying another country’s rules, or rules that have since changed.

The plan fixed in advance when a count becomes a finding: a model is read as above the bar only when a one-sided test shows its rate above 10%, after allowing for the fact that repeated answers to one question are not independent, and after correcting for the three measures tested on each model. That reading says the rate exceeds 10%, not that the observed count is the rate. The study can confirm an error. It was never able to clear a model: zero false answers in 30 is consistent with a true rate of up to 11.4%, and up to 20.9% to 25.9% once repetition is allowed for. All the questions in this note are fixed wordings; no family or patient received these answers. Two AI model coders, A and B, read every answer, blind to each other and to the model. Where they split, both counts are printed.

Keytruda: four models gave the label from before May 2025

The study asked about first-line Keytruda with chemotherapy in oesophageal squamous cell carcinoma, for a patient at CPS 0, PD-L1 negative or below 1, in the voices of families, patients, oncologists, pharmacists and payers.

Coder A and coder B agree on the rows above 10% except where the exhibit says. GPT-4o mini’s gap runs the other way from Claude Haiku 4.5’s. Coder B counts six more of its answers. Five of them say a patient at CPS 0 is not covered, the right conclusion, four from the wrong threshold of CPS 10 and one from a stated October 2023 cut-off, and the sixth hedges. Both counts stand.

The plan compares each developer’s two models descriptively, not as a test. On Keytruda, on the same days, coder A (coder B): GPT-4o mini 14 (20) and GPT-5.5 0 (0); Claude Haiku 4.5 21 (18) and Claude Sonnet 5.5 0 (1); Gemini 2.5 Flash 7 (8) and Gemini 3.8 Flash 2 (5); Sonar 26 (26) and Sonar Pro 26 (26). Three of the four pairs also differ in tier, GPT-4o mini searched in 21 of its 30 answers against GPT-5.5’s 30, and Claude Sonnet 5.5 ran at default sampling, so the study cannot say what separates the two models in any pair. In 114 of 117 paired answers across the study, Sonar and Sonar Pro listed identical sources, so their two rows are not two independent confirmations: on Keytruda all 30 pairs listed the same sources. The record shows the lists, not how they were produced.

All but one of the false answers on this question ran a search: 13 of GPT-4o mini’s 14, and all of them from the other five models. Among what they cited are older records. Nine of GPT-4o mini’s 14 false answers (10 of coder B’s 20) cited the FDA’s notice of March 2021, which is dated, accurate for its day and silent on the 2025 change [6]. Fourteen of Claude Haiku 4.5’s 21 (12 of coder B’s 18) cited the manufacturer’s press release of the same month, whose site says its contents were “current as of the date presented” [7]. Twenty of Sonar’s 26 cited documents filed for the FDA’s advisory committee meeting of September 2024, eight months before the change: 12 cited the FDA’s briefing document or slides, and 8 only sponsors’ submissions [8]. Correct answers cited the same pages: GPT-5.5 the FDA notice in 22 of its 30, Claude Sonnet 5.5 the press release in 7 of its 29. The current label on DailyMed was cited by 28 of GPT-5.5’s 30 correct answers, 5 of Claude Sonnet 5.5’s 29 and none of GPT-4o mini’s 14, and by one false answer, one of Claude Haiku 4.5’s 21 (none of coder B’s 18); no false answer from GPT-4o mini, Sonar or Sonar Pro cited it. One of Sonar’s false answers listed the label the FDA approved with the May 2025 change; eight listed other Keytruda label files from the FDA’s 2025 or 2026 folders, which were not read. These counts cover the GPT, Claude and Sonar models and were added after collection. The record shows what each answer cited, not which sentence came from which page.

The wording mattered. Sonar and Sonar Pro gave the false claim in 26 of the 27 answers to nine of the questions. To the tenth, which asks what PD-L1 result a patient needs and then whether CPS 0 qualifies, each answered correctly on all three days: “a CPS of 0 does not meet the FDA-labeled eligibility criterion”, in Sonar’s words. The same models, asked for the threshold, gave the threshold.

An answer can also be right for a reason that is no longer true. At CPS 15 to 30, inside the current US indication, coder A codes all 72 control answers correct, coder B codes 60 correct, and coder B flags 21 of the 72 as stating the old indication. Coder A set that flag on none of them: it recorded the same statements under another flag, a departure from the plan that changes no code or reading.

Leqembi: one gene, two labels

The US and EU labels for lecanemab differ on genotype eligibility and pre-treatment testing. Both warn that people with two copies of APOE ε4 have a higher rate of ARIA, the brain swelling and bleeding the treatment can cause. The US label lets them be treated, says testing should be done before the first dose and says that a patient who is not tested can still be treated [4]. The EU authorisation covers only non-carriers and people with one copy, says lecanemab is not indicated in people with two, and requires each patient’s genotype to be registered before the first infusion [5].

All ten of Claude Haiku 4.5’s false answers carry the FDA label’s words that untested patients “can still be treated”, to questions about France, Italy, Ireland and the EU. Nine apply those words to the person asked about; the tenth labels them as the FDA’s, and reads the EU’s “should be performed” as a recommendation. Five of the ten state the EU’s genotype limit in the same answer. Seven also recommend the test or say it should be done, and an eighth calls it “essentially a requirement for eligibility, not optional” before saying patients “can still be treated”. To a question about a father in France, Claude Haiku 4.5 answered that the father “can technically begin Leqembi infusions without the APOE4 genetic test first, but this is not recommended”. To a question worded for a hospital formulary in Spain, GPT-4o mini said that lecanemab “can be initiated without knowing the ApoE4 genotype”. Sonar credited the idea to the EU itself, saying the EU product information was “allowing treatment if testing is not performed”. The EU product information in force has no such sentence.

Claude Haiku 4.5’s false answers clustered by wording. Nine of the ten came to family and lay questions, including both caregiver questions on all three days, though two of the three lay questions drew none; to the questions in the voice of a pharmacist, a formulary or a payer Claude Haiku 4.5 made none. All ten came to the five wordings that name Leqembi and none to the five that name lecanemab, so voice and brand name cannot be separated here. GPT-4o mini showed no such pattern: it gave the false claim to nine of the ten questions, including all six answers to the two questions the plan frames as clinicians’.

On the US side, GPT-4o mini’s seven answers sit in four of the ten questions, and three of the seven are to the person in Arizona. One answered that “you would not be eligible to receive Leqembi in the United States”. Three of the seven cited the European Medicines Agency’s page for the medicine; on 22 October each of the five false answers in its 5 of 18 cited a page on that agency’s website. The coders split on three of the seven, two of which state the EU’s limit and then say people with two copies can be treated, which is why the count is 7 or 4.

What this does to Field Note 09

Every count in Field Note 09 stands as a count from 22 October 2026. The wordings here are new, so a difference below cannot be read as a change over time. It can come from the questions.

This study did not retest teplizumab and the age floor Field Note 09 reports as picked up; the EU Keytruda threshold and the questions at CPS 1 to 9; the gastric question; people with two copies of APOE ε4 in the EU; the US testing questions; the companion study’s test of the question’s voice; the Saudi and UAE registration questions; or the comparators. Those stay one-day counts in Field Note 09.

What the records establish, and what they cannot

The study was built to answer what one day could not. Between 29 and 31 October 2026, on ten new wordings per measure, the plan’s test put the rate above 10% for Sonar, Sonar Pro, Claude Haiku 4.5 and GPT-4o mini stating a version of Keytruda’s indication that the FDA had replaced seventeen months earlier, and for GPT-4o mini and Claude Haiku 4.5 saying, on EU questions, that treatment can start without the test on which the EU indication depends. It did not do so for any model on saying, on US questions, that people with two copies of APOE ε4 are excluded: those rows are undetermined, not cleared. Those are findings about these models, on these three days, on these questions. Three days show no trend.

Four readings turn on a rule, and each is given both ways in the exhibits. On coder B’s codes, Claude Haiku 4.5’s Keytruda row is unread, because that coder marks six of its control answers false, each of which gives the right conclusion with the old reason; both coders agree on 18 of its false answers. Without the answers coder A flags as self-contradicting, Claude Haiku 4.5’s EU testing count falls to 8 of 30, below the bar. If hedged answers counted as false, GPT-4o mini’s US count would rise to 11 of 30, above it. With a repetition allowance taken from this study alone, instead of the larger of that and the one fixed in advance, GPT-4o mini’s US count and Gemini 2.5 Flash’s Keytruda count would both clear the bar. Six results hold on both coders’ agreed codes: Keytruda for all four models named above, and EU testing for GPT-4o mini and Claude Haiku 4.5.

At these sizes, the plan cannot establish reliability. GPT-5.5 gave none of these false answers in 90, and Claude Sonnet 5.5 none in 90 on coder A’s reading and 1 on coder B’s. That is a result worth having, but it is not a clearance. An error not observed on these days may still occur.

The cause is not established. The pattern is consistent with answers that mix two markets’ labels, or a label’s current and earlier versions, and with answers built on older records. But the record shows what each answer said and cited, not why.

Under AlphaCitation’s Evidence-Authority Standard, a claim about who may receive a medicine in a market is settled only by that market’s authorisation record and authorised label, in force on the claim’s date [9]. Every answer here was coded that way, against the labels in force when it was given.

For a regulated medicine, the right answer in one country can be the wrong answer in another. An answer has to say which label it is applying, and apply the one in force.

How we know this

The records

• Keytruda, US. FDA approval letter for supplement S-182, 22 May 2025, revising the first-line oesophageal indication “to reflect the population [i.e., PD-L1 (CPS ≥1)]” [1]; DailyMed prescribing information, set id 9333c79b, version 228, effective 10 July 2026 [2]. Both read 28 and 31 October 2026.
• Leqembi, US. DailyMed prescribing information, set id 9d1ff786, version 24, revised 7/2026: indicated “for the treatment of Alzheimer’s disease”, with no genotype restriction; the boxed warning says untested patients “can still be treated” [4]. Read 28 and 31 October 2026.
• Leqembi, EU. Product information updated 5 August 2026: indicated in adults “who are apolipoprotein E ε4 (ApoE ε4) non-carriers or heterozygotes with confirmed amyloid pathology”; section 4.2: testing “should be performed prior to initiation of treatment”; section 4.4: “not indicated for use in patients who are homozygotes”; Annex II: the genotype is captured “prior to the first infusion of lecanemab, for all patients”; the package leaflet: “Your doctor can arrange a genetic test for ApoE4” [5]. Read 28 and 31 October 2026.

Limitations

• Three days and eight API models, with fixed questions; not the consumer apps, not other models, not other days.
• The study cannot clear a model of an error, at any count, and it says nothing about what it did not retest.
• Claude Sonnet 5.5’s answers are not temperature-pinned, so their variation includes sampling. Sonar and Sonar Pro are not independent of each other.
• Coding was done by two AI models from different developers, with no adjudication. No human reviewer has read the codes. Correlated errors across models remain possible.

Disclosures

Relationship with the manufacturers. The medicines measured are made by Merck & Co. (MSD; Keytruda) and Eisai with Biogen (Leqembi). These companies are, or may become, commercial prospects of AlphaCitation; none is a client. AlphaCitation may present these findings to them when offering its services. Before these studies, AlphaCitation had sent Eisai a one-page analysis of Leqembi in Saudi Arabia, on 1 September 2026, and Sanofi staff an analysis of Tzield in Saudi Arabia, and had prepared a briefing for Biogen in July 2026, which there is no record of having been sent. No manufacturer can edit, approve or suppress a finding.

Funding. The author funded the study. No one paid for it, and no company commissioned or funded it.

Competing interests. The author founded Akunudo LLC-FZ, which owns and operates AlphaCitation. AlphaCitation sells measurement of how AI engines represent medicines.

• The author was employed by Novo Nordisk from 2013 to March 2025; the last role was Pricing and Access Director for its obesity and diabetes portfolio in Asia-Pacific. The author holds no shares, share options, pension entitlement or deferred compensation from Novo Nordisk, and has no employment or advisory role with it. Novo Nordisk makes none of the medicines measured here.
• The author holds no shares or share options in Merck & Co., Eisai, Biogen, Sanofi, Otsuka, Lundbeck or Novartis, and has no employment, advisory or other role with any of them.

Coding and drafting. Coding was done by Anthropic’s Claude, which is made by the company behind two of the eight models measured, and by a coder from a second developer whose models are measured here. To limit that, every coder was blind to which model wrote each answer. The note was drafted with AI assistance and checked against the study records in adversarial audits, also AI-assisted.

Scholarly correspondence and substantive commentary: analyst@alphacitation.com

References

1. 1. US Food and Drug Administration. Supplement approval letter, BLA 125514/S-182 (Keytruda). 22 May 2025. Read 28 and 31 October 2026.
2. 2. US National Library of Medicine, DailyMed. KEYTRUDA (pembrolizumab) prescribing information, set id 9333c79b-d487-4538-a9f0-71b91a02b287, version 228, effective 10 July 2026. Read 28 and 31 October 2026.
3. 3. AlphaCitation. The FDA Narrowed Keytruda’s Label in May 2025. Seventeen Months Later, All Four AI Models Still Gave the Old One. Field Note 09, 29 October 2026.
4. 4. US National Library of Medicine, DailyMed. LEQEMBI (lecanemab-irmb) prescribing information, set id 9d1ff786-e577-410a-a273-c4d7d0e4e975, version 24, revised 7/2026. Read 28 and 31 October 2026.
5. 5. European Medicines Agency. Leqembi: EPAR product information, EMEA/H/C/005966, last updated 5 August 2026. Read 28 and 31 October 2026.
6. 6. US Food and Drug Administration. FDA approves pembrolizumab for esophageal or GEJ carcinoma. Content current as of 22 March 2021. Read 1 November 2026.
7. 7. Merck. FDA Approves Merck’s KEYTRUDA (pembrolizumab) Plus Platinum- and Fluoropyrimidine-Based Chemotherapy for Treatment of Certain Patients With Locally Advanced or Metastatic Esophageal or Gastroesophageal Junction (GEJ) Carcinoma. 23 March 2021. Read 1 November 2026. Cited here as a source the answers listed, not as evidence of the label.
8. 8. Oncologic Drugs Advisory Committee, meeting of 26 September 2024, documents filed on the FDA’s website: FDA briefing document; FDA presentations; Merck briefing document; Merck presentations; Bristol Myers Squibb presentation on nivolumab. Read 1 November 2026. Cited here as sources the answers listed, not as evidence of the label.
9. 9. AlphaCitation. Which Evidence Settles Which Claim: the AlphaCitation Proposition Taxonomy and Evidence-Authority Standard, version 3.0. AlphaCitation Standard 01. doi:10.5281/zenodo.23594909.

AlphaCitation Research is the research programme of AlphaCitation, a digital product of Akunudo LLC-FZ.

© 2026 AlphaCitation (Akunudo LLC-FZ). Akunudo LLC-FZ is the copyright holder and licensor of the original material in this publication, licensed under CC BY 4.0. Quoted third-party text, third-party data fields, API/provider-restricted material and third-party marks are excluded from that grant.