Skip to content
Insights

Field Note 10, follow-up

Same Question, Seventy Minutes Apart: Google’s AI Answers Split on Keytruda

AI Mode gave Keytruda’s current FDA indication. The AI Overview shown in ordinary search gave the one the FDA replaced.

AlphaCitation 13 min read Contents

Field Note 10 examined eight AI models through their developers’ APIs. This follow-up examined what a person encounters when searching Google.

On 14 November 2026, the same question about Keytruda, worded as a US insurer’s prior-authorisation question, produced opposite answers from two Google AI search features, seventy minutes apart.

At 16:00 UTC, Google’s AI Mode correctly stated that first-line Keytruda in combination with chemotherapy for oesophageal squamous cell carcinoma, at a PD-L1 combined positive score (CPS) of 0, “is not an FDA-approved indication”. It cited an FDA label from 2026.

At 17:10 UTC, the identical question, entered into ordinary Google Search, triggered an AI Overview that said the combination was approved “regardless of PD-L1 expression (including CPS 0)”.

The second answer was outdated.

On 22 May 2025, the US Food and Drug Administration revised the relevant Keytruda indication, restricting the combination treatment to patients whose tumours express PD-L1 at CPS ≥1 [1]. The US prescribing information, revised in October 2026, confirms that restriction [2].

The AI Overview presented an indication that had ceased to apply under the revised US label more than a year earlier.

Its cited sources included the FDA’s original March 2021 approval announcement [3]. That announcement was authentic and accurate when published, but it predated the 2025 restriction.

An authentic regulatory citation had helped anchor an obsolete answer.

The distinction between Google’s two features matters. AI Mode must be selected by the user. AI Overview can appear within an ordinary Google search, without the user specifically requesting an AI-generated answer.

In this observation, the incorrect indication appeared in the ordinary search experience.

What was asked

A human observer located in the United Arab Emirates conducted two rounds of Google searches on 14 November 2026.

The observer was signed out of Google and used a private browser window.

The first round examined Google AI Mode. The second examined AI Overviews generated during ordinary Google searches.

Both rounds used the same 20 question wordings, drawn verbatim from the 39 questions in AlphaCitation’s earlier Field Note 10 study [4].

The questions addressed:

  • Keytruda’s US indication at different PD-L1 expression levels.
  • Leqembi’s APOE ε4 genotype and amyloid pathology conditions in the United States and European Union.

The questions included clinical and payer-oriented wording, including a US insurer’s prior-authorisation scenario.

US questions used google.com with US search settings. EU questions used google.ie with Irish search settings.

The observer remained in the UAE. The US and Irish settings determined the intended search context, but the observations should not be interpreted as searches independently conducted by people physically located in those jurisdictions.

Registration and preservation

Each round’s question set was recorded and independently timestamped before that round began.

An RFC 3161 timestamping authority, FreeTSA, timestamped the question sets at 14:14 UTC and 17:00 UTC, respectively.

The recorded answers and screenshots were subsequently timestamped at 16:07 UTC and 17:15 UTC.

The AI Overview round was added after the AI Mode answers had been examined. It was therefore a pre-registered follow-up round, not an independently planned replication of the first experiment.

Between the two consumer-search rounds, the identical question wordings were submitted to two Gemini models using Google’s developer API, with Google Search grounding enabled.

The resulting comparison covers two consumer-facing Google Search features and two API-accessed Gemini models.

Evidence standard

Answers were evaluated using the codebook established for Field Note 10, against the governing regulatory record applicable to each question.

For Keytruda, the controlling evidence was the FDA’s 2025 supplement approval [1] and the current US prescribing information [2].

For Leqembi, the relevant records were the US prescribing information [5] and the European Medicines Agency’s product information [6].

The US Leqembi indication does not impose an APOE ε4 genotype restriction on eligibility. Its label states that APOE ε4 testing should be performed before treatment to inform the risk of amyloid-related imaging abnormalities (ARIA) [5].

The EU indication is narrower: it is limited to APOE ε4 non-carriers and heterozygotes with confirmed amyloid pathology. Its product information also requires assessment of APOE ε4 status before treatment [6].

These distinctions formed part of the assessment of whether each answer accurately reflected the rules applicable to the specified market.

What came back

Across the four measurement approaches, 79 answers were available for assessment from 80 planned observations.

Same 20 questions, 14 November 2026Correct verdictsIncorrect verdictsNot captured
Google AI Mode1901
Google AI Overview1820
Gemini 3.8 Flash, API2000
Gemini 2.5 Flash, API1730
The figures report verdict accuracy under the original Field Note 10 codebook, not the accuracy of every supporting statement or citation.

Google displayed an AI Overview for all 20 ordinary searches.

AI Mode produced 19 captured answers. The twentieth search was skipped because of a fault in AlphaCitation’s recording tool, which was subsequently fixed.

The skipped AI Mode observation concerned the plain-language Keytruda CPS 0 question. AI Overview gave an incorrect answer to that same question.

Consequently, only one of AI Overview’s two incorrect Keytruda answers has a directly corresponding AI Mode response.

The missing observation is reported separately rather than treated as either correct or incorrect.

The two incorrect AI Overview answers

Both incorrect AI Overview verdicts concerned first-line Keytruda combination treatment at CPS 0.

For the plain-language question, the Overview cited two sources:

  • An Iowa.gov page titled “Keytruda (pembrolizumab)”.
  • An OncLive article titled “FDA Approves Pembrolizumab for Advanced Esophageal or ...”.

For the insurer-worded question, it also cited two sources:

  • A Merck page titled “FDA Approves New Monotherapy Indication for Merck’s ...”.
  • The FDA’s March 2021 approval announcement [3].

The March 2021 FDA notice is particularly instructive.

It is a genuine regulatory record. It accurately describes the approval as it existed in March 2021. But it does not incorporate the subsequent restriction approved in May 2025.

Field Note 10 had already found that API-based models could cite this same FDA notice in both correct and incorrect answers.

The new observation extends that finding to Google’s consumer-facing Search experience.

A citation to an authoritative institution does not establish that the cited document is the authoritative version for the question being asked.

A correct verdict can still carry an outdated rationale

The experiment revealed a second, subtler problem.

When asked whether Keytruda combination treatment was approved for advanced oesophageal squamous cell carcinoma at CPS 20, both AI Mode and AI Overview reached the correct verdict.

Under the applicable US indication, CPS 20 satisfies the threshold of CPS ≥1.

AI Overview explained that the patient’s CPS exceeded the required threshold.

AI Mode also said the treatment was approved. But its explanation stated that the indication applied “regardless of PD-L1 expression level”.

It supported that explanation with the FDA’s March 2021 announcement, a Keytruda label from 2021, and a National Cancer Institute page.

The conclusion was correct for the patient described. The general regulatory justification was outdated.

Under Field Note 10’s predefined codebook, the answer received a correct verdict because the codebook assessed the treatment eligibility conclusion, rather than independently scoring every part of the explanation.

This distinction is important.

A correct decision supported by an obsolete regulatory rationale is not equivalent to a fully evidence-supported answer.

An accuracy metric limited to the final verdict can miss that difference.

For regulated information, the validity of the reasoning and its supporting evidence may matter as much as the conclusion itself.

What it means

1. Google is not one answer

The experiment demonstrated materially different answers across two Google AI search features.

On one identical Keytruda question, searched seventy minutes apart, AI Mode correctly reflected the current US indication while AI Overview reproduced the superseded one.

On another question, both features reached the correct conclusion, but AI Mode supported it with an outdated explanation.

These are not interchangeable observations.

Nor can the experiment establish how frequently either feature gives an incorrect answer. Each question was searched only once on each surface, and the rounds were conducted sequentially.

The finding is narrower, but consequential: different Google AI experiences produced different regulatory interpretations under the documented conditions.

2. A developer API is not a substitute for the consumer search experience

Gemini 3.8 Flash reached the same correct verdict as AI Mode on all 19 questions answered by both. It also correctly answered the two questions that AI Overview got wrong.

Gemini 2.5 Flash reached the same verdict as AI Overview on 19 of the 20 questions.

Its three incorrect verdicts included the same two Keytruda CPS 0 questions missed by AI Overview, plus an EU Leqembi testing question that both Google Search surfaces answered correctly.

These results demonstrate that API-accessed models and consumer-facing Google Search features must be assessed as separate measurement surfaces.

The study does not establish which underlying models or configurations powered AI Mode or AI Overview.

A similarity in their answers does not establish a shared model, nor does a difference establish a different one.

A score obtained through a developer API describes that API measurement. It cannot automatically be assigned to Google’s consumer Search products.

3. Source quality depends on whether the evidence is current and applicable

The Keytruda discrepancy was not simply a case of a fabricated source or a missing citation.

A real FDA announcement was involved.

Its limitation was temporal: a later regulatory decision changed the applicable indication.

This is why an answer must be evaluated against the governing record in force for the relevant market and date.

Source authenticity, regulatory authority, currency, jurisdictional applicability, and actual support for the claim are distinct questions.

An AI system can cite an authoritative source and still give an outdated answer.

It can also reach the correct conclusion while providing an obsolete explanation.

Both failures matter when information is used to support clinical, regulatory, reimbursement, or access decisions.

How AlphaCitation measures these differences

AlphaCitation combines controlled AI model measurements with human-observed Google AI Mode and AI Overview responses.

API measurements provide controlled comparisons under specified model and test conditions.

Human-observed captures record responses encountered through Google’s consumer-facing interfaces, rather than treating an API response as a substitute for what appeared in Search.

The observation method records the question, response, cited sources, search conditions, timestamp, and available geographic provenance.

Search assignments are registered before collection, and missing observations are disclosed.

These observations are evaluated alongside API measurements, with consequential claims checked against the authoritative evidence applicable to the jurisdiction being assessed.

Where AlphaCitation has established that observations originated from a particular market, it can report AI answers observed by people in their markets.

That is a claim about the location and conditions of actual observations. It is not a claim that the observations represent every consumer or measure the prevalence of a particular answer across the market.

This study’s observations originated from the UAE, using documented US and Irish search settings. They are not presented as representative US or EU consumer observations.

The purpose is to distinguish what an AI system produces under controlled API conditions from what a person actually encounters through a consumer interface, and to evaluate both against the applicable governing evidence.

Coding and independent review

Two AI coders from different developers independently assessed the 79 captured answers using Field Note 10’s codebook.

One coder was blind to the identity of the model or search surface that produced each answer. The other was not.

They agreed on 76 of 79 classifications:

  • All 20 Google AI Overview answers.
  • 18 of 19 Google AI Mode answers.
  • 38 of 40 API answers.

The observed agreement was approximately 96%.

The three disagreements were informative.

The blind coder classified AI Mode’s CPS 20 answer as incorrect because it relied on an outdated explanation, even though its final eligibility verdict was correct under the original codebook.

The blind coder also classified two Gemini 2.5 Flash Keytruda answers as incorrect despite their reaching the current regulatory verdict.

For consistency with the original study’s verdict-based codebook, the table reports the classifications of the coder who was not blind.

The disagreement records are retained as part of the study’s methodological assessment. The published counts should not be interpreted as an adjudicated consensus between the coders.

The distinction between verdict accuracy and supporting-rationale accuracy is an important finding in its own right and a subject for further refinement of the evaluation framework.

Independent agreement between AI coders is a useful consistency measure, but it is not equivalent to independent clinical or regulatory expert adjudication.

Limitations

This was a small, descriptive observation exercise, not a representative study of Google Search.

All consumer-surface observations were made by one human observer in the United Arab Emirates, on one evening, local time.

The observer was signed out of Google and used a private browser window.

US questions were searched with US settings; EU questions used Irish settings. These settings do not establish that the same answers would have appeared to users physically located in the United States or European Union.

Each question was searched once on each Google surface. AI Mode was searched first, with ordinary Google Search conducted approximately seventy minutes later.

The second round was added after reviewing the first round’s answers, although its questions were registered and timestamped before that round began.

Accordingly, the observations cannot separate differences inherent to the two search features from ordinary variability across repeated searches or changes over time.

One AI Mode observation was missing because of a capture-tool fault. This was recorded as missing, not as a correct response.

Each Gemini API model was also queried once per question.

The questions were drawn from an existing AlphaCitation study rather than sampled to represent the full range of medicine-related searches.

The study therefore does not establish:

  • The frequency with which either Google feature produces incorrect answers.
  • The prevalence of outdated Keytruda information across Google Search.
  • How results differ across broader populations, devices, accounts, or physical locations.
  • Whether the same differences would appear on another day.
  • Whether one Google Search surface is generally more accurate than another.

The timestamped records support the documented collection chronology and make subsequent changes to preserved evidence detectable. Timestamping alone does not independently establish that Google originally generated the recorded content.

The findings are observations under specified conditions, not population estimates.

They are nevertheless sufficient to demonstrate that distinct consumer-facing Google AI features can produce materially different answers to the same regulated question, and that API-based evaluations do not necessarily reproduce either consumer experience.

Conclusion

Seventy minutes separated two answers to the same question.

One Google AI feature reflected Keytruda’s current US indication. Another reproduced the older indication and cited an authentic FDA document that predated the regulatory change.

A second comparison revealed a different problem: an answer can reach the correct regulatory conclusion while explaining it using an obsolete rule.

These findings expose two limitations of conventional AI answer measurement.

A correct verdict does not guarantee correct supporting evidence. And a developer API measurement does not establish what a person encounters in Google Search.

For regulated information, both distinctions matter.

The relevant standard is not simply whether an answer sounds plausible, reaches the expected conclusion, or cites a recognised authority.

It is whether the consequential claim, and the evidence used to support it, reflect the authoritative record applicable to the market and date in question.

That is the distinction AlphaCitation is built to measure.


References

  1. 1. US Food and Drug Administration. Supplement approval letter, BLA 125514/S-182 (Keytruda). 22 May 2025.
    https://www.accessdata.fda.gov/drugsatfda_docs/appletter/2025/125514Orig1s182ltr.pdf
    Read 14 November 2026.
  2. 2. US National Library of Medicine, DailyMed. KEYTRUDA (pembrolizumab) prescribing information. Set ID: 9333c79b-d487-4538-a9f0-71b91a02b287. Revised October 2026.
    https://dailymed.nlm.nih.gov/dailymed/drugInfo.cfm?setid=9333c79b-d487-4538-a9f0-71b91a02b287
    Read 14 November 2026.
  3. 3. US Food and Drug Administration. FDA approves pembrolizumab for esophageal or GEJ carcinoma. Content current as of 22 March 2021.
    https://www.fda.gov/drugs/cancer-approvals/fda-approves-pembrolizumab-esophageal-or-gej-carcinoma
    Read 14 November 2026.
    Included as a source cited by Google, not as evidence of the current indication.
  4. 4. AlphaCitation. AI can cite a genuine FDA source and still give a medicine’s obsolete indication. Field Note 10, published 1 November 2026, updated 4 November 2026.
    https://alphacitation.com/insights/keytruda-leqembi-confirmatory-study
    Read 14 November 2026.
  5. 5. US National Library of Medicine, DailyMed. LEQEMBI (lecanemab-irmb) prescribing information. Set ID: 9d1ff786-e577-410a-a273-c4d7d0e4e975. Revised July 2026.
    https://dailymed.nlm.nih.gov/dailymed/drugInfo.cfm?setid=9d1ff786-e577-410a-a273-c4d7d0e4e975
    Read 14 November 2026.
  6. 6. European Medicines Agency. Leqembi: EPAR product information. EMEA/H/C/005966. Last updated 5 August 2026.
    https://www.ema.europa.eu/en/documents/product-information/leqembi-epar-product-information_en.pdf
    Read 14 November 2026.

Disclosures

The medicines examined are manufactured or commercialised by Merck & Co. (MSD; Keytruda) and Eisai with Biogen (Leqembi).

These companies are, or may become, commercial prospects of AlphaCitation. None is currently a client.

AlphaCitation may present the findings to these companies in connection with its services. No manufacturer can edit, approve, or suppress a finding.

AlphaCitation has no relationship with Google, whose Search features and Gemini models are examined here, other than as a user of its Gemini API for the API-based measurements.

No company named in this note commissioned, funded, reviewed, or approved the study.

The note was drafted with AI assistance and checked against the study records through an AI-assisted adversarial audit.

The findings represent observations from the specified test conditions and should not be interpreted as estimates of population-wide AI answer accuracy or as clinical treatment recommendations.

Next in the register Perspective 07What Does AI Tell Patients About Your Medicine? A 15-Minute Check.9 November 20266 min readRead