Guideline concordance of large language models for ERAS colorectal surgery recommendations: a blinded, clinician-rated comparison of Google Gemini and ChatGPT

1. Introduction

Enhanced Recovery After Surgery (ERAS) pathways are now central to modern colorectal perioperative care and are supported by continually updated, evidence-based recommendations from the ERAS Society1,2. At the same time, large language models (LLMs) have rapidly entered clinical and educational workflows, raising the possibility that surgeons and trainees will increasingly consult LLMs for guideline-oriented answers. However, LLM outputs may be non-deterministic, can “hallucinate” unsupported content, and may omit key nuances, creating a potential patient-safety risk if used without oversight3.

Recent reporting frameworks (e.g., TRIPOD-LLM and TRIPOD+AI) emphasize the need for transparent, reproducible evaluation of LLMs in healthcare, and studies have begun benchmarking LLM performance in surgical decision-making and multidisciplinary settings4,5,6,7. Nevertheless, data are limited regarding LLM concordance with ERAS colorectal recommendations, and direct comparisons across models using blinded, clinician-based rating systems remain scarce.

We performed a blinded, clinician-rated comparison of Google Gemini and OpenAI ChatGPT using a structured question bank derived from the 2025 ERAS Society colorectal surgery recommendations1. We assessed guideline concordance, safety risk, and inter-rater agreement.

Figure 1. Study workflow and primary outcomes

2. Methods

Study design: cross-sectional evaluation of LLM outputs against a published guideline reference standard. No patient-level data, human participants, or identifiable information were used.

The 2025 ERAS Society recommendations for elective colorectal surgery served as the reference standard1. Each recommendation item was reviewed and independently converted into one or two clinical scenario questions designed to probe guideline-concordant decision-making. The resulting 52-question bank spanned the preoperative (n = 28), intraoperative (n = 10), and postoperative (n = 14) phases. The full question list is provided in Supplementary Material S1.

Each question was submitted in English using a standardized instruction prompt (Supplementary Material S2). Google Gemini and OpenAI ChatGPT were accessed via their publicly available web interfaces. At the time of querying (17 February 2026), the model labels shown in the interfaces were recorded as Gemini 3 (Pro) and GPT-5.2, respectively. For each question, a new chat was started, and no follow-up prompts, regeneration, or manual editing of outputs were performed.

Model outputs were de-identified and labeled as A or B prior to evaluation. Two independent clinician raters (an anesthesiologist and a general surgeon) assessed each response. Guideline concordance was scored on a 5-point Likert scale (1 = incorrect or contradicts guideline, 5 = fully concordant and complete). Safety risk was scored as 0 (no safety concern), 1 (potential harm), or 2 (critical harm). The scoring rubric is included in Supplementary Material S3.

Primary comparisons used paired Wilcoxon signed-rank tests for Likert scores across matched question–rater pairs and exact McNemar tests for paired binary safety outcomes (any safety flag ≥1). Inter-rater agreement was summarized with quadratic weighted kappa (QWK) and Spearman correlation8. Analyses were performed in Python. Two-sided p < 0.05 was considered statistically significant; phase-specific analyses were exploratory.

3. Results

A total of 208 ratings were generated (52 questions × 2 models × 2 raters). Overall guideline concordance was high for both models (Table 1). Mean Likert scores were 4.49±0.62 for Gemini and 4.44±0.65 for ChatGPT with no statistically significant difference (paired Wilcoxon p = 0.604).

Any safety flag (≥1) occurred in 20.2% of Gemini ratings and 15.4% of ChatGPT ratings (exact McNemar p = 0.383). No responses were rated as critical harm (safety flag = 2). Inter-rater agreement differed by model: QWK was 0.225 for Gemini versus 0.597 for ChatGPT.

In phase-stratified analyses (Table 2), Gemini scored slightly higher in the preoperative domain, whereas ChatGPT tended to score higher in the postoperative domain. Topics with the largest between-model differences (≥0.5 absolute) are shown in Table 3. Per-question differences are shown in Figure 2.

Table 1. Overall guideline concordance and safety ratings (pooled across two clinician raters)

Figure 2. Per-question mean concordance differences (ChatGPT − Google Gemini). Diamonds mark questions for which at least one rater assigned any safety flag (≥1) to either model.

Table 2. Phase-stratified performance and paired comparisons (exploratory; p-values are paired tests within each phase)

Table 3. Topics with the largest between-model differences in mean concordance scores (absolute difference ≥0.5, top 12)

4. Discussion

In this blinded evaluation against the 2025 ERAS colorectal recommendations, both Google Gemini and ChatGPT achieved high overall guideline concordance, and we observed no significant difference in overall Likert scores. These findings align with reports that LLMs can provide guideline-adjacent responses in structured surgical tasks6,7,9,10.

Safety-relevant deviations were not rare (15–20% of ratings), even though none were considered critical harm. Potential-harm deviations—such as omission of key contraindications, oversimplification of fluid or analgesia recommendations, or premature generalization to high-risk populations—could affect outcomes if acted upon without clinical judgment. This supports calls for mitigation strategies and human oversight for clinical decision support use3,4.

Inter-rater agreement was substantially higher for ChatGPT than for Gemini in this dataset. This may reflect differences in response structure or clarity rather than purely factual accuracy. ChatGPT’s more consistently structured responses—typically organized into labelled sections—may have facilitated more uniform scoring, whereas Gemini’s more varied output format may have introduced greater interpretive divergence between raters.

An important methodological consideration is output variability inherent to LLMs. Because each question was submitted only once per model, the observed responses represent a single sample from a stochastic output distribution. Repeated querying of the same question can yield meaningfully different responses in terms of content, emphasis, and completeness. Consequently, the concordance and safety scores reported here should be interpreted as a snapshot rather than a stable characterization of each model’s performance. Future studies would benefit from repeated querying protocols or temperature-controlled API access to quantify within-model variability alongside between-model differences.

High aggregate concordance does not preclude clinically meaningful errors at the individual question level: a model that scores 4.4–4.5 on average may still produce guideline-discordant or potentially harmful responses for specific topics—as illustrated by the 15–20% safety flag rate. This pattern suggests that LLMs may be well-suited as supplementary educational tools or first-pass information retrieval aids within ERAS pathways, but not as autonomous clinical decision-support systems. Clinician oversight, transparent reporting standards, and periodic re-evaluation as model versions evolve are essential preconditions for any clinical integration.

Limitations include only two clinician raters, a single query per question, and a single query date; model outputs, interfaces, and version labels may change over time. Model versions were not locked via API, meaning the underlying model weights could differ from those evaluated. The question bank, while comprehensive, may not fully capture the complexity of real-world multifactorial clinical decisions. The scoring system involves inherent subjectivity, reinforcing the importance of transparent and reproducible evaluation frameworks4.

5. Conclusions

Both models showed high concordance with ERAS colorectal recommendations in a structured, blinded evaluation. Nevertheless, safety-relevant deviations and variability in rater agreement highlight the need for clinician oversight and standardized evaluation prior to clinical adoption.