Comparative Performance of Frontier Large Language Models for Extracting High-Risk Pathologic Features from Unstructured Gastrointestinal Oncology Reports: A Systematic Benchmarking Study with Human and Traditional NLP Baselines

<title>Abstract</title> <p>Background: Frontier large multimodal models (LMMs) are increasingly proposed for oncologic diagnostic tasks, yet rigorous comparative benchmarking on clinically consequential, high-variability tasks is lacking. Perineural invasion (PNI) is a critical prognostic determinant in gastrointestinal oncology that directly modifies adjuvant radiation therapy decisions, yet it is frequently buried within the microscopic description of pathology reports rather than the diagnostic summary, making automated extraction non-trivial. We systematically benchmarked GPT-4o (gpt-4o-2024-08-06), Gemini 2.5 Pro (gemini-2.5-pro-exp-03-25), Claude 3.7 Sonnet (claude-3-7-sonnet-20250219), and DeepSeek-R1 (deepseek-reasoner) against time-pressured human and fine-tuned BioBERT baselines for PNI status extraction. Methods: We curated 445 de-identified gastrointestinal pathology reports (colorectal 48.5%, gastric 27.8%, pancreatic 23.7%), split 70/10/20 (train/val/test; random seed 42, stratified by PNI status). Expert-panel consensus (κ = 0.85, 95% CI: 0.78–0.92) served as ground truth; 34 reports (7.6%) flagged as Gold Standard Uncertainty were excluded from a pre-specified sensitivity analysis. LMMs were queried via their respective APIs (temperature = 0.0; max_tokens = 500) using a standardised two-stage chain-of-thought plus structured JSON protocol. Baselines: three residents with 90-second per-case time limit (human); fine-tuned BioBERT-base-cased-v1.2 on 70% training data (traditional NLP). Primary outcomes: accuracy, sensitivity, specificity, AUC-ROC, F1; secondary: BLEU, ROUGE-L, response latency, hallucination rate, and a novel Failure Mode Taxonomy. Bootstrap 95% CIs used 10,000 iterations; pairwise McNemar's tests were corrected by Benjamini-Hochberg false discovery rate (FDR). Results: Gemini 2.5 Pro achieved the highest overall accuracy (95.8%) and GPT-4o closely followed (94.2% (95% CI: 87.7–100.0)), significantly outperforming all LMM comparators (all FDR-corrected p &lt; 0.05) and both baselines (vs. residents: +9.8%, p &lt; 0.001; vs. BioBERT F1: +0.090, p = 0.012). Claude 3.7 Sonnet achieved the highest sensitivity (86.1%) but lower specificity (76.9%). DeepSeekR1 showed the longest mean latency (9.6 ± 2.1 s) and highest hallucination rate (4.2%). The Failure Mode Taxonomy revealed qualitatively distinct error signatures: Negation Failure predominated for Gemini (7.2% of test cases) while Hedging Ambiguity predominated for GPT-4o and Claude 3.7 Sonnet (12.5% and 12.0%). Conclusions: Gemini 2.5 Pro achieved the highest accuracy (95.8%) and GPT-4o achieved the highest AUC-ROC (0.908) for PNI extraction from gastrointestinal pathology reports, with both models significantly outperforming human and traditional NLP baselines. The Failure Mode Taxonomy enables model selection tailored to institutional documentation patterns, moving beyond aggregate rankings to actionable deployment guidance. All prompt templates, code, and anonymized model outputs are available on [GitHub].</p>