A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis
A dermatologist's open test on 200 biopsied lesions: a fine-tuned 2015 Inception v3 scored 62% top-1 vs 42% for GPT-6 Astra and Claude Fable 5.1.
On October 25, 2026, Dr. Magnus Lynch, a consultant dermatologist and Mohs surgeon at Guy’s Hospital, published a head-to-head test on his LynchLab site: 200 biopsy-confirmed skin lesions, each handed to GPT-6 Astra, Claude Fable 5.1, and an Inception v3 CNN first published by Google in 2015 and fine-tuned on 68,011 ISIC dermatology images. Per his results table, the 2015 architecture named the right diagnosis as its top pick for 125 of 200 lesions (62%); the two frontier LLMs managed 84 and 85 (42%). The cases, prompts, every model reply and the scoring script ship in one download.
Key points
- Accuracy: Inception v3 scored 125/200 (62%) top-1 across six diagnoses, against 84 (42%) for GPT-6 Astra and 85 (42%) for Claude Fable 5.1. On the 30 melanomas, top-1 was 13 for the CNN, 11 for Claude, 8 for GPT-6.
- Discharge safety: at its preset threshold the CNN released 18 of 30 melanomas — worse than the LLMs’ 3 and 4. Once its threshold was matched to the LLMs’ sensitivity, it released 1–2 melanomas and kept 98–99 of 100 unsafe lesions.
- Reproducibility: all 200 cases, the prompts, every LLM reply and the scoring code are public; the MILK10k dataset has a DOI (10.34970/648456), and the CNN is small enough to run locally in a phone browser via WebGPU.
Background
This test has two installments. On October 11, 2026, Lynch published an LLM-only version on his personal site: both models hit 42% top-1, with the data pack already public. The October 25 LynchLab version adds the Inception v3 comparison and a browser demo, and the CNN immediately opens a 20-point lead over both LLMs. The author’s day job matters here: consultant dermatologist and Mohs surgeon at Guy’s Hospital, honorary reader at King’s College London. He also discloses that he ran the analysis with Claude Code and that one of the models under test is Anthropic’s — a conflict of interest worth weighing yourself.
The 200 lesions come from MILK10k, a dataset the International Skin Imaging Collaboration (ISIC) released in 2025, and every one carries histopathology confirmation. The LLM-only version already surfaced a calibration detail: GPT-6 Astra’s self-reported confidence averaged 82 when right and 79 when wrong, while Claude’s tracked its accuracy far more closely. Checking settled models against your own data has come up on this site before — the softmax numeric bugs live one layer down, in the frameworks; this one sits at the application layer with smaller samples and more direct consequences.
The facts
Per Lynch’s published method and results (2026-10-25): the 200 lesions split into 100 not safe to discharge (30 melanomas, 25 basal cell carcinomas, 25 squamous cell carcinomas, 20 actinic keratoses) and 100 safe to discharge (50 benign naevi, 50 seborrhoeic keratoses), each with a clinical photo and a dermoscopic image. Inception v3 was fine-tuned on all layers over 68,011 ISIC images on a single H100, with the test lesions and every image of the same patient removed beforehand. Each LLM case ran once, at default settings, under one prompt offering six diagnosis options plus “unsure”; a script scored replies against histology.
| Model | Top-1 of 200 | Melanoma top-1 of 30 |
|---|---|---|
| Inception v3 (2015 architecture, fine-tuned) | 125 (62%) | 13 (43%) |
| Claude Fable 5.1 | 85 (42%) | 11 (37%) |
| GPT-6 Astra | 84 (42%) | 8 (27%) |
With second choices allowed, the gap narrows but holds: 173/200 for the CNN versus 134 and 138 for the LLMs. Discharge decisions — send the patient home or refer for biopsy — fill a different table:
| Model | Melanomas discharged (/30) | Unsafe lesions kept (/100) | Safe lesions correctly discharged (/100) |
|---|---|---|---|
| Inception v3 (preset threshold) | 18 | 77 | 76 |
| Inception v3 (threshold matched) | 1–2 | 98–99 | — |
| GPT-6 Astra | 3 | 97 | 32 |
| Claude Fable 5.1 | 4 | 96 | 22 |
Read the direction carefully: at its preset threshold the CNN releases more melanomas than either LLM and is the least safe option on that row; only after threshold matching does it match or beat them, and its clear win is diagnostic accuracy. There is also a patient-perspective test: asked “do I need to see a doctor?” with only the clinical photo, both LLMs advised seeing one in nearly every case, including 99 and 98 of 100 additional melanomas — as a triage warning light, their conservatism is correct.
What others say
The Hacker News thread drew over 170 comments (posted by the author himself as sungam; around 330 points as of October 26). The loudest objection is “benchmaxxing”: to Legend2440, a fine-tuned model beating zero-shot generalists on its own training distribution is the expected outcome, and generalization to real-world photos is the open question. A recurring analogy holds that a calculator beating an LLM at arithmetic, or KataGo at Go, is no news at all; some practitioners add that 30 melanomas cannot carry a conclusion this size. The pushback is just as direct — training a task-specific model is standard practice, hardly benchmark gaming. Lynch himself replied that the CNN architecture was a deliberately dated baseline, “small enough to run using WebGPU in a phone browser for demonstration,” while clinical-grade systems use trial-validated ensembles of vision transformers; he had assumed that with tens of thousands of labeled skin-cancer images in the public internet, general image understanding would transfer.
Yash Thakker’s write-up on explainx.ai adds the calibration layer: both LLMs once labeled an ear melanoma a seborrhoeic keratosis and recommended discharge, and GPT-6 Astra put 95% confidence on a wrong basal cell carcinoma call. His quoted line: “A model that sounds sure while missing a melanoma is more dangerous than one that hedges.” Trusting evaluation numbers is a live theme here — Anthropic cut its internal evals off the live internet over reward hacking (Anthropic takes its evals offline), a different instance of the same problem.
Our take
For anyone wiring LLMs into a vertical product, the useful part of this study is the set of caveats Lynch attaches to it: 30 melanomas leave wide error bars; one run per case means different prompts or sampling settings could move results; all 200 lesions are post-biopsy hard cases, and the CNN’s AUC drops from 0.96 on ordinary archive images to 0.86 here; MILK10k has been public for nearly two years and may sit in LLM training data — if that contamination exists, it helped the LLMs, which makes the gap harder to explain away; and this CNN is far from clinical state of the art, where regulated products use vision-transformer ensembles that perform considerably better. Lynch’s own line: he expected the LLMs to do better.
What remains unknown is how any of this holds up on a real primary-care distribution, and whether LLMs recover the gap with multi-turn follow-up or richer prompts. Still, the default assumption that a stronger general model settles a narrow domain now has a downloadable counterexample attached, in medical imaging. The gap between leaderboard standing and real tasks is the same theme as StartLux topping the community decision index.
How to try it
The LynchLab page (lynchlab.net/llm-skin-cancer) hosts the whole test as one download: the 200 cases with ISIC image ids, the prompts, every reply from both models, and the scoring scripts. The MILK10k dataset carries DOI 10.34970/648456, with images under CC BY-NC. The CNN is small enough to run locally in a phone browser through WebGPU — nothing gets uploaded — and the page links the demo.