十年旧CNN在皮肤癌诊断上超越前沿LLM
论证要点
- 作者用200个活检证实的皮肤病变（100个需处理、100个可安全出院）测试GPT-6 Astra、Claude Fable 5.1和微调后的Inception v3，CNN正确诊断125个（62%），两个LLM仅84和85个（42%）。
- CNN在黑色素瘤识别上更准（30个中识别13个，LLM为8和11），在良性病变识别上优势更大（痣和脂溢性角化病），但LLM在光化性角化病上表现更好。
- 两个LLM有“偏好答案”问题：GPT-6 Astra将75个病变误判为基底细胞癌（实际25个），Claude Fable 5.1将53个误判为光化性角化病（实际20个）。
- CNN故意选用过时架构（Inception v3），仅用单张H100微调20分钟，且训练数据已排除测试集，证明专用小模型在特定任务上可超越通用大模型。
- 作者承认样本量小（200个病变、30个黑色素瘤）、测试集均为活检病例（更难）、LLM可能见过公开数据集（若如此反而对LLM有利），且CNN并非最先进技术。
- 当以患者身份提问（仅临床照片、问“需要看医生吗”），两个LLM几乎对所有病例都建议就医，包括良性病变，显示其保守倾向。
对实践的启示
对AI应用开发者而言，在垂直领域（如医疗影像）中，针对特定任务微调的小型专用模型可能比通用大模型更准确、更高效，且可本地运行。不要盲目迷信前沿LLM的通用能力，应针对具体任务做基准测试。
作者承认的反方与局限
作者承认样本量小、测试集均为活检病例（更难）、LLM可能见过公开数据集（若如此反而对LLM有利），且CNN并非最先进技术，最先进的皮肤癌诊断AI使用图像变换器集成，性能更好。
原文摘引
A ten-year-old network, fine-tuned for about twenty minutes, beat both of them at recognising skin cancer.
The CNN was right more often overall and on most diagnoses.
Each LLM had a favourite answer: GPT-6 Astra called 75 lesions basal cell carcinoma when 25 were, including 14 of the 30 melanomas.
▸ 展开原文全文（38 段，抓取自 Hacker News 首页）
A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis
I am a dermatologist. I tested the ability of GPT-6 Astra and Claude Fable 5.1 to diagnosis skin cancer and found that they were less accurate than I expected. I compared to a 10-year old CNN and to my surprise found that this was superior.
This is a melanoma.
I showed these two images - a clinical photograph and a dermoscopic (skin microscope) image - to GPT-6 Astra and Claude Fable 5.1, two of the most capable AI models available in October 2026. Both said it was a seborrhoeic keratosis, a harmless warty growth, and that the patient could be discharged. I then showed them to fine-tuned version of Inception v3, a convolutional neural network (CNN) first published by Google in 2015 and fine-tuned here on public dermatology images. The CNN got the right answer based on the dermoscopy view.
It was one of 200 biopsy-proven lesions I gave to all three. The CNN named the right diagnosis (out of 6 possiblities) for 125 of them (62%). The two frontier LLMs named it for 84 and 85 (42%). A ten-year-old network, fine-tuned for about twenty minutes, beat both of them at recognising skin cancer.
The test#
The lesions come from MILK10k, a public dataset released through the International Skin Imaging Collaboration in 2025, in which each lesion is photographed twice: a clinical close-up and a dermoscopic image taken through the illuminated magnifying lens dermatologists use. I drew 200 lesions at random, all confirmed by histopathology:
- 100 that should not be discharged: 30 melanomas, 25 basal cell carcinomas, 25 squamous cell carcinomas (including Bowen’s disease) and 20 actinic keratoses.
- 100 that are safe to discharge: 50 benign naevi (moles) and 50 seborrhoeic keratoses.
Every lesion was biopsied, including the benign ones, so these are harder to diagnose: a mole that somebody decided to cut out is one that worried a clinician.
The LLMs were given both images and nothing else, and asked, as a clinician might use them, for the most likely of the six diagnoses (or “unsure”), a confidence, a second choice, and whether the lesion could be safely discharged with no treatment, biopsy or follow-up. Both models got the same prompt at default settings, and neither refused a case.
The CNN is deliberately not state of the art. I used Inception v3 pretrained on everyday photographs and then fine-tuned all layers on 68,011 dermoscopic and clinical images from the ISIC Archive on a single H100 GPU. These 200 test lesions, and every other photograph of the same lesion or patient, were removed from its data (including the validation set) before training, so it had never seen them, and they were not used to choose the model or set its threshold. It scored both images of each lesion, combined them with most of the weight on the dermoscopic image, and chose from the same six diagnoses.
You can try the CNN yourself. Remarkably the model is small enough to run entirely in a phone browser using WebGPU. Nothing is uploaded, and its random sample button draws from these same 200 lesions.
Diagnosis: 62% against 42%#
The CNN was right more often overall and on most diagnoses. Allowing a second choice, it reached 173 of 200 against 134 and 138. It named 13 of the 30 melanomas against 8 and 11. The biggest gap was on the harmless lesions: it recognised 37 of 50 moles and 26 of 50 seborrhoeic keratoses, against 16 and 13 for GPT-6 Astra and 33 and 6 for Claude Fable 5.1. It was one short of GPT-6 Astra on basal cell carcinoma and behind both LLMs on actinic keratosis.
The models also failed differently. Each LLM had a favourite answer: GPT-6 Astra called 75 lesions basal cell carcinoma when 25 were, including 14 of the 30 melanomas, and Claude Fable 5.1 called 53 lesions actinic keratosis when 20 were. The CNN’s errors on melanoma went the other way: of the 17 it got wrong, it called 8 a naevus and 6 a seborrhoeic keratosis.
The results in one table#
| | GPT-6 Astra | Claude Fable 5.1 | CNN (Inception v3) |
|---|---|---|---|
| Correct diagnosis (of 200) | 84 (42%) | 85 (42%) | 125 (62%) |
| Correct in first or second choice | 134 | 138 | 173 |
| Melanoma as first choice (of 30) | 8 | 11 | 13 |
| Unsafe lesions kept (of 100) | 97 | 96 | 77 at its preset threshold; 98 or 99 when matched |
| Safe lesions discharged (of 100) | 32 | 22 | 76 at its preset threshold |
| Melanomas discharged (of 30) | 3 | 4 | 18 at its preset threshold; 2 or 1 when matched |
Caveats#
- Small numbers. Two hundred lesions are enough to show the pattern but not to rank close results, and 30 melanomas give wide uncertainty on the figure that matters most. Each LLM case was run once; answers vary between runs.
- Hard cases. Every lesion was biopsied. On ordinary images from the archive the CNN separated lesions that should and should not be discharged far better (area under the ROC curve 0.96, against 0.86 here).
- Contamination. MILK10k has been public since 2025, so the LLMs could have seen it in training; if anything that would have helped them. The CNN certainly had not.
- Not state of the art. Inception v3 is a baseline. State of the art skin cancer diagnosis AI uses ensembles of images transformers trained on more data, and the regulated products built on them perform considerably better.
Asked as a patient#
Asked as a patient instead, with the clinical photograph alone and the question “do I need to see a doctor?”, both LLMs advised seeing a doctor in almost every case, harmless lesions included: for 99 and 98 of a further 100 biopsy-proven melanomas.
Conclusion#
A small, old, specialised network using an outdated architecture recognises skin lesions far better than two frontier LLMs.
Run it yourself#
The cases, prompts, every LLM reply and the scoring code are in one download, and the CNN is here.
I designed the test and made the clinical rulings. The analysis and the first drafts were produced with Claude Code, Anthropic’s coding tool, which runs on a Claude model; one of the two LLMs tested is also Anthropic’s. Scoring was done by script against the histology.
Images are from the MILK10k dataset (MILK study team, International Skin Imaging Collaboration), used under a CC BY-NC licence.