Small CNN vs Frontier LLMs: Run a Fair Head-to-Head

A dermatologist at Guy's Hospital fine-tuned a 2015 image classifier for about twenty minutes and then pitted it against two frontier chat models on 200 biopsy-proven skin lesions. The old network got the first-choice diagnosis right 125 times. GPT-6 Astra got 84 and Claude Fable 5.1 got 85. The write-up hit the front page of Hacker News this week with 654 points, and the argument in the comments is the one every AI team eventually has: when do you reach for a frontier API, and when do you train something small?

This guide does not just retell the result. It shows you how to run the same kind of head-to-head on your own labeled data: a leak-free split, a fine-tuned Inception v3, a harness that asks a vision LLM the same question, and the part most people skip, which is comparing the two at the same operating point instead of comparing raw accuracy. By the end you will have a small repo you can point at any fixed-label image task.