AlexNet
The 2012 architecture reproduced to the paper rather than to torchvision — the two-GPU grouped convolutions, LRN before pooling, and the bias initialisation that most implementations drop. Verified at exactly 60,965,224 parameters, with 33 questions the paper raised and how each one resolved.
PAPER
ImageNet Classification with Deep Convolutional Neural Networks
Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton
Advances in Neural Information Processing Systems 25 (NeurIPS 2012) · 2012
PAPER NOTES
Select a section to read the notes one part at a time.
The problem the paper was solving
What this paper argued
- The data was not the missing piece. A model large enough to use 1.2 million images was, and nobody had trained one.
- Depth itself matters. Removing any single convolutional layer costs about 2% top-1, even though each holds a small fraction of the parameters.
- The engineering decisions are the contribution — ReLU for training speed, dropout where the capacity is, two GPUs because one did not fit.
By 2012, object recognition worked well on datasets of tens of thousands of images and did not scale beyond them. The prevailing view was that the limit was the data: with enough labelled examples, existing methods would keep improving. ImageNet supplied those examples — roughly 1.2 million labelled training images across 1,000 categories — and the paper's claim is that the data alone was not the missing piece. A model large enough to use it was, and nobody had trained one.
The obstacle was practical rather than theoretical. A network with 60 million parameters overfits a million images, and training it on 2012 hardware was infeasible at the sizes that worked. The paper's contribution is the set of engineering decisions that made it possible at all: ReLU activations so the network trains in a reasonable number of epochs, two GPUs working in parallel because one did not have the memory, dropout to control overfitting in the fully connected layers, and aggressive data augmentation.
The result ended the debate. Top-5 error of 15.3% in ILSVRC-2012 against 26.2% for the runner-up, which used the hand-engineered SIFT and Fisher Vector pipeline that represented the state of the art. The margin is what made the field switch.