LeNet-5
The 1998 architecture reproduced faithfully in PyTorch — trainable subsampling coefficients, the C3 connection table, RBF output and MAP loss — verified at exactly 60,000 parameters. With 21 questions the paper raised and how each one resolved.
PAPER
Gradient-Based Learning Applied to Document Recognition
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner
Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324 · 1998
PAPER NOTES
Select a section to read the notes one part at a time.
The problem the paper was solving
What this paper argued
- The feature extractor should be learned, not hand-designed. Deciding by hand what to measure and training only the classifier is the bottleneck.
- Tolerance belongs in the architecture rather than the training set. Local receptive fields, shared weights and spatial subsampling build in translation and distortion tolerance structurally.
- Recognition systems should be trained globally, end to end, rather than assembled from separately optimised modules.
In 1998, the standard approach to recognising handwritten characters was to hand-design a feature extractor — someone decided what to measure, by hand — and feed its output to a trainable classifier. The paper's argument is that this division is the bottleneck, and that the feature extractor should be learned too.
The specific obstacle was that a fully-connected network has no structural notion of position. It holds a separate weight for every pixel location, so a digit trained in the centre of the frame shares nothing with the same digit shifted two pixels right. The network can be taught shift tolerance by brute force, with enough examples, but nothing in its architecture provides it.
The paper's answer is three structural constraints — local receptive fields, shared weights, and spatial subsampling — which build a degree of translation and distortion tolerance into the architecture rather than into the training set. LeNet-5 is the demonstration, and it was deployed commercially reading bank cheques.