ResNet, reproduced
The CIFAR-10 architecture of §4.2 rebuilt from the paper's description, checked against Table 6, and run twenty-two times to find which parts of the degradation claim survive reseeding.
PAPER
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · 2016
PAPER NOTES
Select a section to read the notes one part at a time.
The problem the paper was solving
A result that should not have been possible
By 2015 depth was the lever everyone was pulling, and VGG had just shown what pulling it was worth. Then it stopped working. Stacking more layers onto a working network did not merely stop helping — past a point it made the network worse, and worse on the data it had already been shown.
That last part is the whole paper. Test error rising with depth has an ordinary explanation, and §1 spends its first page ruling that explanation out. The claim that needs an answer is the awkward one: a 56-layer plain network fits the training set worse than a 20-layer one. Nothing about overfitting explains a model that cannot even memorise.
What this paper argued
- The failure is in the optimiser, not the model. A deeper network can represent everything a shallower one can — copy the shallow net and make the added layers identity — so if it ends up worse, the fault is that we could not find the solution, not that the solution is missing.
- Then make that fallback the default. Fit the residual F(x) = H(x) − x and add x back at the end, so "change nothing" costs zero effort and anything the layer does is a departure it had to earn.
- And make the shortcut free. On CIFAR the paper uses a parameter-free shortcut, which is what lets a plain network and its residual twin be compared at identical size.
I rebuilt the CIFAR-10 architecture of §4.2 from the paper's own description and ran it twenty-two times — four seeds on each of the four headline configurations, two more depths, and two longer schedules — to find out which parts of the claim survive being run more than once.
Most of them do. One does not, and it is the one everybody quotes.