Convolutional neural network

In 1989 Yann LeCun and colleagues at AT&T Bell Laboratories built a network for handwritten digits that did not use a separate detector at every position. One neuron with a small window scans the image, and the states are stored in a feature map. They wrote that this scan is equivalent to a convolution with a small kernel, followed by a squashing function, and that every unit in the map is constrained to use the same weights. Sharing weights cuts the number of free parameters. It also gives a simple kind of shift invariance: move the input and the feature map shifts, but is otherwise unchanged.¹

The digits came from the US Postal Service. The network took a normalised image of an isolated zip-code digit and classified it, with little hand-built preprocessing. LeCun and colleagues reported a 1% error rate and about a 9% reject rate.¹

The same design was later scaled up to photographs. In 2012 Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton trained a deep convolutional network on the 1.2 million high-resolution images of the ImageNet LSVRC-2010 contest, sorted into 1,000 classes. It had five convolutional layers, some followed by max-pooling, and three fully connected layers, with 60 million parameters and 650,000 neurons. On the test set its top-1 and top-5 error rates were 37.5% and 17.0%, which they reported as considerably better than the previous best.²

A convolutional network is not a transformer. The transformer paper treats a convolutional layer as a different operation: one layer with a small kernel does not connect every position to every other position, whereas attention can.³ It is also not just any neural network trained on pictures. The shared local filter is the constraint LeCun's group used to keep the number of free parameters down.¹