Here is an interesting story from Prof. Yassine Benajiba:

architect Left: Australian termite colony; right: Basilica de la Sagrada Família

The church on the right is built from a blueprint: a blueprint which specify from steps A to steps Z of how we are going to build the church. If at any time, you ask Antoni Gaudí, what are you doing? He should be able to tell you the story of, for example, we are building this part to support this structure and how it contribute to the entire design. The church reflects centralized design.

The photo on the left is an Australian termite colony shared by one of the world’s leading evolutionary biologists Richard Dawkins in a tweet. Individual Australian termite execute simple tasks, but the aggregation of their work ends up as a cathedral like structure. If you stop a termite at any time and ask what is it doing? They will not be able to tell you: this is what we are doing now and how this form a ventilation system. How the termite built this structure is interesting because, unlike human, non of termites are able to articulate the final product of their work.

There is an analogy between the above story and deep learning. In traditional machine learning or expert system, researchers design the system and are able the explain individual parts, like the architect of the church. Deep learning functions more like termites: they contains millions of parameters and are trained by an optimizer. Asking how a neuron works in a neural network, is similar to asking a termites to articulate its job. A neuron has an input, an output and an activation function, but somehow the aggregate of neurons can detect patterns, build representations and make predictions.

I read many of the papers referenced below from Prof. Parijat Dube’s topic course on Deep Learning System Performance. Special thanks to his excellent teaching.

Topics