image/svg+xml
This is a LinkMap: A 2D layout of links, controlled by a wiki. [Lecture 1](https://www.youtube.com/watch?v=qMp3s7D_8Xw&list=PLemsnf33Vij4eFWwtoQCrt9AHjLe3uo9_&index=1)<!--361|78|deepskyblue--> [Lecture 2](https://www.youtube.com/watch?v=0X610-2cH8I&list=PLemsnf33Vij4eFWwtoQCrt9AHjLe3uo9_&index=3)<!--391|462| deepskyblue--> [Lecture 3](https://www.youtube.com/watch?v=1uLzkX19zPc&list=PLemsnf33Vij4eFWwtoQCrt9AHjLe3uo9_&index=4)<!--353|805|deepskyblue--> [Lecture 4](https://www.youtube.com/watch?v=ens7rnhk0e0&list=PLemsnf33Vij4eFWwtoQCrt9AHjLe3uo9_&index=5)<!--804|235|deepskyblue--> [Lecture 4](https://www.youtube.com/watch?v=ens7rnhk0e0&list=PLemsnf33Vij4eFWwtoQCrt9AHjLe3uo9_&index=5)<!--519|537|deepskyblue--> [Lecture 5](https://www.youtube.com/watch?v=Mx0GMr_3Z4U&list=PLemsnf33Vij4eFWwtoQCrt9AHjLe3uo9_&index=6)<!--770|861|deepskyblue--> [Lecture 5](https://www.youtube.com/watch?v=Mx0GMr_3Z4U&list=PLemsnf33Vij4eFWwtoQCrt9AHjLe3uo9_&index=6)<!--1251|128|deepskyblue--> [Lecture 6](https://www.youtube.com/watch?v=vWg9cWUqSnI&list=PLemsnf33Vij4eFWwtoQCrt9AHjLe3uo9_&index=7)<!--1252|160|deepskyblue--> [Lecture 6](https://www.youtube.com/watch?v=vWg9cWUqSnI&list=PLemsnf33Vij4eFWwtoQCrt9AHjLe3uo9_&index=7)<!--1247|461|deepskyblue--> [Lecture 7](https://www.youtube.com/watch?v=JEhHvY5k3Q0&list=PLemsnf33Vij4eFWwtoQCrt9AHjLe3uo9_&index=8)<!--1246|495|deepskyblue--> [Lecture 8](https://www.youtube.com/watch?v=hz7DBfGLDV4&list=PLemsnf33Vij4eFWwtoQCrt9AHjLe3uo9_&index=9)<!--1246|531|deepskyblue--> [Lecture 9](https://www.youtube.com/watch?v=HU6HtpQL_YA&list=PLemsnf33Vij4eFWwtoQCrt9AHjLe3uo9_&index=10)<!--1226|737|deepskyblue--> [58 Rosenblatt](https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.335.3398&rep=rep1&type=pdf)<!--235|184|darkorange--> The perceptron: the original artificial neural network. [Review 89 Hinton](http://www.cs.toronto.edu/~fritz/absps/clp.pdf)<!--232|218|darkorange--> "Connectionist Learning Procedures". This review covers the state of the art in 1989, including backpropagation, Hopfield models, Boltzmann machines, reinforcement learning, a first mention of autoencoders (self-supervision), some remarks on mutual information, and much more. Marvellous! [89 Cybenko](https://link.springer.com/article/10.1007/BF02551274)<!--235|308|darkorange--> The original proof of the fact that neural networks with at least one hidden layer can approximate arbitrary smooth functions (with sufficiently many neurons). [Review 89 Hinton](http://www.cs.toronto.edu/~fritz/absps/clp.pdf)<!--255|518|darkorange--> "Connectionist Learning Procedures". This review covers the state of the art in 1989, including backpropagation, Hopfield models, Boltzmann machines, reinforcement learning, a first mention of autoencoders (self-supervision), some remarks on mutual information, and much more. Marvellous! [76 Linnainmaa](https://link.springer.com/article/10.1007/BF01931367)<!--259|616|darkorange--> Original introduction of the backpropagation algorithm: "Taylor expansion of the accumulated rounding error". Discusses automatic differentiation. Not yet applied to the special case of neural networks. [85 Rumelhart et al](http://www.cs.utoronto.ca/~hinton/absps/naturebp.pdf)<!--259|657|darkorange--> Rumelhart, Hinton, Williams. The famous introduction of backpropagation for neural networks. Practical applications. Already includes gradient descent with 'momentum'. [80 Fukushima](https://www.cs.princeton.edu/courses/archive/spr08/cos598B/Readings/Fukushima1980.pdf)<!--725|285|darkorange--> Neocognitron: The grandparent of all convolutional neural networks. Far ahead of its time: a multilayer (deep) convolutional structure, with unsupervised learning. Inspired by the human visual cortex. [89 LeCun et al](https://papers.nips.cc/paper/1989/file/53c3bce66e43be4f209556518c2fcb54-Paper.pdf)<!--727|324|darkorange--> Deep CNN for digit recognition: First to use backpropagation to train a multilayer CNN. Here applied to MNIST-style dataset of handwritten digits. [12 Krizhevsky et al](https://dl.acm.org/doi/10.1145/3065386)<!--727|370|darkorange--> The summary of the famous deep CNN that won the 2012 ImageNet competition, using ReLU and dropout in a deep CNN. [89 Baldi et al](https://www.sciencedirect.com/science/article/abs/pii/0893608089900142?via%3Dihub)<!--544|586|darkorange--> They introduce a linear autoencoder and show that it is doing PCA. [06 Hinton et al](https://www.cs.toronto.edu/~hinton/science.pdf)<!--655|586| darkorange--> Deep autoencoders & pretraining: Uses restricted Boltzmann machine for pretraining, proposes layer-wise pretraining for deep architectures. Examples from different applications. [Review 12 Bengio et al](https://arxiv.org/abs/1206.5538)<!--572|623|darkorange--> Representation learning, i.e. unsupervised learning of useful features. Covers the various autoencoder types, PCA, sparse coding, distributed representations, and more. [08 van der Maaten et al](http://www.cs.toronto.edu/~hinton/absps/tsne.pdf)<!--681|935|darkorange--> t-SNE: Everyone's favourite nonlinear dimensionality reduction technique for visualization. [95 Hochreiter et al](http://people.idsia.ch/~juergen/FKI-207-95ocr.pdf)<!--1138|212|darkorange--> Long Short-Term Memory (LSTM): The classic paper describing how to get rid of exploding/vanishing gradients in recurrent networks using gated units. (Technical report version; there is a 97 version) [13 Mikolov et al](https://arxiv.org/abs/1310.4546)<!--1063|295|darkorange--> Word vectors: This paper shows you can indeed do basic arithmetic on word vectors and get meaningful results. [92 Watkins et al](http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=98ECF011CFFA9E02C015F96A1DF3A471?doi=10.1.1.466.7149&rep=rep1&type=pdf)<!--1104|534|darkorange--> Introduces Q Learning [92 Williams](https://link.springer.com/article/10.1007/BF00992696)<!--1104|498|darkorange--> Introduces REINFORCE (policy gradient) [85 Sejnowski](https://onlinelibrary.wiley.com/doi/abs/10.1207/s15516709cog0901_7)<!--1102|778|darkorange--> The original paper that introduced restricted Boltzmann machines as a tool for learning to sample from an observed probability distribution. Explains the connections to statistical physics and the learning procedure. [Lecture Notes: 20 Marquardt](https://arxiv.org/abs/2101.01759)<!--1039|891|deepskyblue--> Lecture Notes for the 2019 Les Houches Lectures. [Book: 16 Goodfellow et al](https://www.deeplearningbook.org)<!--1037|949|steelblue--> Complete book on deep learning, online. [Book: 15 Nielsen](http://neuralnetworksanddeeplearning.com)<!--1037|978|steelblue--> A short online book introducing the very basics. [Online course: Machine Learning for Physicists](https://pad.gwdg.de/s/Machine_Learning_For_Physicists_2021)<!--948|1009| deepskyblue--> Free online course, including github repository, slides, videos (see links to lecture videos on this link map page here). LinkMap Code by Florian Marquardt, 2020 (MIT license). See florianmarquardt.github.io. Contents by Florian Marquardt (Creative Commons license).