Machine Learning: Chenhao Tan University of Colorado Boulder LECTURE 8

Transcription

1 Machine Learning: Chenhao Tan University of Colorado Boulder LECTURE 8 Slides adapted from Jordan Boyd-Graber, Justin Johnson, Andrej Karpathy, Chris Ketelsen, Fei-Fei Li, Mike Mozer, Michael Nielson Machine Learning: Chenhao Tan Boulder 1 of 53

2 HW1 grades Most people did well! We are extra lenient this time Submit only your code in a zip file, with no folder structure Machine Learning: Chenhao Tan Boulder 2 of 53

3 Final projects Team formation Machine Learning: Chenhao Tan Boulder 3 of 53

4 Quiz 1 Which of the following statements is true? (Suppose that training data is large.) A. In training, K-nearest neighbors takes shorter time than neural networks. B. In training, K-nearest neighbors takes longer time than neural networks. C. In testing, K-nearest neighbors takes shorter time than neural networks. D. In testing, K-nearest neighbors takes longer time than neural networks. Machine Learning: Chenhao Tan Boulder 4 of 53

5 Quiz 2 How many parameters are there in the following feed-forward neural networks (assuming no biases)? x 1 x 2... x 100 h 1... h 50 A. 100 * * * 5 h 1... h 20 B. 100 * * * o 1... o 5 Machine Learning: Chenhao Tan Boulder 5 of 53

6 Quiz 3 How many parameters are there in the following convolutional neural networks? (assuming no biases, convolution with 4 filters, max pooling, ReLu, and finally a fully-connected layer) input image (10*10) 4@6*6 4@3*3 5*1 4 filters 5*5, stride 1 Max pooling 2*2, stride 2 Machine Learning: Chenhao Tan Boulder 6 of 53

7 Quiz 3 How many ReLU operations are performed on the forward pass? (assuming no biases, convolution with 4 filters, max pooling, ReLu, and finally a fully-connected layer) input image (10*10) 4@6*6 4@3*3 5*1 4 filters 5*5, stride 1 Max pooling 2*2, stride 2 Machine Learning: Chenhao Tan Boulder 6 of 53

8 Overview History lesson Deep learning in practice Improve stochastic gradient descent Unstable gradients Data preprocessing Weight Initialization Model Architecture Machine Learning: Chenhao Tan Boulder 7 of 53

9 History lesson Outline History lesson Deep learning in practice Improve stochastic gradient descent Unstable gradients Data preprocessing Weight Initialization Model Architecture Machine Learning: Chenhao Tan Boulder 8 of 53

10 History lesson History lesson 1962: Rosenblatt, Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms First neuron-based learning algorithm Could learning anything that you could program Machine Learning: Chenhao Tan Boulder 9 of 53

11 History lesson History lesson 1962: Rosenblatt, Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms First neuron-based learning algorithm Could learning anything that you could program 1969: Minsky & Papert, Perceptron: An Introduction to Computational Geometry First real complexity analysis Showed, in principle, many things that perceptrons can t learn to do Shut down any interest in neural networks Machine Learning: Chenhao Tan Boulder 9 of 53

12 History lesson History lesson 1986: Rumelhart, Hinton & Williams, Back Propagation Overcame many difficulties raised by Minsky et al. Neural networks wildly popular again (for a while) Machine Learning: Chenhao Tan Boulder 10 of 53

13 History lesson History lesson Shift to Bayesian Methods Best ideas from neural networks Direct statistical computing Support Vector Machines Nice mathematical properties Kernel tricks A few people still playing with NNs Bengio Hinton LeCun Machine Learning: Chenhao Tan Boulder 11 of 53

14 History lesson History lesson Core group continues to make improvements Various tricks to make NNs practical 2010-present BOOM! Machine Learning: Chenhao Tan Boulder 12 of 53

15 History lesson AlexNet Krizhevsky et al. [2012] Machine Learning: Chenhao Tan Boulder 13 of 53

16 History lesson History lesson Question: Why? What made neural networks amazing again? Massive datasets Computing power Algorithmic improvements Machine Learning: Chenhao Tan Boulder 14 of 53

17 History lesson History lesson Machine learning has a short history, but seems cyclic. What is next? Machine Learning: Chenhao Tan Boulder 15 of 53

18 Deep learning in practice Outline History lesson Deep learning in practice Improve stochastic gradient descent Unstable gradients Data preprocessing Weight Initialization Model Architecture Machine Learning: Chenhao Tan Boulder 16 of 53

19 Deep learning in practice Improve stochastic gradient descent Outline History lesson Deep learning in practice Improve stochastic gradient descent Unstable gradients Data preprocessing Weight Initialization Model Architecture Machine Learning: Chenhao Tan Boulder 17 of 53

20 Deep learning in practice Improve stochastic gradient descent Gradient descent Gradient descent w t+1 = w t η f (w t ) Machine Learning: Chenhao Tan Boulder 18 of 53

21 Deep learning in practice Improve stochastic gradient descent AdaGrad Not all features are created equal! Machine Learning: Chenhao Tan Boulder 19 of 53

22 Deep learning in practice Improve stochastic gradient descent AdaGrad Not all features are created equal! Gradient descent w t+1 = w t η f (w t ) Machine Learning: Chenhao Tan Boulder 19 of 53

23 Deep learning in practice Improve stochastic gradient descent AdaGrad Not all features are created equal! Gradient descent w t+1 = w t η f (w t ) Adagrad [Duchi et al., 2011] cache = cache + ( f (w t )) 2 1 w t+1 = w t η cache f (w t) Machine Learning: Chenhao Tan Boulder 19 of 53

24 Deep learning in practice Improve stochastic gradient descent Momentum Gradient descent w t+1 = w t η f (w t ) Machine Learning: Chenhao Tan Boulder 20 of 53

25 Deep learning in practice Improve stochastic gradient descent Momentum Gradient descent w t+1 = w t η f (w t ) Machine Learning: Chenhao Tan Boulder 21 of 53

26 Deep learning in practice Improve stochastic gradient descent Momentum Gradient descent w t+1 = w t η f (w t ) Physical interpretation: Imagine a object is falling, but it does not accumulate any velocity. Machine Learning: Chenhao Tan Boulder 21 of 53

27 Deep learning in practice Improve stochastic gradient descent Momentum Gradient descent w t+1 = w t η f (w t ) Physical interpretation: Imagine a object is falling, but it does not accumulate any velocity. Let us fix that! Machine Learning: Chenhao Tan Boulder 21 of 53

28 Deep learning in practice Improve stochastic gradient descent Momentum Gradient descent w t+1 = w t η f (w t ) Physical interpretation: Imagine a object is falling, but it does not accumulate any velocity. Let us fix that! Momentum v t+1 = βv t f (w t ) w t+1 = w t + ηv t+1 Machine Learning: Chenhao Tan Boulder 21 of 53

29 Deep learning in practice Improve stochastic gradient descent Momentum Image credit: Alec Radford Machine Learning: Chenhao Tan Boulder 22 of 53

30 Deep learning in practice Improve stochastic gradient descent More variations Adam [Kingma and Ba, 2014] RMSProp Machine Learning: Chenhao Tan Boulder 23 of 53

31 Deep learning in practice Improve stochastic gradient descent Dropout layer "randomly set some neurons to zero in the forward pass" [Srivastava et al., 2014] Machine Learning: Chenhao Tan Boulder 24 of 53

32 Deep learning in practice Improve stochastic gradient descent Dropout layer Forces the network to have a redundant representation Machine Learning: Chenhao Tan Boulder 25 of 53

33 Deep learning in practice Improve stochastic gradient descent Dropout layer Another interpretation: Dropout is training a large ensemble of models Machine Learning: Chenhao Tan Boulder 25 of 53

34 Deep learning in practice Unstable gradients Outline History lesson Deep learning in practice Improve stochastic gradient descent Unstable gradients Data preprocessing Weight Initialization Model Architecture Machine Learning: Chenhao Tan Boulder 26 of 53

35 Deep learning in practice Unstable gradients Unstable gradients x h 1 h 2... h L o Machine Learning: Chenhao Tan Boulder 27 of 53

36 Deep learning in practice Unstable gradients Unstable gradients x h 1 h 2... h L o L b 1 = σ (z 1 ) w 2 σ (z 2 ) w 3... σ (z L ) L a L Machine Learning: Chenhao Tan Boulder 27 of 53

37 Deep learning in practice Unstable gradients Unstable gradients sigmoid derivative Machine Learning: Chenhao Tan Boulder 28 of 53

38 Deep learning in practice Unstable gradients Vanishing gradients If we use Gaussian initialization for weights, w j N (0, 1), w j < 1 w j σ (z j ) < 1 4 L decay to zero exponentially b1 Machine Learning: Chenhao Tan Boulder 29 of 53

39 Deep learning in practice Unstable gradients Vanishing gradients ReLu ReLu derivative Machine Learning: Chenhao Tan Boulder 30 of 53

40 Deep learning in practice Unstable gradients Exploding gradients If w j = 100, w j σ (z j ) k > 1 Machine Learning: Chenhao Tan Boulder 31 of 53

41 Deep learning in practice Data preprocessing Outline History lesson Deep learning in practice Improve stochastic gradient descent Unstable gradients Data preprocessing Weight Initialization Model Architecture Machine Learning: Chenhao Tan Boulder 32 of 53

42 Deep learning in practice Data preprocessing Another subtle issue of activation function If all inputs x are positive, the gradients on w are either all positive or all negative. Machine Learning: Chenhao Tan Boulder 33 of 53

43 Deep learning in practice Data preprocessing Another subtle issue of activation function If all inputs x are positive, the gradients on w are either all positive or all negative. Zero-center the inputs! Machine Learning: Chenhao Tan Boulder 33 of 53

44 Deep learning in practice Data preprocessing Data preprocessing Original data Machine Learning: Chenhao Tan Boulder 34 of 53

45 Deep learning in practice Data preprocessing Data preprocessing Original data Zero-centered data (X X.mean(axis = 0)) Machine Learning: Chenhao Tan Boulder 34 of 53

46 Deep learning in practice Data preprocessing Data preprocessing Original data Zero-centered data (X X.mean(axis = 0)) Normalized data (X/ = np.std(x, axis = 0)) Machine Learning: Chenhao Tan Boulder 34 of 53

47 Deep learning in practice Data preprocessing Data preprocessing Original data Zero-centered data (X X.mean(axis = 0)) Normalized data (X/ = np.std(x, axis = 0)) PCA, whitening Machine Learning: Chenhao Tan Boulder 34 of 53

48 Deep learning in practice Data preprocessing Batch normalization Why only for the input data? [Ioffe and Szegedy, 2015] Consider a batch of activations at some layer. Make each dimension unit gaussian: â k = ak E[a k ] Var[a k ] Machine Learning: Chenhao Tan Boulder 35 of 53

49 Deep learning in practice Data preprocessing Batch normalization Reduces internal covariant shift Reduces the dependence of gradients o the scale of the parameters or their initial values Allows higher learning rates and use of saturating nonlinearities Reduce the need for dropout (maybe) Machine Learning: Chenhao Tan Boulder 36 of 53

50 Deep learning in practice Data preprocessing Batch normalization During training, use batch mean and batch variance; during testing use empirical mean and variance on training data Machine Learning: Chenhao Tan Boulder 36 of 53

51 Deep learning in practice Data preprocessing Batch normalization Add batch normalization before nonlinear activation or after nonlinear activation? master/batchnorm.md Machine Learning: Chenhao Tan Boulder 37 of 53

52 Deep learning in practice Weight Initialization Outline History lesson Deep learning in practice Improve stochastic gradient descent Unstable gradients Data preprocessing Weight Initialization Model Architecture Machine Learning: Chenhao Tan Boulder 38 of 53

53 Deep learning in practice Weight Initialization Non-convexity Machine Learning: Chenhao Tan Boulder 39 of 53

54 Deep learning in practice Weight Initialization Weight initialization Old idea: W = 0, what happens? Machine Learning: Chenhao Tan Boulder 40 of 53

55 Deep learning in practice Weight Initialization Weight initialization Old idea: W = 0, what happens? There is no source of asymmetry. (Every neuron looks the same and leads to a slow start.) Machine Learning: Chenhao Tan Boulder 40 of 53

56 Deep learning in practice Weight Initialization Weight initialization Old idea: W = 0, what happens? There is no source of asymmetry. (Every neuron looks the same and leads to a slow start.) δ L = a LL σ (z L ) # Compute δ s on output layer For l = L,..., 1 L = δ l (a l 1 ) T # Compute weight derivatives W l L b l = δl # Compute bias derivatives δ l 1 = ( W l) T δ l σ (z l 1 ) # Back prop δ s to previous layer Machine Learning: Chenhao Tan Boulder 40 of 53

57 Deep learning in practice Weight Initialization Weight initialization First idea: small random numbers, W N (0, 0.01) Machine Learning: Chenhao Tan Boulder 41 of 53

58 Deep learning in practice Weight Initialization Weight initialization Var(z) = Var( i w i x i ) = nvar(w i )Var(x i ) Machine Learning: Chenhao Tan Boulder 42 of 53

59 Deep learning in practice Weight Initialization Weight initialization Xavier initialization [Glorot and Bengio, 2010] W N (0, 2 n in + n out ) Machine Learning: Chenhao Tan Boulder 43 of 53

60 Deep learning in practice Weight Initialization Weight initialization Xavier initialization [Glorot and Bengio, 2010] Does not work for ReLU W N (0, 2 n in + n out ) Machine Learning: Chenhao Tan Boulder 43 of 53

61 Deep learning in practice Weight Initialization Weight initialization He initialization [He et al., 2015] W N (0, 2 n in ) Machine Learning: Chenhao Tan Boulder 44 of 53

62 Deep learning in practice Weight Initialization Weight initialization This is an actively research area and next great idea may come from you! Machine Learning: Chenhao Tan Boulder 45 of 53

63 Deep learning in practice Model Architecture Outline History lesson Deep learning in practice Improve stochastic gradient descent Unstable gradients Data preprocessing Weight Initialization Model Architecture Machine Learning: Chenhao Tan Boulder 46 of 53

64 Deep learning in practice Model Architecture ResNet How to train a neural network with 100 layers? [He et al., 2016] Machine Learning: Chenhao Tan Boulder 47 of 53

65 Deep learning in practice Model Architecture ResNet Why is it hard to train a large number of layers? Machine Learning: Chenhao Tan Boulder 48 of 53

66 Deep learning in practice Model Architecture ResNet Simple solution: Machine Learning: Chenhao Tan Boulder 49 of 53

67 Deep learning in practice Model Architecture ResNet Machine Learning: Chenhao Tan Boulder 50 of 53

68 Deep learning in practice Model Architecture ResNet Machine Learning: Chenhao Tan Boulder 51 of 53

69 Deep learning in practice Model Architecture References (1) John Duchi, Elad Hazan, and Yoram Singer. Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. J. Mach. Learn. Res., 12: , URL Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Teh and Mike Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pages , Chia Laguna Resort, Sardinia, Italy, May PMLR. URL Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In The IEEE International Conference on Computer Vision (ICCV), December Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages , Lille, France, Jul PMLR. URL Machine Learning: Chenhao Tan Boulder 52 of 53

70 Deep learning in practice Model Architecture References (2) Diederik P Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In Proceedings of ICLR, Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet Classification with Deep Convolutional Neural Networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems, Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15: , URL Machine Learning: Chenhao Tan Boulder 53 of 53