Regularized Linear Models in Stacked Generalization

Transcription

1 Regularized Linear Models in Stacked Generalization Sam Reid and Greg Grudic Department of Computer Science University of Colorado at Boulder USA June 11, 2009 Reid & Grudic (Univ. of Colo. at Boulder) Regularized Linear Models in Stacking June 11, / 33

2 How to combine classifiers? Which classifiers? How to combine? Adaboost, Random Forest prescribe classifiers and combiner We want L 1000 heterogeneous classifiers Vote/Average/Forward Stepwise Selection/Linear/Nonlinear? Our combiner: Regularized Linear Model Reid & Grudic (Univ. of Colo. at Boulder) Regularized Linear Models in Stacking June 11, / 33

3 Outline 1 Introduction How to combine classifiers? 2 Model Stacked Generalization StackingC Linear Regression and Regularization 3 Experiments Setup Results Discussion Reid & Grudic (Univ. of Colo. at Boulder) Regularized Linear Models in Stacking June 11, / 33

4 Outline 1 Introduction How to combine classifiers? 2 Model Stacked Generalization StackingC Linear Regression and Regularization 3 Experiments Setup Results Discussion Reid & Grudic (Univ. of Colo. at Boulder) Regularized Linear Models in Stacking June 11, / 33

5 Stacked Generalization Combiner is produced by a classification algorithm Training set = base classifier predictions on unseen data + labels Learn to compensate for classifier biases Linear and nonlinear combiners What classification algorithm should be used? Reid & Grudic (Univ. of Colo. at Boulder) Regularized Linear Models in Stacking June 11, / 33

6 Stacked Generalization - Combiners Wolpert, 1992: relatively global, smooth combiners Ting & Witten, 1999: linear regression combiners Seewald, 2002: low-dimensional combiner inputs Reid & Grudic (Univ. of Colo. at Boulder) Regularized Linear Models in Stacking June 11, / 33

7 Problems Caruana et al., 2004: Stacking performs poorly because regression overfits dramatically when there are 2000 highly correlated input models and only 1k points in the validation set. How can we scale up stacking to a large number of classifiers? Our hypothesis: regularized linear combiner will reduce variance prevent overfitting increase accuracy Reid & Grudic (Univ. of Colo. at Boulder) Regularized Linear Models in Stacking June 11, / 33

8 Posterior Predictions in Multiclass Classification p y(x) x Classification with d = 4, k = 3 Reid & Grudic (Univ. of Colo. at Boulder) Regularized Linear Models in Stacking June 11, / 33

9 Ensemble Methods for Multiclass Classification ŷ y (x 1, x 2 ) x 1 x 2 y 1 (x) y 2 (x) Multiple classifier system with 2 classifiers x Reid & Grudic (Univ. of Colo. at Boulder) Regularized Linear Models in Stacking June 11, / 33

10 Stacked Generalization ŷ y (x ) x y 1 (x) y 2 (x) Stacked generalization with 2 classifiers x Reid & Grudic (Univ. of Colo. at Boulder) Regularized Linear Models in Stacking June 11, / 33

11 Classification via Regression ŷ y A (x ) y B (x ) y C (x ) x y 1 (x) y 2 (x) Stacking using Classification via Regression x Reid & Grudic (Univ. of Colo. at Boulder) Regularized Linear Models in Stacking June 11, / 33

12 StackingC ŷ y A (x A ) y B (x B ) y C (x C ) x y 1 (x) y 2 (x) StackingC, class-conscious stacked generalization x Reid & Grudic (Univ. of Colo. at Boulder) Regularized Linear Models in Stacking June 11, / 33

13 Linear Models Linear model for use in Stacking or StackingC ŷ = d i=1 β ix i + β 0 Least Squares: L = y X β 2 Problems: High variance Overfitting Ill-posed problem Poor accuracy Reid & Grudic (Univ. of Colo. at Boulder) Regularized Linear Models in Stacking June 11, / 33

14 Regularization Increase bias a little, decrease variance a lot Constrain weights reduce flexibility prevent overfitting Penalty terms in our studies: Ridge Regression: L = y X β 2 + λ β 2 Lasso Regression: L = y X β 2 + λ β 1 Elastic Net Regression: L = y X β 2 + λ β 2 + (1 λ) β 1 Lasso/Elastic Net produce sparse models Reid & Grudic (Univ. of Colo. at Boulder) Regularized Linear Models in Stacking June 11, / 33

15 Model About 1000 base classifiers making probabilistic predictions Stacked Generalization to create combiner StackingC to reduce dimensionality Convert multiclass to regression Use linear regression Regularization on the weights Reid & Grudic (Univ. of Colo. at Boulder) Regularized Linear Models in Stacking June 11, / 33

16 Model About 1000 base classifiers making probabilistic predictions Stacked Generalization to create combiner StackingC to reduce dimensionality Convert multiclass to regression Use linear regression Regularization on the weights Reid & Grudic (Univ. of Colo. at Boulder) Regularized Linear Models in Stacking June 11, / 33

17 Outline 1 Introduction How to combine classifiers? 2 Model Stacked Generalization StackingC Linear Regression and Regularization 3 Experiments Setup Results Discussion Reid & Grudic (Univ. of Colo. at Boulder) Regularized Linear Models in Stacking June 11, / 33

18 Datasets Table: Datasets and their properties Dataset Attributes Instances Classes balance-scale glass letter mfeat-morphological optdigits sat-image segment vehicle waveform yeast Reid & Grudic (Univ. of Colo. at Boulder) Regularized Linear Models in Stacking June 11, / 33

19 Base Classifiers About 1000 base classifiers for each problem 1 Neural Network 2 Support Vector Machine (C-SVM from LibSVM) 3 K-Nearest Neighbor 4 Decision Stump 5 Decision Tree 6 AdaBoost.M1 7 Bagging classifier 8 Random Forest (Weka) 9 Random Forest (R) Reid & Grudic (Univ. of Colo. at Boulder) Regularized Linear Models in Stacking June 11, / 33

20 Results select vote average sg sg sg best linear lasso ridge balance glass letter mfeat optdigits sat-image segment vehicle waveform yeast Reid & Grudic (Univ. of Colo. at Boulder) Regularized Linear Models in Stacking June 11, / 33

21 Statistical Analysis Pairwise Wilcoxon Signed-Rank Tests Ridge outperforms unregularized at p Lasso outperforms unregularized at p Validates hypothesis: regularization improves accuracy Ridge outperforms lasso at p Dense techniques outperform sparse techniques Ridge outperforms Select-Best at p Properly trained model better than single best Reid & Grudic (Univ. of Colo. at Boulder) Regularized Linear Models in Stacking June 11, / 33

22 Baseline Algorithms Average outperforms Vote at p Probabilistic predictions are valuable Select-Best outperforms Average at p Validation/training is valuable Reid & Grudic (Univ. of Colo. at Boulder) Regularized Linear Models in Stacking June 11, / 33

23 Subproblem/Overall Accuracy - I RMSE Ridge Parameter..... Reid & Grudic (Univ. of Colo. at Boulder) Regularized Linear Models in Stacking June 11, / 33

24 Subproblem/Overall Accuracy - II Accuracy Ridge Parameter Reid & Grudic (Univ. of Colo. at Boulder) Regularized Linear Models in Stacking June 11, / 33

25 Subproblem/Overall Accuracy - III Accuracy RMSE on Subproblem 1... Reid & Grudic (Univ. of Colo. at Boulder) Regularized Linear Models in Stacking June 11, / 33

26 Accuracy for Elastic Nets Accuracy alpha=0.95 alpha=0.5 alpha=0.05 select-best Penalty Figure: Overall accuracy on sat-image with various parameters for elastic-net. Reid & Grudic (Univ. of Colo. at Boulder) Regularized Linear Models in Stacking June 11, / 33

27 Partial Ensemble Selection Sparse techniques perform Partial Ensemble Selection Choose from classifiers and predictions Allow classifiers to focus on subproblems Example: Benefit from a classifier good at separating A from B but poor at A/C, B/C Reid & Grudic (Univ. of Colo. at Boulder) Regularized Linear Models in Stacking June 11, / 33

28 Partial Selection Log Lambda Class Log Lambda Class Log Lambda Class Figure: Coefficient profiles for the first three subproblems in StackingC for the sat-image dataset with elastic net regression at α = Reid & Grudic (Univ. of Colo. at Boulder) Regularized Linear Models in Stacking June 11, / 33

29 Selected Classifiers Classifier red cotton grey damp veg v.damp total adaboost ann ann ann ann knn Table: Selected posterior probabilities and corresponding weights for the sat-image problem for elastic net StackingC with α = 0.95 for the 6 models with highest total weights. Reid & Grudic (Univ. of Colo. at Boulder) Regularized Linear Models in Stacking June 11, / 33

30 Conclusions Regularization is essential in Linear StackingC Trained linear combination outperforms Select-Best Dense combiners outperform sparse combiners Sparse models allow classifiers to specialize in subproblems Reid & Grudic (Univ. of Colo. at Boulder) Regularized Linear Models in Stacking June 11, / 33

31 Future Work Examine full Bayesian solutions Constrain coefficients to be positive Choose a single regularizer for all subproblems Reid & Grudic (Univ. of Colo. at Boulder) Regularized Linear Models in Stacking June 11, / 33

32 Acknowledgments PhET Interactive Simulations Turing Institute UCI Repository University of Colorado at Boulder Reid & Grudic (Univ. of Colo. at Boulder) Regularized Linear Models in Stacking June 11, / 33

33 Questions? Questions? Reid & Grudic (Univ. of Colo. at Boulder) Regularized Linear Models in Stacking June 11, / 33