Model Combination in Multiclass Classification

Transcription

1 Model Combination in Multiclass Classification Sam Reid Advisors: Mike Mozer, Greg Grudic Department of Computer Science University of Colorado at Boulder USA April 5, 2010 Sam Reid Model Combination in Multiclass Classification 1/ 76

2 Multiclass Classification From examples, make multiclass predictions on unseen data. Applications in: Heartbeat arrythmia monitoring Protein structure classification Handwritten digit recognition Part of speech tagging Vehicle identification Many others... Our approach: model combination Sam Reid Model Combination in Multiclass Classification 2/ 76

3 Multiclass Classification: Example Heartbeat Arrhythmia Monitoring Data Set (truncated) age gender height weight BPM QRS 274 other wave class (yrs) (cm) (kg) duration (ms) characteristics 75 m Supraventricular Pre. 56 f Sinus bradycardy 54 m Right bundle block 55 m normal 75 m ?... Ventricular Pre. 13 m Left ventricule hyper. 40 f normal 49 f normal 44 m normal 50 f Right bundle block m ? 45 f ? Sam Reid Model Combination in Multiclass Classification 3/ 76

4 Model Combination Combine multiclass classifiers (e.g. KNN, Decision Trees, Random Forests) Voting Averaging Linear Nonlinear Combine binary classifiers (e.g. SVM, AdaBoost) to solve multiclass One vs. All Pairwise Classification Error Correcting Output Coding Sam Reid Model Combination in Multiclass Classification 4/ 76

5 Outline Regularization in Linear Combinations of Multiclass Classifiers Model Discussion Our Method Sam Reid Model Combination in Multiclass Classification 5/ 76

6 Model Outline Regularization in Linear Combinations of Multiclass Classifiers Model Discussion Our Method Sam Reid Model Combination in Multiclass Classification 6/ 76

7 Model Classifier Combination Goal: optimize predictions on test data Maintain diversity without sacrificing accuracy Train many classifiers with different algorithms/hyperparameters Combine with a linear combination function Ting & Witten, 1999 Seewald, 2002 Caruana et al., 2004 Sam Reid Model Combination in Multiclass Classification 7/ 76

8 Model Linear StackingC 1/2 Stacked Generalization Predictions on validation data are meta-training data Linear StackingC, class-conscious stacked generalization ˆp j ( x) = w ij y ij ( x) i=1..l ˆp j ( x) is the predicted probability for class c j w ij is the weight corresponding to classifier y i and class c j y ij ( x) is the i th classifier s output on class c j Training set = classifier predictions on unseen data + labels Determine weights using linear regression Sam Reid Model Combination in Multiclass Classification 8/ 76

9 Model Linear StackingC 2/2 ŷ y A (x A ) y B (x B ) y C (x C ) x y 1 (x) y 2 (x) x Sam Reid Model Combination in Multiclass Classification 9/ 76

10 Model Problems Caruana et al., 2004: Stacking [linear] performs poorly because regression overfits dramatically when there are 2000 highly correlated input models and only 1k points in the validation set. How can we scale up stacking to a large number of classifiers? Sam Reid Model Combination in Multiclass Classification 10/ 76

11 Model Problems Caruana et al., 2004: Stacking [linear] performs poorly because regression overfits dramatically when there are 2000 highly correlated input models and only 1k points in the validation set. How can we scale up stacking to a large number of classifiers? Our hypothesis: regularized linear combiner will reduce variance & prevent overfitting on indicator subproblems increase accuracy on multiclass problem Penalty terms in our studies: Ridge Regression: L = y X β 2 + λ β 2 Lasso Regression: L = y X β 2 + λ β 1 Elastic Net Regression: L = y X β 2 + (1 α) β 2 + α β 1 Sam Reid Model Combination in Multiclass Classification 10/ 76

12 Model Thesis Statement - Part I In linear combinations of multiclass classifiers, regularization significantly improves performance. Sam Reid Model Combination in Multiclass Classification 11/ 76

13 Model Multiclass Classification Data Sets Dataset Att.(numeric) Instances Classes balance-scale glass letter mfeat-morphological optdigits sat-image segment vehicle waveform yeast Sam Reid Model Combination in Multiclass Classification 12/ 76

14 Model Algorithms About 1000 base classifiers for each problem 1. Neural Network 2. Support Vector Machine (C-SVM from LibSVM) 3. K-Nearest Neighbor 4. Decision Stump 5. Decision Tree 6. AdaBoost.M1 7. Bagging classifier 8. Random Forest (Weka) 9. Random Forest (R) Sam Reid Model Combination in Multiclass Classification 13/ 76

15 Model Results: Average Accuracy Accuracy (%) sg-linear sg-lasso vote average select-best sg-ridge Sam Reid Model Combination in Multiclass Classification 14/ 76

16 Model Statistical Analysis Ridge outperforms unregularized at p Validates hypothesis: regularization improves accuracy Ridge outperforms lasso at p Dense better than sparse Voting and averaging all models not competitive Sam Reid Model Combination in Multiclass Classification 15/ 76

17 Model Multiclass Accuracy Binary Accuracy 1/3 RMSE Ridge Parameter..... Root mean squared error for the first (class-1) indicator subproblem in sat-image, over 10 folds of Dietterich s 5x2 CV. Sam Reid Model Combination in Multiclass Classification 16/ 76

18 Model Multiclass Accuracy Binary Accuracy 2/3 Accuracy Ridge Parameter Multiclass classification accuracy as a function of the regularization hyperparameter λ ridge. Sam Reid Model Combination in Multiclass Classification 17/ 76

19 Model Multiclass Accuracy Binary Accuracy 3/3 Accuracy RMSE on Subproblem 1 Accuracy vs RMSE on the first (class-1) indicator subproblem.... Multiclass Accuracy Binary Accuracy Sam Reid Model Combination in Multiclass Classification 18/ 76

20 Model Ridge More Effective than Lasso Accuracy alpha=0.95 alpha=0.5 alpha=0.05 select-best Penalty Overall accuracy on sat-image with various parameters for elastic-net. Sam Reid Model Combination in Multiclass Classification 19/ 76

21 Model Focus on Subproblems Choose from classifiers and predictions Allow classifiers to focus on subproblems Example: Benefit from a classifier that predicts well-calibrated probabilities for class A but has B & C backwards This advantage possible on multiclass classification but not binary classification, since k i=1 p i( x) = 1 Sam Reid Model Combination in Multiclass Classification 20/ 76

22 Model Sparse Linear Combinations Log Lambda Class Log Lambda Class Log Lambda Class Class Class Class Coefficient profiles for the first three subproblems in StackingC for the sat-image dataset with elastic net regression at α = 0.95 Sam Reid Model Combination in Multiclass Classification 21/ 76

23 Model Selected Classifiers Classifier red cotton grey damp veg v.damp total adaboost ann ann ann ann knn Weights (%) for the sat-image problem in elastic net StackingC with α = 0.95 for the 6 models with highest total weights. Sam Reid Model Combination in Multiclass Classification 22/ 76

24 Model Conclusions & Future Work Regularization is essential in linear combinations of multiclass classifiers Dense combiners outperform sparse combiners One-weight-per-output (instead of one-weight-per-classifier) allows classifiers to specialize in subproblems Future Work Bayesian treatment, Gaussian/Laplacian priors over weights Constrain coefficients to be positive This work published as: Regularized Linear Models in Stacked Generalization, Sam Reid and Greg Grudic, Multiple Classifier Systems, 2009, Springer LNCS Sam Reid Model Combination in Multiclass Classification 23/ 76

25 Discussion Outline Regularization in Linear Combinations of Multiclass Classifiers Model Discussion Our Method Sam Reid Model Combination in Multiclass Classification 24/ 76

26 Discussion Reducing Multiclass to Binary Some classifiers designed for binary (e.g. SVM, Adaboost) Transform multiclass set of binary problems Combine binary predictions predict multiclass A vs B,C in one-vs-all A vs C in all-pairs Sam Reid Model Combination in Multiclass Classification 25/ 76

27 Discussion Model Selection in Reducing Multiclass to Binary No Model Selection Dietterich and Bakiri, 1995 Allwein et al., 2000 Sam Reid Model Combination in Multiclass Classification 26/ 76

28 Discussion Model Selection in Reducing Multiclass to Binary No Model Selection Dietterich and Bakiri, 1995 Allwein et al., 2000 Shared Hyperparameters Rifkin uses greedy 1d hillclimbing, with OVA + LBD, Rifkin & Klautau, 2004 Model selection in LibSVM, Chang & Lin, 2001 Sam Reid Model Combination in Multiclass Classification 26/ 76

29 Discussion Model Selection in Reducing Multiclass to Binary No Model Selection Dietterich and Bakiri, 1995 Allwein et al., 2000 Shared Hyperparameters Rifkin uses greedy 1d hillclimbing, with OVA + LBD, Rifkin & Klautau, 2004 Model selection in LibSVM, Chang & Lin, 2001 Optimize Subproblems Independently Homogeneous, Friedman 1996 Heterogeneous, Szepannek et al Sam Reid Model Combination in Multiclass Classification 26/ 76

30 Discussion Model Selection in Reducing Multiclass to Binary No Model Selection Dietterich and Bakiri, 1995 Allwein et al., 2000 Shared Hyperparameters Rifkin uses greedy 1d hillclimbing, with OVA + LBD, Rifkin & Klautau, 2004 Model selection in LibSVM, Chang & Lin, 2001 Optimize Subproblems Independently Homogeneous, Friedman 1996 Heterogeneous, Szepannek et al Optimize the Joint Distribution Evolutionary search, de Souza et al., 2006, Lebrun et al., 2007 Sam Reid Model Combination in Multiclass Classification 26/ 76

31 Discussion Shared Hyperparameters vs Independent Optimization Shared Hyperparameters Optimizes to the target multiclass metric Increases bias and reduces variance for model selection Independent Optimization Accommodate subproblems with different structure Improved subproblem performance improved performance Sam Reid Model Combination in Multiclass Classification 27/ 76

32 Discussion Thesis Statement - Part II When solving a multiclass problem with a set of binary classifiers, it is more effective to constrain subproblems to use the same hyperparameters than to optimize each independently. Sam Reid Model Combination in Multiclass Classification 28/ 76

33 Discussion Multiclass Classification Data Sets 1/2 dataset classes numeric train test sampled-from anneal arrhythmia authorship autos cars collins dj ecoli eucalyptus halloffame Sam Reid Model Combination in Multiclass Classification 29/ 76

34 Discussion Multiclass Classification Data Sets 2/2 dataset classes numeric train test sampled-from hypothyroid letter mfeat-morphological optdigits page-blocks segment synthetic-control vehicle vowel waveform Sam Reid Model Combination in Multiclass Classification 30/ 76

35 Discussion Methods Reductions: {one-vs-all, all-pairs} {hamming, squared} Model selection: {shared, independent} Base classifier: LibSVM with 2-phase grid search Sam Reid Model Combination in Multiclass Classification 31/ 76

36 Discussion Shared vs Independent: Test Set Accuracy Average accuracy (%) p <= p <= p <= p <= shared independent one-vs-all all-pairs one-vs-all-hamming all-pairs-squared Sam Reid Model Combination in Multiclass Classification 32/ 76

37 Discussion Subproblems are Similar - Vehicle, one-vs-all vehicle: one-vs-all accuracy log2(g) subproblem 0 subproblem 1 subproblem 2 subproblem 3 Independent model selection curves for one-vs-all on vehicle Sam Reid Model Combination in Multiclass Classification 33/ 76

38 Discussion Subproblems are Similar - Vehicle, all-pairs accuracy vehicle: all-pairs log2(g) subproblem 0 subproblem 1 subproblem 2 subproblem 3 subproblem 4 subproblem 5 Independent model selection curves for all-pairs on vehicle Sam Reid Model Combination in Multiclass Classification 34/ 76

39 log2(g) subproblem 0 subproblem 1 subproblem log2(g) subproblem 0 subproblem 1 subproblem subproblem 0 subproblem 1 subproblem 2 subproblem 3 subproblem 4 Discussion log2(g) log2(g) log2(g) log2(g) Subproblems are Similar - Examples cars: one-vs-all page-blocks: one-vs-all letter: one-vs-all accuracy accuracy accuracy cars: one-vs-all page-blocks: one-vs-all letter: one-vs-all accuracy cars: all-pairs accuracy page-blocks: all-pairs accuracy letter: all-pairs cars: all-pairs page-blocks: all-pairs letter: all-pairs Sam Reid Model Combination in Multiclass Classification 35/ 76

40 Discussion Subproblems are Similar - Aggregate Results 1/3 Define γ s = optimal shared hyperparameter γ i = optimal independent hyperparameter Compute accuracy difference d = ā(γ i ) a(γ s ) Where ā indicates an average over subproblems Sam Reid Model Combination in Multiclass Classification 36/ 76

41 Discussion Subproblems are Similar - Aggregate Results 2/3 Average Accuracy Loss (%) halloffame vehicle synthetic-control Average Subproblem Loss at Selected Optimum For each dataset i, d i < 0.80% Average d = 0.30% authorship optdigits anneal waveform vowel letter dj collins segment page-blocks mfeat-morphological hypothyroid eucalyptus ecoli cars arrhythmia autos one-vs-all Sam Reid Model Combination in Multiclass Classification 37/ 76

42 Discussion Subproblems are Similar - Aggregate Results 3/3 Average Accuracy Loss (%) Average Subproblem Loss at Selected Optimum anneal waveform authorship halloffame hypothyroid optdigits eucalyptus segment ecoli page-blocks vowel vehicle collins cars synthetic-control arrhythmia autos mfeat-morphological dj letter all-pairs Largest values: 36.6% (letter), 29.4% (dj ) Average d = 4.24% Sam Reid Model Combination in Multiclass Classification 38/ 76

43 Discussion Differing Subproblems Favor Independent Construct a synthetic problem with different shapes of decision boundaries Requires different hyperparameters Requires independent optimization First, a control experiment with only linear decision boundaries Sam Reid Model Combination in Multiclass Classification 39/ 76

44 Discussion Differing Subproblems Favor Independent - Linear Synthetic Data 1/2 Linear Decision Boundaries with Varying Noise y x Class_0 Class_1 Class_2 Sam Reid Model Combination in Multiclass Classification 40/ 76

45 Discussion Differing Subproblems Favor Independent - Linear Synthetic Data 2/2 Accuracy (%) results for linear decision boundaries. Standard error over 10 random samplings is indicated in parentheses. reduction shared independent one-vs-all 66.7 (1.3) 66.1 (1.3) one-vs-all-hamming 58.2 (2.5) 58.1 (1.9) all-pairs 67.6 (1.3) 66.5 (1.9) Sam Reid Model Combination in Multiclass Classification 41/ 76

46 Discussion Differing Subproblems Favor Independent - Mixed Synthetic Data 1/2 Linear and Nonlinear Decision Boundaries y x A B C Sam Reid Model Combination in Multiclass Classification 42/ 76

47 Discussion Differing Subproblems Favor Independent - Mixed Synthetic Data 2/2 Accuracy (%) results for mixed linear and nonlinear decision boundaries. Standard error over 10 random samplings is indicated in parentheses. reduction shared independent one-vs-all 82.4 (0.6) 83.5 (0.9) one-vs-all-hamming 78.5 (1.3) 79.5 (1.3) all-pairs 82.4 (1.3) 84.2 (0.9) Sam Reid Model Combination in Multiclass Classification 43/ 76

48 one-vs-all-shared one-vs-all-sharedsub one-vs-all-shared-oracle all-pairs-shared all-pairs-sharedsub all-pairs-shared-oracle Discussion one-vs-all all-pairs one-vs-all-hamming all-pairs-squared Multiclass Accuracy Binary Accuracy + Noise arrhythmia: one-vs-all arrhythmia: one-vs-all accuracy multiclass accuracy (%) log2(g) average binary accuracy (%) one-vs-all one-vs-all multi vs binary arrhythmia: all-pairs arrhythmia: all-pairs accuracy multiclass accuracy (%) log2(g) average binary accuracy (%) all-pairs all-pairs multi vs binary Sam Reid Model Combination in Multiclass Classification 44/ 76

49 one-vs-all-shared one-vs-all-sharedsub one-vs-all-shared-oracle all-pairs-shared all-pairs-sharedsub all-pairs-shared-oracle Discussion one-vs-all one-vs-all-hamming all-pairs all-pairs-squared Multiclass Accuracy Binary Accuracy + Noise: Anneal accuracy anneal: one-vs-all log2(g) multiclass accuracy (%) anneal: one-vs-all average binary accuracy (%) one-vs-all one-vs-all multi vs binary anneal: all-pairs anneal: all-pairs accuracy multiclass accuracy (%) log2(g) average binary accuracy (%) all-pairs all-pairs multi vs binary Sam Reid Model Combination in Multiclass Classification 45/ 76

50 Discussion Multiclass Accuracy Binary Accuracy + Noise: Aggregate R-Squared Value for One-vs-All dj collins mfeat-morphological optdigits letter autos segment vowel authorship synthetic-control eucalyptus vehicle arrhythmia page-blocks ecoli cars hypothyroid waveform halloffame anneal Average R-Squared Value: One-vs-all=0.791, All-pairs=0.910 Sam Reid Model Combination in Multiclass Classification 46/ 76

51 Discussion Multiclass Metric Non-Essential Hypothesis: Advantage of shared due to selection on target multiclass metric To test, implement shared-sub Constraints models to be shared But selected based on average binary accuracy Results comparing shared vs shared-sub one-vs-all: p 0.65 all-pairs: p 0.10 ova-hamming: p 0.57 No statistically significant differences Conclusion: Sharing hyperparameters valuable whether you use avg binary or multiclass metric Sam Reid Model Combination in Multiclass Classification 47/ 76

52 Discussion Oracle Selection favors Shared To rule out sampling problems, use an oracle to select the optimal model Use oracle for both shared and independent one-vs-all all-pairs one-vs-all-hamming all-pairs-squared accuracy shared indep indep indep P-values from the Wilcoxon signed-ranks test are indicated by the winning strategy. For one-vs-all, shared still beats independent Independent wins for all-pairs and one-vs-all-hamming No difference for all-pairs-squared Sam Reid Model Combination in Multiclass Classification 48/ 76

53 Discussion Supplementary Result: Comparing Methods Average ranks of the 7 algorithms under study (omitted ova-ham-indep); algorithms not statistically significantly different from the top-scoring algorithm are connected to it with a vertical line. Sam Reid Model Combination in Multiclass Classification 49/ 76

54 Discussion Conclusions Shared hyperparameters often better than independent optimization Subproblems often similar, especially in one-vs-all If there are different decision boundary shapes, use independent Future Work Multiclass metrics with no binary analog in independent optimization? (e.g. multiclass cost matrix) Relationship to regret transform, Langford & Beygelzimer, 2005 Sam Reid Model Combination in Multiclass Classification 50/ 76

55 Our Method Outline Regularization in Linear Combinations of Multiclass Classifiers Model Discussion Our Method Sam Reid Model Combination in Multiclass Classification 51/ 76

56 Our Method Pairwise Classification Assuming a classification problem with k 3 classes k(k 1)/2 subproblems, one for each pair of classes Estimate ˆµ ij ( x) µ ij ( x) = P(y = c i y = c i or c j, x) Note that µ ij = p i p i +p j Combine: p = {p 1, p 2,..., p k } = f (ˆµ ij ( x)) Sam Reid Model Combination in Multiclass Classification 52/ 76

57 Our Method Pairwise Classification Subproblem Example Illustration of an A-C decision boundary in a 2D, 3-class example of pairwise classification. Sam Reid Model Combination in Multiclass Classification 53/ 76

58 Our Method Pairwise Classification Methods Voted pairwise classification (VPC): Friedman, 1996 ŷ( x) = argmaxi j:j i 1(ˆµ ij( x) > ˆµ ji ( x)) Equivalent to Bayes optimal prediction if ˆµij ( x) = µ ij ( x) Sam Reid Model Combination in Multiclass Classification 54/ 76

59 Our Method Pairwise Classification Methods Voted pairwise classification (VPC): Friedman, 1996 ŷ( x) = argmaxi j:j i 1(ˆµ ij( x) > ˆµ ji ( x)) Equivalent to Bayes optimal prediction if ˆµij ( x) = µ ij ( x) Hastie & Tibshirani (HT), 1996 Iteratively update p = {p1, p 2,..., p k } Min KL-Divergence between µ and ˆµ, l(p) = i j n ij ˆµ ij ˆµ ij µ ij Converges to minimum of KL divergence Sam Reid Model Combination in Multiclass Classification 54/ 76

60 Our Method Pairwise Classification Methods Voted pairwise classification (VPC): Friedman, 1996 ŷ( x) = argmaxi j:j i 1(ˆµ ij( x) > ˆµ ji ( x)) Equivalent to Bayes optimal prediction if ˆµij ( x) = µ ij ( x) Hastie & Tibshirani (HT), 1996 Iteratively update p = {p1, p 2,..., p k } Min KL-Divergence between µ and ˆµ, l(p) = i j n ij ˆµ ij ˆµ ij µ ij Converges to minimum of KL divergence Wu, Lin, Weng (WLW), 2004 µij = p i p i +p j µ ij µ ji = p i p j k Approx min p i=1 Guaranteed convergence j i (ˆµ jip i ˆµ ij p j ) 2 s.t. k i=1 p i = 1, p i 0 Sam Reid Model Combination in Multiclass Classification 54/ 76

61 Our Method Pairwise Classification Pros (Furnkranz, 2002) Smaller Subproblems Simpler Subproblems Improved Accuracy (disputed by Rifkin & Klautau, 2004) Cons Larger number of subproblems than one-vs-all Each pairwise classifier is trained on only two of the classes but makes predictions for instances from any class (Hastie & Tibshirani, 1996, Cutzu, 2003) e.g. a classifier trained on c A and c B may have unpredictable behavior for instances with y( x) = c C Sam Reid Model Combination in Multiclass Classification 55/ 76

62 Our Method Thesis Statement - Part III When solving a multiclass problem with a set of pairwise binary classifiers, incorporation of the probability of membership in each pair improves performance. Sam Reid Model Combination in Multiclass Classification 56/ 76

63 Our Method : Derivation 1/2 Theorem of Total Probability: Assumes p(b x) = N p(b a i, x)p(a i x) (1) i=1 a 1..a N mutually exclusive and exhaustive so N i=1 p(a i x) = 1 Let b = c i N = 2 a 1 = c i c j a 2 = L c i c j, for L = {c 1..c k } p(c i L, x) = p(c i c i c j, x)p(c i c j L, x) +p(c i L c i c j, x)p(l c i c j L, x) Sam Reid Model Combination in Multiclass Classification 57/ 76

64 Our Method : Derivation 2/2 But p(c i L c i c j, x) = 0 (2) Average over all j i p(c i x) = p(c i c i c j, x)p(c i c j L, x) (3) ˆp(c i L, x) = 1 k 1 ˆp(c i c i c j, x)ˆp(c i c j L, x) (4) j i Normalize so that i ˆp(c i L, x) = 1. Sam Reid Model Combination in Multiclass Classification 58/ 76

65 Our Method Comparison to Other Pairwise Classification Methods PPC Solves for each term pi ( x) independently Models pi + p j = p(i or j L, x) directly Conceptually simpler Easier to implement Theoretically well motivated Hastie-Tibshirani (HT) method approximates p i = j i ( 2 k(k 1) )µ ij (Wu et al., 2004) Equivalent to our method with the assumption pi + p j = 2/k Sam Reid Model Combination in Multiclass Classification 59/ 76

66 Our Method Computational Complexity Computational complexity of one-vs-all (OVA), pairwise coupling (PC) and probabilistic pairwise classification (PPC) OVA PC PPC subproblems k k(k-1)/2 k(k-1) instances per subproblem N 2N/k N (half) + 2N/k (other half) computational complexity/svm O(kN 3 ) O(k 1 N 3 ) O(k 2 N 3 ) Sam Reid Model Combination in Multiclass Classification 60/ 76

67 Our Method Base Classifiers Decision Tree (J48) K-Nearest Neighbor (KNN) Random Forests (RF-100) Support Vector Machines (SVM-121) Sam Reid Model Combination in Multiclass Classification 61/ 76

68 Our Method Base Classifiers Decision Tree (J48) K-Nearest Neighbor (KNN) Random Forests (RF-100) Support Vector Machines (SVM-121) Multiclass Classification Methods Multi (for J48, KNN, RF-100) Voted Pairwise Classification (VPC) Hastie-Tibshirani (HT) Wu, Lin, Weng (WLW) (PPC) Sam Reid Model Combination in Multiclass Classification 61/ 76

69 Our Method Base Classifiers Decision Tree (J48) K-Nearest Neighbor (KNN) Random Forests (RF-100) Support Vector Machines (SVM-121) Multiclass Classification Methods Multi (for J48, KNN, RF-100) Voted Pairwise Classification (VPC) Hastie-Tibshirani (HT) Wu, Lin, Weng (WLW) (PPC) Metrics Accuracy Brier 1 b( x) = 1 1 d j (t j( x) ˆp j ( x)) 2, t j ( x) = 1(y( x) = c j ) Sam Reid Model Combination in Multiclass Classification 61/ 76

70 Our Method Average Accuracy accuracy (%) multiclass vpc ht wlw ppc j48 knn rf100 svm121 Accuracy averaged over all 20 data sets. Sam Reid Model Combination in Multiclass Classification 62/ 76

71 Our Method Average Brier Score 95.5 Rectified Brier score (%) multiclass vpc ht wlw ppc 91.5 j48 knn rf100 svm121 Rectified Brier score averaged over all 20 data sets. Sam Reid Model Combination in Multiclass Classification 63/ 76

72 Our Method Average Ranks Sam Reid Model Combination in Multiclass Classification 64/ 76

73 Our Method Varying Base Classifier Accuracy Accuracy vs. Number of Trees Averaged over 20 Data Sets Accuracy (%) log_10(number of Trees) multi vpc ht wlw ppc Accuracy vs number of trees in random forest Sam Reid Model Combination in Multiclass Classification 65/ 76

74 Our Method Learning Curves Accuracy (%) Learning Curves ,000 Number of Data Points multi vpc ht wlw ppc Accuracy vs sample size for 10 largest data sets Sam Reid Model Combination in Multiclass Classification 66/ 76

75 Our Method Duplicate Decision Boundaries Favors MULTI Hypothesis: Direct multiclass method will outperform PPC when decision boundaries are shared Construct a synthetic data set meant to favor multi-j48 Decision boundaries are shared Sam Reid Model Combination in Multiclass Classification 67/ 76

76 y Our Method Duplicate Decision Boundaries: Noiseless Synthetic Data 1.6 Noiseless Synthetic Data Set x A B C D multi-j48 ppc-j (0.08) 98.7 (0.10) Sam Reid Model Combination in Multiclass Classification 68/ 76

77 y Our Method Duplicate Decision Boundaries: Noisy Synthetic Data 2.00 Noisy Synthetic Data Set x A B C D multi-j48 ppc-j (0.34) 86.0 (0.31) Sam Reid Model Combination in Multiclass Classification 69/ 76

78 Our Method PPC More Accurate at Large Number of Classes 1/2 Relative Accuracy (%) Accuracy relative to random forest vs. # classes Number of Classes Method accuracy relative to RF-100 vpc ht wlw ppc Sam Reid Model Combination in Multiclass Classification 70/ 76

79 Our Method PPC More Accurate at Large Number of Classes 2/2 Relative accuracy (%) Accuracy for Discretized Regression Data Sets Number of Classes housing autompg meta pbc quake sensory strike cholesterol cleveland average PPC relative to RF-100 for discretized regression data sets Sam Reid Model Combination in Multiclass Classification 71/ 76

80 Our Method Terms in PPC estimate equally important Hypothesis: Both terms in the PPC estimate are equally important ˆp(c i L, x) = 1 ˆp(c i c i c j, x)ˆp(c i c j L, x) k 1 j i Pairwise term: ˆp(c i c i c j, x) Weight (pair-vs-rest) term: ˆp(c i c j L, x) Use J48 decision trees, 100 replications, 20 data sets. Adjusted p-values under various degradations. hypothesis p Holm both vs. none 2.25E-10 no-pair vs. none 6.87E-05 no-weight vs. none 7.49E-04 both vs. no-weight both vs. no-pair no-weight vs. no-pair Sam Reid Model Combination in Multiclass Classification 72/ 76

81 Our Method PPC Summary & Conclusions Introduced new pairwise classification algorithm, PPC Based on Theorem of Total Probability Explicitly models p(c i c j L, x) Outperforms or ties related methods For several base classifiers, metrics, data sets Some data sets benefit from direct multiclass methods PPC works well at large # classes Future Work Faster but less accurate pair-vs-rest classifier? Independent vs. shared in PPC? Sam Reid Model Combination in Multiclass Classification 73/ 76

82 Our Method Thesis Statement Multiclass classification problems can be productively solved by combining multiple classifiers. Specifically: In linear combinations of multiclass classifiers, regularization significantly improves performance. When solving a multiclass problem with a set of binary classifiers, it is more effective to constrain subproblems to use the same hyperparameters than to optimize each independently. When solving a multiclass problem with a set of pairwise binary classifiers, incorporation of the probability of membership in each pair improves performance. Sam Reid Model Combination in Multiclass Classification 74/ 76

83 Our Method Acknowledgments PhET Interactive Simulations NSF Grants SBE Science of Learning Center (Garrison Cottrell, PI) BCS BCS SBE Mike Mozer, Greg Grudic Dissertation Support Group/CAPS Turing Institute UCI Repository Sam Reid Model Combination in Multiclass Classification 75/ 76

84 Our Method Questions? Questions? Sam Reid Model Combination in Multiclass Classification 76/ 76