Learning Multi-class Theories in ILP

Transcription

1 Learig Multi-class Theories i ILP Tarek Abudawood ad Peter A. Flach Itelliget Systems Laboratory, Uiversity of Bristol, UK Dawood@cs.bris.ac.uk ad Peter.Flach@bristol.ac.uk Abstract. I this paper we ivestigate the lack of reliability ad cosistecy of those biary rule learers i ILP that employ the oe-vs-rest biarisatio techique whe dealig with multi-class domais. We show that we ca lear a simple, cosistet ad reliable multi-class theory by combiig the rules of the multiple oe-vs-rest theories ito oe rule list or set. We experimetally show that our proposed methods produce coheret ad accurate rule models from the rules leared by a well kow ILP learer Aleph. 1 Itroductio Iductive Logic Programmig (ILP) is cocered with iducig first-order clausal models from examples ad backgroud kowledge. Symbolic rule learig systems i ILP, such as FOIL [1], PROGOL [2] ad Aleph [3], lear rules from positive ad egative examples. They are kow for their ability to lear from complex structured data ad build effective classificatio models i a rage of domais. Ufortuately, they struggle i dealig with multi-class problems. I most situatios they reduce a multi-class problem ito multiple biary problems followig the pairwise oe-vs-oe or oe-vs-rest biarisatio techiques. Aleph, for example, ca lear a multi-class theory i the oe-vs-rest paradigm where the outcome of its iductio ca be see as a combiatio of several black-box models. Each model iduces rules for oe specific (positive) class, ad a default rule is added to predict the remaiig classes. This oe-vs-rest approach is a commoly used machie learig techique due to its simplicity i solvig multi-class problems. It has bee prove to be powerful whe compared to other multi-class approaches [4]. However, we argue i this paper that the oe-vs-rest techique is ot suitable for first-order rule learers as there is a strog bias toward the egative classes leadig to urealistic estimates of predictive power. I additio, the lack of itegrity betwee the differet biary models leads to icosistet predictios. We ivestigate the reliability (how much oe ca rely o the quality of a model) ad cosistecy (how cosistet are the predictios of multiple related models) of oevs-rest biary models ad illustrate the differece with a proper multi-class model i Sect. 2. I Sect. 3 our goal is to ivestigate several methods to overcome the problems of the curret applicatio of oe-vs-rest techique i ILP rule learers. We experimetally demostrate the performace of our suggested methods i Sect. 4 ad compare them with the stadard biary method of Aleph. I Sect. 5 we briefly revisit related work before we draw the coclusio i the fial sectio.

2 2 2 Motivatio I machie learig accuracy is commoly used for comparig the classificatio performace ad thus may researchers report their results i terms of accuracy, ad compare their results agaist accuracies of other algorithms. The accuracy of a model ca be iterpreted as the expectatio of correctly classifyig a radomly selected example. c + c Tot. c Act. + T P FN E + c FP T N E Tot. Ê + Ê E (a) c 1 c 2 c 3... c Tot. c 1 T P 1 FN 1 FN 1... FN 1 E 1 c 2 FN 2 T P 2 FN 2... FN 2 E 2 Act. c 3 FN 3 FN 3 T P 3... FN 3 E c FN FN FN... T P E Tot. Ê i Ê 2 Ê 3... Ê E (b) Fig. 1. Cotigecy tables for a biary model (a) ad a multi-class model (b). Usig the otatio explaied i Fig. 1, we itroduce the followig defiitios. Defiitio 1 (Recall). The recall of a give class c i, deoted Recall i or Recall i +, is the proportio of examples of class c i that is correctly classified by a model (Recall i = T P i /E i ). The egative recall of class c i, deoted Recalli, is the proportio of examples of class c i icorrectly classified (Recalli = 1 T P i /E i ). I case of two classes, positive ad egative, we deote the recall of the positive class as Recall + = T P/E + ad of the egative class as Recall = T N/E. Defiitio 2 (Accuracy). Give two classes c + ad c, the biary accuracy of a model is defied as Accuracy bi = T P + T N E = E+ E Recall+ + E E Recall That is, biary accuracy is a weighted average of the positive ad egative recall, weighted by the class prior. This exteds to multiple classes: Accuracy = T P i E = E i T P i = E E i E i E Recall+ i For this reaso we sometimes refer to accuracy as (weighted) average positive recall. Defiitio 3 (Multi-Model Accuracy). Give classes ad oe-vs-rest models, oe for each class, the multi-model accuracy is defied as the average biary accuracy of the models: Accuracy mm = 1 ( E+ i E Recall+ i + E i E Recall i ) The followig simple result is worth otig.

3 3 Lemma 1. The accuracy of a sigle multi-class model is ot equivalet to the multimodel accuracy of the oe-vs-rest models derived from the multi-class model. Proof. Accuracy mm = 1 = 1 ( E+ i E Recall+ i E + i E Recall+ i = 1 Accuracy E i E Recall i ) (1) + 1 E i E Recall i (2) E i E Recall i (3) I goig from (2) to (3) we rely o the fact that the oe-vs-rest models are derived from a sigle multi-class model. If this is t the case (as i Aleph, for istace), the weighted average positive recall is ot the same as accuracy, which compouds the issue. It ca be see from Lemma 1 that the two accuracies are differet. Accuracy of a multiclass model relies o the positive recalls weighted by the class priors. O the other had, the average accuracy of multiple biary models relies o the recalls of both classes where the importace of the positive recalls is decreased times, hece, there is a icrease of the importace of classifyig a egative example times. The followig example demostrates why multi-model accuracy is misleadig. Example 1 (A radom classifier). Let us cosider a 3-class problem comprisig 108 examples uiformly distributed amog the classes. A radom 3-class classifier would result i the uiform cotigecy table show i Fig. 2(a). O the other had, if a radom biary classifier is applied to the three oe-vs-rest biary problems we obtai the three cotigecy tables i Figs. 2(b)-2(d). The accuracy of the multi-class model is 0.33, while the multi-model accuracy of the biary models is c 1 c 2 c 3 Tot. c Act. c c Tot (a) c 1 c 2,3 Tot. c Act c 2, Tot (b) c 2 c 1,3 Tot. c Act c 1, Tot (c) c 3 c 1,2 Tot. c Act c 1, Tot (d) Fig. 2. Four cotigecy tables o a three-class problem showig the predictios of a radom multi-class classifier (a) ad three radom oe-vs-rest classifiers (b)-(d). It is clear that the average accuracy of the biary models is 1.5 times more tha the accuracy of the multi-class model because the weight of the egative class is twice the weight of the positive class. Whe havig a proper multi-class model, there are oly credits for classifyig examples correctly. Averagig the positive ad egative recalls for multiple oe-vs-oe theories could be misleadig but it is eve more harmful whe it comes to oe-vs-rest theories as the problem is propagated.

4 4 Aother problem arisig whe iducig multiple idepedet biary theories is the lack of itegrity betwee the predictios of the differet biary theories. This may cause a example to have differet possible predictios i several cotigecy tables because each model produces predictios idepedetly of the others. The predictios of the models o each example should be cosistet. For istace, by cosiderig oe-vsrest models where each model is traied to predict oe class as positive, the the predictio for a example x o the i-th model should be be cosistet with its predictio o the j-th model, ĉ i (x) = +ve ad cˆ j (x) = ve j i, where cˆ j (x) ad cˆ j (x) express the predictio of the i-th ad the jth biary model respectively for example x. If the predictios are icosistet the such coflicts eed to be solved to esure the cosistecy i the predictios for each example i all models. All oe-vs-rest models of support vector machies ad aive Bayes [5] resolve these collisios by obtaiig scores from each oe of the models ad the model with the maximum score wis the predictio [4]. A rule learer such as CN2 [6] lears ordered rule lists i oe of its settigs to avoid such coflicts. I pairwise techiques votig methods [7 10] ca be cosidered to itegrate the predictios. The discussio about ureliability ad icosistecy holds geerally whe applyig oe-vs-rest techique i ay learig system but we would like to emphasise the importace of this issue particularly i ILP biary rule learig systems such as Aleph. This is because we oly iduce rules for the positive class i each oe-vs-rest model while a default rule that always predicts the egative class is added i case a example ca ot be classified by ay iduced rule. The default rules give credits for ot classifyig egative examples which makes it easy to obtai high egative recalls without iducig ay rules for the egatives. For istace, oe could obtai 0.67 multi-model accuracy with three empty theories 1 o the problem of Example 1. Hece, there is a eed to itegrate the differet biary models of such rule learig systems i order to esure the reliability ad cosistecy of their predictios. 3 Improved Learig of Multi-class Theories I this sectio we ivestigate how oe could improve the reliability of the all oe-vsrest theories i ILP by combiig their biary models ito a sigle rule listor rule set model. Our approach is differet from the other first-order rule learig approaches i various respects. First, it does ot treat the various models as idepedet black-box models, but istead combies the rules of all the models ito a sigle model. Secodly, there is oly oe default rule ad the class of the default rule is determied probabilistically accordig to the distributio of the ucovered traiig examples of all the classes. Fially, a sigle predictio is obtaied for each example i oe multi-class cotigecy table. Despite the simplicity of our approaches, their predictios are reliable, cosistet ad accurate, as we will show i our experimets. I ay rule list model, the rules are ordered i the fial theory accordig to a certai criterio. Whe a usee example is ecoutered, the rules are tried oe by oe i the order of the list ad the first rule that fires determies the class of the example. So the 1 A empty theory is a theory where a biary rule learer fails to iduce ay rule for the positive examples.

5 5 key idea is how to order these rules. Oe eeds to evaluate the rules iduced by the models ad assig them scores. We adopt Chi 2 as our multi-class evaluatio measure for the rules ad used it to build a Multi-class Rule List (MRL) model. Defiitio 4 (Chi-Squared [11]). The Chi-squared score of a rule r j of the i-th class is defied as Chi 2 (r j ) = [e i E ee i ] 2 ee i (E e) where e is the umber of examples covered by r j, e i the umber of examples correctly classified by r j, E i is the total umber of examples of the i-th class, ad E is the total umber of examples. MRL I this method, after learig rules for all classes, the rules are re-ordered o decreasig Chi 2. The ties are broke radomly. If a rule is added to the rule list, the all examples it covers are removed from the traiig set ad the rest of the rules are re-evaluated based o the remaiig examples util o further rule is left. At the ed, a sigle default rule is assiged predictig the majority class of the ucovered examples. I a rule set model, the rules are uordered ad the class of a ew example is determied based o the traiig statistics of all rules that fire for that particular example. For istace, the CN2 rule learer [6] lears a rule set model ad tags the rules with their coverage over all the classes. If a ew example is to be classified, CN2 sums up the coverage of all rules that fire over each class ad the class with the highest coverage wis. We propose two methods to hadle multi-class rule set theories, the Multi-class Rule Set Itersectio (MRSI) method ad the Multi-class Rule Set Uio (MRSU) method. The descriptios of the two methods are discussed below. MRSI I MRSI every rule from the multiple oe-vs-rest models is evaluated over the etire traiig set oce, ad the idetifiers of the examples they cover are stored. A default rule is formed based o the majority class of the ucovered traiig examples. If a ew example is to be classified, all the rules are tried. For those rules that fire, we determie the itersectio of their traiig set coverage usig the example idetifiers, ad their class distributio gives us the empirical probability of each class. The class with the maximum probability is predicted for the example. Agai the ties are broke radomly. I the case of a empty itersectio, the majority class is assiged to the example. MRSU The MRSU method differs from the MRSI method i that it determies the class of a ew example based o the uio of the traiig coverage of all rules that cover the ew example, istead of the itersectio. The MRSU method is closer i spirit to the CN2 method, which adds up the coverage of all rules that fire. However, by usig example idetifiers we avoid doublecoutig of examples that are covered by several rules, which meas that we obtai proper empirical probabilities rather tha CN2 s estimates. 4 Empirical Evaluatio I this sectio we evaluate ad compare our proposed sigle multi-class theory learig methods (MRL, MRSU ad MRSI) over 6 multi-class data sets ad 5 biary data sets

6 6 Table 1. Data sets used i the experimets. The group to the left are multi-class data sets while the group to the right are biary data sets. Starred data sets are propositioal; the rest is relatioal. Data set o. Name Class dist. Data set o. Class. dist 1 Car* 1210, 384, 69, 65 7 Mutageesis 125, 63 2 Diterpee 447, 355, 352, 155, 71 8 Amie (Alzheimer) 1026, Ecoli* 143, 77, 52 9 Cholie (Alzheimer) 1026, Eglish 50, 50, Scopolamie (Alzheimer) 1026, Protei 116, 115, 77, Toxic (Alzheimer) 1026, Scale* 288, 288, 49 (Table 1). We use Aleph as our base-learer, learig rules for each class i tur. We the tur the rules leared by Aleph ito coheret multi-class models usig the techiques proposed i this paper. We compare agaist the CN2 rule set method described above. For each data set, cross-validated accuracies (Table 2) ad AUCs (Table 3) were recorded. MRL method does ot produce class probabilities ad hece produces a sigle poit i a ROC plot: i this case, AUC boils dow to the (uweighted) average of true positive ad true egative rates. MRSU, MRSI ad CN2 produce class probabilities ad hece AUC evaluates their rakig performace i the usual way. A multi-class AUC is obtaied by averagig each oe-vs-rest AUC weighted by the class prior. Sice averagig performace across data sets has limited meaig as the values may ot be commesurate, we report the raks (1 is best, 4 is worst) of the accuracies ad AUCs o each data set. We use the Friedma sigificace test o these raks at p = 0.05 with Boferroi-Du post-hoc test o our three proposed methods. I the Friedma test we record wis ad losses i the form of raks ad igore the magitude of these wis ad losses. The use of the Friedma test to evaluate multiple classifiers o multiple data sets is cosidered to be more appropriate tha the covetioal tests, see [12] for further details. By lookig at the average performace rak, ad calculatig the post-hoc test ad Critical Differece (CD = 1.78/2) with CN2 as cotrol, o the multi-class data sets, MRSI is sigificatly better tha CN2 o both accuracy ad AUC, while MRSU performs sigificatly worse o AUC. If we take a look at the biary data sets (CD = 1.95/2) we ca see that both MRL ad MRSI are sigificatly superior over CN2 w.r.t. AUC while o statistical sigificace is reported regardig their accuracies. The coclusio seems warrated that MRSI is preferable for multi-class data sets, while MRL is preferable for biary data sets. 5 Related Work As discussed earlier, may ILP rule learig systems icludig Aleph, PROGOL ad FOIL ca oly iduce biary theories ad multi-class theories are obtaied by covertig a multi-class problem ito several biary problems. The rules of the fial model are, i practice, a combiatio of idepedet multiple biary theories. Iductive Logic Costrait (ICL) [13] upgraded the propositioal CN2 to hadle multi-class first-order theories. Our CN2 implemetatio is similar to ICL (learig from ad hadlig multiclass structural domais) but was built over Aleph. The experimets demostrate that we ca improve over the CN2 probability estimatio method.

7 Table 2. Accuracies of our ew multi-class methods (MRL, MRSU ad MRSI) compared agaist CN2 accuracy, with average raks i brackets. The 6th colum shows the multi-model accuracy as reported by Aleph, which is particularly optimistic for multi-class problems due to overemphasisig the default rules. The right-most colum shows the average positive recall, which igores the default rules but is still ot equal to multi-class accuracy as coflictig predictios are ot take ito accout. MRL MRSU MRSI CN2 Aleph Stadard Multi-class accuracy Multi-model accuracy Average recall (2.00) (4.00) (1.00) (3.00) (2.00) (3.50) (1.00) (3.50) (1.00) (4.00) (2.00) (3.00) (3.00) (4.00) (1.00) (2.00) (3.00) (2.00) (4.00) (1.00) (2.00) (4.00) (1.00) (3.00) Average (2.17) (3.58) (1.67) (2.58) (2.00) (2.00) (4.00) (2.00) (4.00) (3.00) (1.00) (2.00) (1.00) (3.00) (2.00) (4.00) (2.00) (2.00) (4.00) (2.00) (3.00) (2.00) (4.00) (1.00) Average (2.40) (2.40) (3.00) (2.20) Table 3. Average oe-vs-rest AUCs of our multi-class methods (MRL, MRSU ad MRSI) compared agaist CN2, with average raks i brackets. The AUCs reported for Aleph are for referece oly, as these arise from over-emphasisig the default rules. MRL MRSU MRSI CN2 Aleph Stadard (1.00) (3.00) (4.00) (2.00) (4.00) (1.00) (3.00) (2.00) (3.00) (2.00) (1.00) (4.00) (4.00) (2.00) (1.00) (3.00) (2.00) (4.00) (1.00) (3.00) (1.00) (3.00) (2.00) (4.00) Average (2.50) (2.50) (2.00) (3.00) (1.00) (4.00) (2.00) (3.00) (1.00) (3.00) (2.00) (4.00) (1.00) (3.00) (2.00) (4.00) (1.00) (3.50) (2.00) (3.50) (1.00) (3.00) (2.00) (4.00) Average (1.00) (3.30) (2.00) (3.70) While most of the ILP systems implemet the coverig approach (separate-adcoquer), TILDE [14] implemets a divide-ad-coquer approach ad iduces a sigle first-order logic multi-class theory that take a form of decisio tree. Tree models hadle multiple classes aturally. We pla a experimetal compariso with TILDE i future work. Several papers suggested differet approaches of dealig with multiple biary models [4, 5, 7 10, 15]. A compariso of may such approaches were made i [4] suggestig a superiority of the oe-vs-rest approach i geeral but they also poited out that the choice of the biarisatio techique makes little differece oce we lear good biary models. 6 Cocludig Remarks I this paper we ivestigated the lack of reliability ad cosistecy of the oe-vs-rest techique o multi-class domais. We showed that we could lear a simple ad sigle

8 8 multi-class rule list (MRLmethod) or rule set (MRSU ad MRSI methods) model by combiig the rules of all oe-vs-rest models ad tur them ito a coheret multi-class classifier. Our proposed methods geerate cosistet ad reliable multi-class predictios ad we experimetally showed that they produce sigificat results, w.r.t. accuracy ad AUC, o both multi-class ad biary domais whe compared agaist the CN2 method. Whe classificatio is made based o rule itersectio, MRSI, the best accuracies ad AUCs were achieved takig the multi-class data sets ito accout. Multi-class rule list, MRL, method seem to be suitable for two-class problems. The origi of this differece is subject of ogoig ivestigatios. Refereces 1. Quila, J.R., Camero-Joes, R.M.: FOIL: A Midterm Report. I: Proc. Europea Cof. o Machie Learig, Spriger-Verlag (1993) Muggleto, S.: Iverse Etailmet ad Progol. I: Proc. 6th Iteratioal Workshop o Iductive Logic Programmig. Volume 13., Spriger (1995) Sriivasa, A.: The Aleph Maual. Techical report, Uiversity of Oxford (2001) 4. Rifki, R., Klautau, A.: I Defese of Oe-Vs-All Classificatio. Machie Learig Research 5 (2004) Zadrozy, B., Elka, C.: Trasformig Classifier Scores Ito Accurate Multiclass Probability Estimates. I: Proc. 8th ACM SIGKDD It. Cof. o Kowledge Discovery ad Data Miig, New York, USA, ACM (2002) Clark, P., Niblett, T.: The CN2 Iductio Algorithm. Machie Learig 3 (1989) Friedma, J.H.: Aother approach to polychotomous classificatio. Techical report, Staford Uiversity, Departmet of Statistics (1996) 8. Platt, J.C., Cristiaii, N.: Large Margi DAGs for Multiclass Classificatio. I: Advaces i Neural Iformatio Processig Systems. Volume 12., MIT Press (2000) 9. Kijsirikul, B., Ussivakul, N., Mekavi, S.: Adaptive Directed Acyclic Graphs for Multiclass Classificatio. I: Proc. 7th Pacific Rim It. Cof. o Artificial Itelligece: Treds i Artificial Itelligece. Volume 2417., Spriger (2002) Dietterich, T.G., Bakiri, G.: Solvig Multiclass Learig Problems via Error-Correctig Output Codes. Artificial Itelligece Research 2 (1995) Abudawood, T., Flach, P.: Evaluatio Measures for Multi-class Subgroup Discovery. I: Europea Cof. o Machie Learig ad Priciples ad Practice of Kowledge Discovery i Databases. Volume 5781., Spriger (2009) Demšar, J.: Statistical Comparisos of Classifiers Over Multiple Data Sets. Machie Learig Research 7 (2006) De Raedt, L., Va Laer, W.: Iductive Costrait Logic. I: Algorithmic Learig Theory, Spriger (1995) Blockeel, H., De Raedt, L.: Top-dow Iductio of Logical Decisio Trees. I: Artificial Itelligece. Volume 101. (1997) Hsu, C.W., Li, C.J.: A Compariso of Methods for Multiclass Support Vector Machies. Neural Networks 13 (March 2002) Fawcett, T.: A Itroductio to ROC Aalysis. Patter Recogitio Letters 27(8) (Jue 2006)