Professor Dr. Gholamreza Nakhaeizadeh. Professor Dr. Gholamreza Nakhaeizadeh

Transcription

1 Statistic Methods in in Data Mining Business Understanding Data Understanding Data Preparation Deployment Modelling Evaluation Data Mining Process (Part 2) 2) Professor Dr. Gholamreza Nakhaeizadeh Professor Dr. Gholamreza Nakhaeizadeh

2 Short review of the last lecture Data Understanding Collect Collect initial initial --Can the the be be accessed accessed effectively effectively and and efficiently efficiently --Is Is there there any any restriction restriction in in collecting collecting the the --what are are the the needed needed where where are are the the --Examples of of sources sources --Data warehouse warehouse Describe Describe --Some of of characterization characterization measures measures --Data Structure Structure Observation, Observation, attribute attribute type type (nominal, (nominal, ordinal, ordinal, interval, interval, ratio, ratio, qualitative, qualitative, quantitative, quantitative, discrete) discrete) Data Data Type: Type: Cross-section Cross-section,, time time series series,, panel panel,, spatial spatial Explore Explore --Data exploration exploration Tools Tools Using Using descriptive descriptive summarization summarization (mean, (mean, median, median, mode, mode, variance, ) variance, ) --Using Visualization Visualization --OLAP Verify Verify quality quality --Are accurate accurate Are Are complete complete Are Are consistent consistent Data Preprocessing: Select, Clean, Transfer, Integrate Select : Observation reduction, attribute reduction Observation reduction: Sampling

3 Observation Reduction --Sampling Sampling --Intelligent Intelligent Sampling Sampling --Learn Learn to to forget forget.. Observations Attributes Observations Attributes

4 Observation Reduction : Sampling Observation Reduction : Sampling Statisticians: Sampling because obtaining the the entire entire set (population) is is too too expensive or or time time consuming (often (often they they do do not not have have the the and and start start collecting) collecting) Data Data Miners: Sampling because processing of of the the population is is too too expensive or or time time consuming (often (often they they have have the the ) ) good sample ~ representative sample has nearly the same property as the population : sample sample mean mean is is very very close close to to population population mean mean sample sample variance varianceis is very very close close to to population population variance variance 4

5 Observation Reduction : Sampling Observation Reduction : Sampling Task: Choose a sampling method that that with with high high probability leads leads to to a representative sample Choosing the the right right sampling technique Choosing the the right right sample size size 5

6 Observation Reduction : Sampling technique Observation Reduction : Sampling technique Random sampling: Equal Equal and and known known probability of of being being selected for for each each member of of the the population General aspects: General aspects: Sampling without replacement (s.wo.r) Sampling with with replacement (s.w.r.) (s.w.r.) During During the the sampling sampling process process the the probability probability of of selecting selecting any any objects objects remains remains constant constant Analyzing is easier Analyzing is easier 6

7 Observation Reduction : Sampling technique Observation Reduction : Sampling technique Systematic Sampling (called (called also also kth kthname selection method) Selection of of k; k; k= k= population size size // sample size size (( k sampling interval) Selection of of a start start point point Selection of of every every kth kthmember as as sample Example: Population size size = sample size size = k=10 k=10 start start point point = member number then then sample consists of of members number 15, 15, 25, 25, 35, 35, 45, 45, start point k= th 25th 35th 45th 7

8 Observation Reduction : Sampling technique Observation Reduction : Sampling technique Stratified Sampling Stratified Sampling Population consists of of different mutually exclusive subgroups (strata) varying considerably in in size. size. Examples: (120 (120 men, men, women), (1900 (1900 employment, unemployment), (300 (300 white, white, black) black) Random sampling can can fail fail to to adequately represent the the members with with low low frequency Solution: Stratified Sampling: Random sampling in in each each Subgroup (stratum) independently 8

9 Observation Reduction : Sampling technique Observation Reduction : Sampling technique Stratified Sampling Strategies Stratified Sampling Strategies Stratified Stratified sampling sampling strategies strategies Number Number of of members members drawn drawn from from each each subgroupa subgroupa is is proportional proportional to to the the size size of of that that subgroup subgroup Equal Equal numbers numbers of of members members are are drawn drawn from from each each subgroup subgroup even even though though the the gropus gropus are are of of different different sizes sizes Example: Example: Size Size of of population population 2000: 2000: employment, employment, unemployment unemployment Size Size of of needed needed sample: sample: Strategy Strategy 1 :: 50/ /2000 = 1/40 1/ ** 1/40 1/40 = 47,5 47, ** 1/40 1/40 = 2,5 2,5 Sample Sample consists consists of of employment employment and and 3 unemployment unemployment Strategy Strategy 2 :: Sample Sample consists consists of of employment employment and and unemployment unemployment 9

10 Observation Reduction --Sampling Sampling --Intelligent Intelligent Sampling Sampling --Learn Learn to to forget forget.. Observations Attributes Observations Attributes

11 Supervised and unsupervised learning Attributes Target variable Observations (Tuples) 11

12 Supervised Learning Nr. A1 A2 A3 An T 1 a11 a12 a13 a1n t1 2 a21 a22 a23 a2n t m a31 a32 a33 a3n am1 am2 am3 amn t3 tm Examples for Supervised Learning : Classification, Prediction Examples for Supervised Learning : Classification, Prediction 12

13 Unsupervised Learning Nr. A1 A2 A3 An T 1 a11 a12 a13 a1n t1 2 a21 a22 a23 a2n t m a31 a32 a33 a3n am1 am2 am3 amn t3 tm Example Example for for Unsupervised Unsupervised Learning: Learning: Clustering Clustering 13

14 General Aspects Data Data mining mining problems that that deal deal with with classification and and prediction may may involve hundreds or or even even thousands of of attributes that that can can potentially be be used used as as predictors Example: Document classification in in Text Text Mining: Bag-of-words: > attributes,, fault fault analysis in in the the automotive industry, Problem: A lot lot of of time time and and effort effort may may be be needed to to decide decide which which attribute should should be be included in in the the model model Solution: In In the the last last years years Statisticians and and Data Data Miners Miners have have developed many many attribute reduction algorithms 14

15 Why we we need attribute Reduction to to reduce reduce the the effect effect of of the the curse curse of of dimensionality to to speed speed up up learning process to to reduce reduce the the amount of of memory required to to improve model model interpretability to to do do visualization easier easier to to make make scalable the the sets with with many many nominal attributes 15

16 curse of dimensionality curse of dimensionality As As the the dimensionality of of increases often analysis become harder classification classification clustering clustering reduced classification accuracy reduced classification accuracy Poor quality cluster Poor quality cluster 16

17 creating new new attributes (combination (combination of of old old attribute) attribute) attribute attribute extraction extraction Selection a subset subset of of old old attributes FSS: FSS: feature subset subset selection attribute selection no no information lost lost if if redundant and and irrelevant attributes are are present Loss of of information 17

18 First elementary steps First elementary steps Using common sense or or domain Knowledge (if (if available) to to select a subset of of attributes Attribute Screening 18

19 First elementary steps First elementary steps Attribute Screening removes problematic attributes e.g: e.g: - attributes with with many missing values -- attributes with with values that that have too too much or or too too little little variation Example Income Income of of individuals = {{ 20, 20, 20, 20, 20, 20, 20, 20,..20, }} Attribute income is not informative Attribute income is not informative 19

20 Determining attribute importance by by criteria like: like: Information Gain Gini-Index Pearson Chi-Square Correlation coefficient Attribute Ranking Attribute Ranking Akaike information criterion (AIC).. 20

21 Remarks Attribute Ranking Attribute Ranking The The ranking criteria mentioned before before can can be be used used to to measure the the correlation between each each attribute and and the the target target variable (applicable only only to to Supervised Learning between two two attributes, pairwise Attributes Target variable In In case case 1, 1, an an attribute useless by by itself itself can can be be useful useful together with with others others In In case case 2 attribute selection is is independent of of the the target target variable or or,, generally, independent of of the the mining mining task task Observations Known Known as as Filter Filter Approach Approach 21

22 Embedded Methods in in embedded approaches attribute selection is is a part part of of the the training process not not all all Data Data Mining Mining algorithms have have this this built-in mechanism to to perform attribute selection within within the the training process due due to to avoiding retraining for for different attribute subsets,, embedded approaches are are more more efficient Examples: Decision and and Regression Trees Trees Remark Remark in in some some studies, studies, in in a first first step step simple simple linear linear embedded embedded systems systemsare are use use for for attribute attribute selection selection later later in in a second second step, step, the the selected selected attributed attributed are are used used for for training training of of a more more complicated complicated non-linear non-linear system system 22

23 Wrapper Methods Wrapper Methods Main Idea : Main Idea : Using a given classification or or prediction algorithm, evaluate the the prediction performance of of different subsets of of attributes Select the the subset with with highest performance 23

24 Wrapper Methods Wrapper Methods Main Challenges : Main Challenges : Selecting a search search method to to find find all all possible attribute subsets Selecting an an evaluation approach and and an an evaluation function to to compare the the prediction performance of of different attribute subsets About About 1: 1: Total Total search search in in the the case case of of too too large large number of of attributes needs needs massive amounts of of computation. Greedy search search like like forward selection and and backward elimination are are more more appropriate About About 2 :: Validation sets or or cross cross validation as as well well as as evaluation functions (e.g (e.gaccuracy rate rate or or mean mean squared error) error) can can be be used used 24

25 Principal Component Analysis (PCA) Principal Component Analysis (PCA) Main Idea Main Idea B2 A2 B1 Reducing multidimensional sets sets to to lower lower dimensions by by combination of of old old attributes the the variance of of the the observations in in original space space should should be be satisfactory covered by by the the new new created dimensions b1 a1 b2 a2 A1 b1 = p1 a1 + p2 a2 b2= q1 a1 + q2 a2 Interpretation Interpretation Instruments: Covariance Covariance Matrix Matrix Eigenvalues Eigenvalues Eigenvectors Eigenvectors 25