AIC Laboratory R. Leaf November 28, 2016

Transcription

1 AIC Laboratory R. Leaf November 28, 2016 In this lab we will evaluate the role of AIC to help us understand how this index can assist in model selection and model averaging. We will use the mtcars data included in R base, datasets::mtcars. 1. Inspect the data using R functionality, describe its structure, ranges, relationships... We are interested in building models that have independent variables that can predict the mpg of a car. 2a. Using your knowledge and the data - what do you think is the best model? What makes it so - your answer should involve quantitative work and explananation of model output. Preliminary Model Evalation lm.cand.01 <- lm(mpg ~ cyl, data = mtcars) anova(lm.cand.01) Analysis of Variance Table Response: mpg Df Sum Sq Mean Sq F value Pr(>F) cyl e-10 *** Residuals Signif. codes: 0 '***' '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 lm.cand.02 <- lm(mpg ~ cyl + drat, data = mtcars) summary(lm.cand.02)$r.squared [1] # Some tricks in lm lm(mpg ~., data = mtcars) Call: lm(formula = mpg ~., data = mtcars) Coefficients: (Intercept) cyl disp hp drat wt qsec vs am gear carb

2 lm(mpg ~ gear - 1, data = mtcars) Call: lm(formula = mpg ~ gear - 1, data = mtcars) Coefficients: gear Your answer to #2a may have (should have) involved evaluating some candidate models output and input. 2b. In the above box I give some more lm functionality using. and -1. What does this code do? Likely you will need to examine the model output in detail to make an informed answer. 3. How many possible models can you derive from these data to predict the mpg? The short answer is many. Derive some of these, and plot the effect of adding predictors on the r.squared value. r.sq.vect <- c() num.pred.vect <- c() r.sq.vect[1] <- summary(lm.cand.01)$r.squared r.sq.vect[2] <- summary(lm.cand.02)$r.squared #...r.sq.vect[3] <- summary(lm.cand.03)$r.squared... now run some more num.pred.vect[1] <- length(lm.cand.01$coef) num.pred.vect[2] <- length(lm.cand.02$coef) #...num.pred.vect[3] <- length(lm.cand.03$coef)... now run some more # plot(x = num.pred.vect, y = r.sq.vect) AIC in R Lets use some built-in R functionality to evaluate AIC. Use the code below as a guide for you to evaluate the AIC value of each of the candidtate model you derived above. AIC.vect <- c() AIC.vect[1] <- AIC(lm.cand.01) AIC.vect[2] <- AIC(lm.cand.02) # Calculate the AIC value of the candidate models you derived in #3 # and plot the AIC values as a function of the number of coefficient values: # plot(x = num.pred.vect, y = AIC.vect) 4. Make a table:column 1. the model formulation, mpg ~ wt + qsec + am (for example), Column 2. AIC value, Column 3. Delta AIC, Column 4. Relative model weight, Column 5. Provide the number of predictors used in the model 5. Identify four candidate models that have at least on predictor in common and derive the weighted (using the weights from #4) predicted value of Y-hat with respect to the common variable. 2

3 # Determine length of predicted output num.pred. <- length(predict(lm.cand.01)) # Initialize a matrix to hold the values - you will fill this matrix with predictors predict.mat <- matrix(na, nrow = num.pred., ncol = 4) # Populate the model with predicted values predict.mat[,1] <- predict(lm.cand.01) predict.mat[,2] <- predict(lm.cand.02) #... predict.mat[,3] #... predict.mat[,4] # Use the model weights you derived in #4. Put them in a vector # mod.wt <- c(wt.cand.mod.01, wt.cand.mod.02, wt.cand.mod.03, wt.cand.mod.04) # predict.mat[,1] <- wt.cand.mod.01[1]*predict.mat[,1] # pred.vect <- rowsums(predict.mat) # plot(x = KBB$"Common Model parameter""), pred.vect) 6. We will use the step function in base R, package stats. The R function step() can be used to perform variable selection. To perform selection we need to begin by specifying a starting model and the range of models which we want to examine in the search. How does this approach compare to the one above - PS this selection approach is a bit controversial...?stats::step starting httpd help server... done # Use the function to evaluate this model: step(lm(mpg ~., data = mtcars), direction = "both", trace = T) Start: AIC=70.9 mpg ~ cyl + disp + hp + drat + wt + qsec + vs + am + gear + carb Df Sum of Sq RSS AIC - cyl vs carb gear drat disp hp qsec <none> am wt Step: AIC=

4 mpg ~ disp + hp + drat + wt + qsec + vs + am + gear + carb Df Sum of Sq RSS AIC - vs carb gear drat disp hp <none> qsec am cyl wt Step: AIC=66.97 mpg ~ disp + hp + drat + wt + qsec + am + gear + carb Df Sum of Sq RSS AIC - carb gear drat disp hp <none> am qsec vs cyl wt Step: AIC=65.12 mpg ~ disp + hp + drat + wt + qsec + am + gear Df Sum of Sq RSS AIC - gear drat <none> disp am hp carb vs cyl qsec wt Step: AIC=63.46 mpg ~ disp + hp + drat + wt + qsec + am Df Sum of Sq RSS AIC - drat disp <none>

5 - hp gear cyl vs carb am qsec wt Step: AIC=62.16 mpg ~ disp + hp + wt + qsec + am Df Sum of Sq RSS AIC - disp <none> hp drat gear cyl vs carb qsec am wt Step: AIC=61.52 mpg ~ hp + wt + qsec + am Df Sum of Sq RSS AIC - hp <none> disp carb drat qsec cyl vs gear am wt Step: AIC=61.31 mpg ~ wt + qsec + am Df Sum of Sq RSS AIC <none> hp carb disp cyl drat gear vs am

6 - qsec wt Call: lm(formula = mpg ~ wt + qsec + am, data = mtcars) Coefficients: (Intercept) wt qsec am