Stat 301 Lecture 26. Model Selection. Indicator Variables. Explanatory Variables

Transcription

1 Model Selection Response: Highway MPG Explanatory: 13 explanatory variables Indicator variables for types of car Sports Car, SUV, Wagon, Minivan There is an indicator for Pickup but there are no pickups in the data. 1 Indicator Variables The indicator variable takes on the value 1 if it is that kind of vehicle and 0 otherwise. If all four indicator variables are 0, then the vehicle is a Sedan. 2 Explanatory Variables Indicator variables for All Wheel and Rear Wheel drive. If both indicator variables are 0, then the vehicle has Front Wheel drive. 3

2 Explanatory Variables Engine size (liters) Cylinders (number) Horsepower Weight (pounds) Wheel Base (inches) Length (inches) Width (inches) 4 Forward Selection Fit Model Personality: Stepwise Y, Response Highway MPG Put all 13 variables into the Construct Model Effects box. Click on Run Model 5 Stepwise Fit Stopping Rule: P-value Threshold Prob to Enter = Prob to Leave = Direction: Forward Click on Go 6

3 Stepwise Fit for Highway MPG Stepwise Regression Control Stopping Rule: P-value Threshold Prob to Enter Prob to Leave Direction: Forward SSE DFE RMSE RSquare RSquare Adj Cp p AICc BIC Current Estimates Lock Entered Parameter Estimate ndf SS "F Ratio" "Prob>F" Intercept Sports Car SUV Wagon Minivan All Wheel Rear Wheel Engine Cylinders Horsepower Weight e-11 Wheel Base Length Width Step History Step Parameter Action "Sig Prob" Seq SS RSquare Cp p AICc BIC 1 Weight Entered Horsepower Entered Wheel Base Entered Forward Selection Three variables are added Weight Horsepower Wheel Base All variables added are still statistically significant. 8 Forward Selection Model with Weight, Horsepower and Wheel Base. R 2 = , adj R 2 = RMSE = AICc = , BIC = C p =

4 Stepwise Fit Stopping Rule: P-value Threshold Prob to Enter = Prob to Leave = Direction: Backward Enter All Click on Go Backward Selection Eight variables are removed Length, Rear Wheel, Wagon, Width, Engine, Wheel Base, Weight, Sports Car. All variables left are statistically significant. 12

5 Backward Selection Model with SUV, Minivan, All Wheel, Cylinders and Horsepower. R 2 = , adj R 2 = RMSE = AICc = , BIC = C p = Backward Selection The final model from Backward selection is better than the final model from Forward selection. It has a higher R 2 value, higher adj R 2 value, lower RMSE, AICc, BIC and C p value. 14 Mixed Selection (Forward) Stopping Rule: P-value Threshold Prob to Enter = Prob to Leave = Direction: Mixed Click on Go 15

6 16 Mixed Selection (Forward) Three variables are added Weight Horsepower Wheel Base No variables are removed. This is the same as with Forward Selection. 17 Mixed Selection (Backward) Stopping Rule: P-value Threshold Prob to Enter = Prob to Leave = Direction: Mixed Enter All Click on Go 18

7 19 Mixed Selection (Backward) Eight variables are removed Length, Rear Wheel, Wagon, Width, Engine, Wheel Base, Weight, Sports Car. No variables are added. This is the same as with Backward Selection = 8191 models possible. 1-variable models listed in order of the R 2 value. 2-variable models listed in order of the R 2 value. etc. 13-variable (full) model. 21

8 Can specify the maximum number of variables in a model. Can specify the maximum number of models displayed for each number of variables. 22 Model with all 13 variables has the highest R 2 value, R 2 = Adj R 2 = RMSE = C p = 14 AIC c = BIC = Full Model The model with 13 variables has several variables that do not add significantly to the other

9 Is there a better model? Is there a model with: Lower RMSE? Lower C p? Lower AIC c? Lower BIC? 25 Model with 7 variables has the lowest RMSE value. Sports Car, SUV, Minivan, All Weight RMSE = Model with lowest RMSE Several variables are not statistically significant but very close to the threshold of Sports Car: F=3.847, P-value= Horsepower: F=3.761, P-value= Weight: F=3.653, P-value=

10 Model with 7 variables has the lowest C p value. Sports Car, SUV, Minivan, All Weight C p = This is the same model as the one with the lowest RMSE. 28 Model with 7 variables has the lowest AICc value. Sports Car, SUV, Minivan, All Weight AICc = This is the same model as the one with the lowest RMSE and C p. 29 Model with 4 variables has the lowest BIC value. Sports Car, All Wheel, Cylinders, and Weight BIC =

11 Strategy Pick a criterion; RMSE, C p, AIC c or BIC. Identify several good models, i.e. low values for the criterion. Look at R 2, significance of individual variables, behavior of the residuals. 31 RMSE Model Number RSquare RMSE AIC c BIC C p Sports Car, SUV, Minivan, All Weight SUV, Minivan, All Wheel, Rear Wheel, Engine, Horsepower, Weight, Wheel Base SUV, Minivan, All Wheel, Engine, Horsepower, Weight, Wheel Base Sports Car, SUV, Minivan, All Weight, Wheel Base SUV, Minivan, All Wheel, Engine, Cylinders, Horsepower, Weight, Wheel Base AIC c Model Number RSquare RMSE AIC c BIC C p Sports Car, SUV, Minivan, All Weight SUV, Minivan, All Wheel, Engine, Horsepower, Weight, Wheel Base SUV, Minivan, All Wheel, Cylinders, Horsepower, Weight, Wheel Base SUV, Minivan, All Wheel, Cylinders, Horsepower SUV, Minivan, All Wheel, Rear Wheel, Engine, Horsepower, Weight, Wheel Base

12 C p Model Number RSquare RMSE AIC c BIC C p Sports Car, SUV, Minivan, All Weight SUV, Minivan, All Wheel, Engine, Horsepower, Weight, Wheel Base SUV, Minivan, All Wheel, Cylinders, Horsepower, Weight, Wheel Base SUV, Minivan, All Wheel, Rear Wheel, Engine, Horsepower, Weight, Wheel Base Sports Car, SUV, Minivan, All Weight, Wheel Base Final Model The 7-variable model with SUV, Minivan, All Wheel, Engine, Horsepower, Weight and Wheel Base Appears to be a pretty good model. 35 Prediction Equation Predicted Highway MPG = *SUV 3.28*Minivan 2.08*All Wheel 1.65*Engine *Horsepower *Weight *Wheel Base 36

13 Summary All variables add significantly. R 2 = 0.705, adj R 2 = RMSE = AICc = , BIC = C p = Residual Best Model Predicted Highw ay MPG Normal Quantile Plot Count