Performance of the Mean- and Variance-Adjusted ML χ 2 Test Statistic with and without Satterthwaite df Correction

Transcription

1 FORDHAM UNIVERSITY THE JESUIT UNIVERSITY OF NEW YORK Performance of the Mean- and Variance-Adjusted ML χ 2 Test Statistic with and without Satterthwaite df Correction Jonathan M. Lehrfeld Heining Cham Fordham University

2 Outline Introduction Hypothesis Testing in SEM Maximum likelihood (ML) χ 2 test statistic Nonnormal data Alternative test statistics Simulation Design Results Discussion and future directions M3, May 2015 Comparing the Performance of the Mean- and Variance-Adjusted ML χ 2 Test Statistic with and without df Correction 2

3 Introduction Hypothesis Testing in SEM Covariance structure analysis models Σ as a function of θ Test H 0 : Σ = Σ θ against a non-restrictive alternative Discrepancy function F Σ, S quantifies difference between S and Σ = Σ θ Test statistic T = n 1 min θ the model to the data F Σ, S is used to judge the fit of M3, May 2015 Comparing the Performance of the Mean- and Variance-Adjusted ML χ 2 Test Statistic with and without df Correction 3

4 Introduction Normal-Theory ML Test Statistic Satorra & Bentler (1994) Normal-theory ML test statistic and F ML θ = log Σ θ + tr SΣ θ 1 log S p T ML = n 1 F ML θ Implemented in Mplus via ESTIMATOR = ML Variables must be normally distributed in population M3, May 2015 Comparing the Performance of the Mean- and Variance-Adjusted ML χ 2 Test Statistic with and without df Correction 4

5 Introduction Nonnormal Data Nonnormal data proliferate among psychometric studies (Micceri, 1989) Some applied areas are not expected to generate normal variables (Curran, West, & Finch, 1996) T ML has asymptotic χ 2 distribution only when variables are normal in the population (Satorra & Bentler, 1986) M3, May 2015 Comparing the Performance of the Mean- and Variance-Adjusted ML χ 2 Test Statistic with and without df Correction 5

6 Introduction SB Scaled Test Statistic Satorra & Bentler (1986, 1988, 1994) Scaled version of ML statistic where T SBM = T ML d tr UΓ U = V VΔ Δ VΔ 1 Δ V Δ is the Jacobian matrix of σ θ, V is a weight matrix a Γ is acov n 1 s σ, d is the model df T SBM ~ approx. χ d 2 (same means) Implemented in Mplus via ESTIMATOR = MLM M3, May 2015 Comparing the Performance of the Mean- and Variance-Adjusted ML χ 2 Test Statistic with and without df Correction 6

7 Introduction Robust Scaled Test Statistic Extension of T SBM for analyses with missing data Robust to nonnormality, uses unrestricted model to compute scaling corrections (Rosseel, 2012) Asymptotically equivalent to Yuan & Bentler s (2000) T 2 (Yuan & Bentler, 2000) Implemented in Mplus via ESTIMATOR = MLR M3, May 2015 Comparing the Performance of the Mean- and Variance-Adjusted ML χ 2 Test Statistic with and without df Correction 7

8 Introduction SB Adjusted Test Statistic Satorra & Bentler (1986, 1988, 1994) T SBMV adjusts variance of T ML in addition to scaling its mean T SBMV = T ML d tr UΓ d = tr UΓ 2 tr UΓ 2 a T SBMV ~ approx. χ 2 d (same means and variances) Implemented in Mplus via ESTIMATOR = MLMV SATTERTHWAITE = ON M3, May 2015 Comparing the Performance of the Mean- and Variance-Adjusted ML χ 2 Test Statistic with and without df Correction 8

9 Introduction Second-Order Correction Asparouhov & Muthén (2010) Introduced in Mplus 6 as an alternative to T SBMV T AMMV = at ML + b a = d tr UΓ 2 b = d d tr UΓ 2 tr UΓ 2 a T AMMV ~ approx. χ d 2 (same means and variances) Results in same integer-valued df as ML model (= d) Implemented in Mplus via ESTIMATOR = MLMV and SATTERTHWAITE = OFF M3, May 2015 Comparing the Performance of the Mean- and Variance-Adjusted ML χ 2 Test Statistic with and without df Correction 9

10 Simulation Simulation Design Based on simulation in Curran, West, & Finch (1996) 5 sample sizes: [100,250,500,1000,5000] 3 population models (details on next slide) 3 data distributions: [MV normal, UV skew = 2 & kurtosis = 7, UV skew = 3 & kurtosis = 21] 2 numbers of items/factor: [3,7] 5 x 3 x 3 x 2 = 60 unique condition combinations 1,000 replications of each combination = 60,000 data sets M3, May 2015 Comparing the Performance of the Mean- and Variance-Adjusted ML χ 2 Test Statistic with and without df Correction 10

11 Simulation Population Models Population Model 1 Population Model 2 Population Model 3 M3, May 2015 Comparing the Performance of the Mean- and Variance-Adjusted ML χ 2 Test Statistic with and without df Correction 11

12 Simulation Model-Fitting Conditions 7 types of model-data concordance 5 correct specifications* 3 fitted models identical to the 3 population models 1 fitted model with an extra structural parameter 1 fitted model with two extra measurement parameters 2 misspecifications 1 misspecification by structural exclusion 1 misspecification by measurement exclusion] 5 test statistics [T ML, T MLR, T SBM, T SBMV, T AMMV ] 35 x 60,000 data sets = 2,100,000 fitted models M3, May 2015 Comparing the Performance of the Mean- and Variance-Adjusted ML χ 2 Test Statistic with and without df Correction 12

13 Simulation Performance Criteria Based on simulation in Curran, West, & Finch (1996) Obtained χ 2 statistic Expected χ 2 statistic Correct model specification model df Model misspecification model df + estimated non-centrality parameter % Relative bias: abs Obtained χ2 Expected χ 2 Expected χ 2 100% % Models rejected Correct model specification empirical Type-I error rate Model misspecification power All model-fitting in Mplus (Muthén & Muthén, 2014), everything else in R (R Core Team, 2014) M3, May 2015 Comparing the Performance of the Mean- and Variance-Adjusted ML χ 2 Test Statistic with and without df Correction 13

14 Simulation Results Correct Model Specification Normal Moderately Nonnormal Severely Nonnormal Obs. Exp. % % Obs. Exp. % % Obs. Exp. % % Size Estim. χ 2 χ 2 Bias Reject χ 2 χ 2 Bias Reject χ 2 χ 2 Bias Reject ML SBM SBMV AMMV MLR ML SBM SBMV AMMV MLR ML SBM SBMV AMMV MLR ML SBM SBMV AMMV MLR ML SBM SBMV AMMV MLR M3, May 2015 Comparing the Performance of the Mean- and Variance-Adjusted ML χ 2 Test Statistic with and without df Correction 14

15 Simulation Results Correct Model Specification Relative Bias > 10% (Curran et al., 1996) Normal Moderately Nonnormal Severely Nonnormal Obs. Exp. % % Obs. Exp. % % Obs. Exp. % % Size Estim. χ 2 χ 2 Bias Reject χ 2 χ 2 Bias Reject χ 2 χ 2 Bias Reject ML MLR SBM SBMV AMMV ML MLR SBM SBMV AMMV ML MLR SBM SBMV AMMV ML MLR SBM SBMV AMMV ML MLR SBM SBMV AMMV M3, May 2015 Comparing the Performance of the Mean- and Variance-Adjusted ML χ 2 Test Statistic with and without df Correction 15

16 Simulation Results Correct Model Specification Type I error >.064 (95% CI for α =. 05: [.036,.064]) (Savalei, 2010) Normal Moderately Nonnormal Severely Nonnormal Obs. Exp. % % Obs. Exp. % % Obs. Exp. % % Size Estim. χ 2 χ 2 Bias Reject χ 2 χ 2 Bias Reject χ 2 χ 2 Bias Reject ML MLR SBM SBMV AMMV ML MLR SBM SBMV AMMV ML MLR SBM SBMV AMMV ML MLR SBM SBMV AMMV ML MLR SBM SBMV AMMV M3, May 2015 Comparing the Performance of the Mean- and Variance-Adjusted ML χ 2 Test Statistic with and without df Correction 16

17 Simulation Results Measurement Inclusion Specification Normal Moderately Nonnormal Severely Nonnormal Obs. Exp. % % Obs. Exp. % % Obs. Exp. % % Size Estim. χ 2 χ 2 Bias Reject χ 2 χ 2 Bias Reject χ 2 χ 2 Bias Reject ML SBM SBMV AMMV MLR ML SBM SBMV AMMV MLR ML SBM SBMV AMMV MLR ML SBM SBMV AMMV MLR ML SBM SBMV AMMV MLR M3, May 2015 Comparing the Performance of the Mean- and Variance-Adjusted ML χ 2 Test Statistic with and without df Correction 17

18 Simulation Results Measurement Inclusion Specification Relative Bias > 10% (Curran et al., 1996) Normal Moderately Nonnormal Severely Nonnormal Obs. Exp. % % Obs. Exp. % % Obs. Exp. % % Size Estim. χ 2 χ 2 Bias Reject χ 2 χ 2 Bias Reject χ 2 χ 2 Bias Reject ML MLR SBM SBMV AMMV ML MLR SBM SBMV AMMV ML MLR SBM SBMV AMMV ML MLR SBM SBMV AMMV ML MLR SBM SBMV AMMV M3, May 2015 Comparing the Performance of the Mean- and Variance-Adjusted ML χ 2 Test Statistic with and without df Correction 18

19 Simulation Results Measurement Inclusion Specification Type I error >.064 (95% CI for α =. 05: [.036,.064]) (Savalei, 2010) Normal Moderately Nonnormal Severely Nonnormal Obs. Exp. % % Obs. Exp. % % Obs. Exp. % % Size Estim. χ 2 χ 2 Bias Reject χ 2 χ 2 Bias Reject χ 2 χ 2 Bias Reject ML MLR SBM SBMV AMMV ML MLR SBM SBMV AMMV ML MLR SBM SBMV AMMV ML MLR SBM SBMV AMMV ML MLR SBM SBMV AMMV M3, May 2015 Comparing the Performance of the Mean- and Variance-Adjusted ML χ 2 Test Statistic with and without df Correction 19

20 Simulation Results Structural Inclusion Specification Normal Moderately Nonnormal Severely Nonnormal Obs. Exp. % % Obs. Exp. % % Obs. Exp. % % Size Estim. χ 2 χ 2 Bias Reject χ 2 χ 2 Bias Reject χ 2 χ 2 Bias Reject ML SBM SBMV AMMV MLR ML SBM SBMV AMMV MLR ML SBM SBMV AMMV MLR ML SBM SBMV AMMV MLR ML SBM SBMV AMMV MLR M3, May 2015 Comparing the Performance of the Mean- and Variance-Adjusted ML χ 2 Test Statistic with and without df Correction 20

21 Simulation Results Structural Inclusion Specification Relative Bias > 10% (Curran et al., 1996) Normal Moderately Nonnormal Severely Nonnormal Obs. Exp. % % Obs. Exp. % % Obs. Exp. % % Size Estim. χ 2 χ 2 Bias Reject χ 2 χ 2 Bias Reject χ 2 χ 2 Bias Reject ML MLR SBM SBMV AMMV ML MLR SBM SBMV AMMV ML MLR SBM SBMV AMMV ML MLR SBM SBMV AMMV ML MLR SBM SBMV AMMV M3, May 2015 Comparing the Performance of the Mean- and Variance-Adjusted ML χ 2 Test Statistic with and without df Correction 21

22 Simulation Results Structural Inclusion Specification Type I error >.064 (95% CI for α =. 05: [.036,.064]) (Savalei, 2010) Normal Moderately Nonnormal Severely Nonnormal Obs. Exp. % % Obs. Exp. % % Obs. Exp. % % Size Estim. χ 2 χ 2 Bias Reject χ 2 χ 2 Bias Reject χ 2 χ 2 Bias Reject ML MLR SBM SBMV AMMV ML MLR SBM SBMV AMMV ML MLR SBM SBMV AMMV ML MLR SBM SBMV AMMV ML MLR SBM SBMV AMMV M3, May 2015 Comparing the Performance of the Mean- and Variance-Adjusted ML χ 2 Test Statistic with and without df Correction 22

23 Simulation Results Measurement Exclusion Misspecification Normal Moderately Nonnormal Severely Nonnormal Obs. Exp. % % Obs. Exp. % % Obs. Exp. % % Size Estim. χ 2 χ 2 Bias Reject χ 2 χ 2 Bias Reject χ 2 χ 2 Bias Reject ML SBM SBMV AMMV MLR ML SBM SBMV AMMV MLR ML SBM SBMV AMMV MLR ML SBM SBMV AMMV MLR ML SBM SBMV AMMV MLR M3, May 2015 Comparing the Performance of the Mean- and Variance-Adjusted ML χ 2 Test Statistic with and without df Correction 23

24 Simulation Results Measurement Exclusion Misspecification Relative Bias > 10% (Curran et al., 1996) Normal Moderately Nonnormal Severely Nonnormal Obs. Exp. % % Obs. Exp. % % Obs. Exp. % % Size Estim. χ 2 χ 2 Bias Reject χ 2 χ 2 Bias Reject χ 2 χ 2 Bias Reject ML MLR SBM SBMV AMMV ML MLR SBM SBMV AMMV ML MLR SBM SBMV AMMV ML MLR SBM SBMV AMMV ML MLR SBM SBMV AMMV M3, May 2015 Comparing the Performance of the Mean- and Variance-Adjusted ML χ 2 Test Statistic with and without df Correction 24

25 Simulation Results Measurement Exclusion Misspecification Power Normal Moderately Nonnormal Severely Nonnormal Obs. Exp. % % Obs. Exp. % % Obs. Exp. % % Size Estim. χ 2 χ 2 Bias Reject χ 2 χ 2 Bias Reject χ 2 χ 2 Bias Reject ML MLR SBM SBMV AMMV ML MLR SBM SBMV AMMV ML MLR SBM SBMV AMMV ML MLR SBM SBMV AMMV ML MLR SBM SBMV AMMV M3, May 2015 Comparing the Performance of the Mean- and Variance-Adjusted ML χ 2 Test Statistic with and without df Correction 25

26 Simulation Results Structural Exclusion Misspecification Normal Moderately Nonnormal Severely Nonnormal Obs. Exp. % % Obs. Exp. % % Obs. Exp. % % Size Estim. χ 2 χ 2 Bias Reject χ 2 χ 2 Bias Reject χ 2 χ 2 Bias Reject ML SBM SBMV AMMV MLR ML SBM SBMV AMMV MLR ML SBM SBMV AMMV MLR ML SBM SBMV AMMV MLR ML SBM SBMV AMMV MLR M3, May 2015 Comparing the Performance of the Mean- and Variance-Adjusted ML χ 2 Test Statistic with and without df Correction 26

27 Simulation Results Structural Exclusion Misspecification Relative Bias > 10% (Curran et al., 1996) Normal Moderately Nonnormal Severely Nonnormal Obs. Exp. % % Obs. Exp. % % Obs. Exp. % % Size Estim. χ 2 χ 2 Bias Reject χ 2 χ 2 Bias Reject χ 2 χ 2 Bias Reject ML MLR SBM SBMV AMMV ML MLR SBM SBMV AMMV ML MLR SBM SBMV AMMV ML MLR SBM SBMV AMMV ML MLR SBM SBMV AMMV M3, May 2015 Comparing the Performance of the Mean- and Variance-Adjusted ML χ 2 Test Statistic with and without df Correction 27

28 Simulation Results Structural Exclusion Misspecification Power Normal Moderately Nonnormal Severely Nonnormal Obs. Exp. % % Obs. Exp. % % Obs. Exp. % % Size Estim. χ 2 χ 2 Bias Reject χ 2 χ 2 Bias Reject χ 2 χ 2 Bias Reject ML MLR SBM SBMV AMMV ML MLR SBM SBMV AMMV ML MLR SBM SBMV AMMV ML MLR SBM SBMV AMMV ML MLR SBM SBMV AMMV M3, May 2015 Comparing the Performance of the Mean- and Variance-Adjusted ML χ 2 Test Statistic with and without df Correction 28

29 Discussion Discussion: Relative Bias All estimators acceptable as long as data were normal Correct model specifications: T AMMV performed best Incorrect model specifications With normal data and small N, T AMMV performed best With nonnormal data, not clear if there is a best estimator M3, May 2015 Comparing the Performance of the Mean- and Variance-Adjusted ML χ 2 Test Statistic with and without df Correction 29

30 Discussion Discussion: Type-I Error Rate T SBMV slightly outperformed T AMMV, more noticeable for nonnormal data All estimators performed poorly with normal data & N = 100 T ML and T MLR acceptable only for large N and normal data M3, May 2015 Comparing the Performance of the Mean- and Variance-Adjusted ML χ 2 Test Statistic with and without df Correction 30

31 Discussion Discussion: Power Every estimator had terrible power, but T ML and T MLR were the least terrible Model misspecifications might have been too minimal Population parameter values might not have been large enough M3, May 2015 Comparing the Performance of the Mean- and Variance-Adjusted ML χ 2 Test Statistic with and without df Correction 31

32 Discussion Recommendations Minimum N = 500 Data are rarely normal, so use a corrected test statistic Best performance was for T SBMV and T AMMV, but difficult to say if one was clearly better than the other M3, May 2015 Comparing the Performance of the Mean- and Variance-Adjusted ML χ 2 Test Statistic with and without df Correction 32

33 Discussion Future Directions Results with more indicators per factor Likelihood ratio test statistic for nested models Fit indices based on the chi-square statistic (e.g., RMSEA) Missing data problematic: only T MLR will work M3, May 2015 Comparing the Performance of the Mean- and Variance-Adjusted ML χ 2 Test Statistic with and without df Correction 33

34 References Asparouhov, T., & Muthén, B. (2010). Simple second order chi-square correction. Technical report. Curran, P. J., West, S. G., & Finch, J. F. (1996). The robustness of test statistics to nonnormality and specification error in confirmatory factor analysis. Psychological Methods, 1, Micceri, T. (1989). The unicorn, the normal curve, and other improbable creatures. Psychological Bulletin, 105, Muthén, L. K., & Muthén, B. O. ( ). Mplus user s guide (7th ed.). Los Angeles, CA: Muthén & Muthén. R Core Team. (2014). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL Rosseel, Y. (April, 2012). lavaan: A brief user s guide. Retrieved from Satorra, A., & Bentler, P. M. (1986). Some robustness properties of goodness of fit statistics in covariance structure analysis. ASA 1986 Proceedings of the Business and Economic Statistics Section (p ). Alexandria, VA: American Statistical Association. Satorra, A., & Bentler, P. M. (1988). Scaling corrections for chi-square statistics in covariance structure analysis. ASA 1988 Proceedings of the Business and Economic Statistics Section (p ). Alexandria, VA: American Statistical Association. Satorra, A., & Bentler, P. M. (1994). Corrections to test statistics and standard errors in covariance structure analysis. In A. von Eye & C. C. Clogg (Eds.), Latent variable analysis: Applications for developmental research (pp ). Sage Publications. Savalei, V. (2010). Small sample statistics for incomplete nonnormal data: Extensions of complete data formulae and a Monte Carlo comparison. Structural Equation Modeling, 17, Yuan, K.-H., & Bentler, P. M. (2000). Three likelihood-based methods for mean and covariance structure analysis with nonnormal missing data. Sociological Methodology, 30, M3, May 2015 Comparing the Performance of the Mean- and Variance-Adjusted ML χ 2 Test Statistic with and without df Correction 34

35 Contact information: Jonathan M. Lehrfeld Heining Cham THANK YOU! M3, May 2015 Comparing the Performance of the Mean- and Variance-Adjusted ML χ 2 Test Statistic with and without df Correction 35