SOS3003 Applied data analysis for social science Lecture note Erling Berge Department of sociology and political science NTNU.

Transcription

1 SOS3003 Appled data analyss for socal scence Lecture note Erlng Berge Department of socology and poltcal scence NTNU Erlng Berge Lterature Robust Regresson Hamlton Ch 6 p Erlng Berge Erlng Berge

2 Robust Regresson Has been developed to work well n stuatons where OLS breaks down. Where the OLS assumptons are satsfed robust regresson are not as good as OLS, but not by very much Even f robust regresson s better suted for those who do not want to put much effort nto testng the assumptons, t s so far dffcult to use Robust regresson has focused on resduals wth heavy tals (many cases wth hgh nfluence on the regresson) Erlng Berge Regresson of mortalty on ar polluton 1100,00 AGE-ADJUSTED MORTALITY/100K 1000,00 900,00 OLS: Y= ln(ar polluton) SanDego SanFrancsco LosAngeles Fgure 6.1 Hamlton 800,00 SanJose R Sq Lnear = 0,023 0,00 1,00 2,00 3,00 4,00 5,00 6,00 7,00 LN_hc_polluton Erlng Berge Erlng Berge

3 Robust regresson of mortalty on ar polluton 1100,00 Robust Regresson: AGE-ADJUSTED MORTALITY/100K LN_hc_polluton Robust_Pred_Y LN_hc_polluton Y= ln(ar polluton) 1000,00 OLS: Y= , ln(ar polluton) Fgure 6.2 Hamlton 800,00 R Sq Lnear = 0,023 0,00 1,00 2,00 3,00 4,00 5,00 6,00 7,00 Erlng Berge Robust regresson and SPSS SPSS do not have a partcular routne that performs robust regresson It can possbly be done wthn the Generalzed lnear models procedure <but I have not tested t our> It can be done by weghted OLS regresson, but then t s requred that we make the weght functons and go through the teratons one by one ncludng computaton of weghts every tme Ths procedure wll be outlned below Erlng Berge Erlng Berge

4 ROBUST AND RESISTANT RESISTANT methods are not affected by small errors or changes n the sample data ROBUST methods are not affected by small devatons from the assumptons of the model Most resstant estmators are also robust n relaton to the assumpton about normally dstrbuted resduals OLS s nether ROBUST nor RESISTANT Erlng Berge Outlers s a problem for OLS Outlers affect the estmates of Parameters Standard errors (standard devaton of parameters) Coeffcent of determnaton Test statstcs And many other statstcs Robust regresson tres to protect aganst ths by gvng less weght to such cases, not by excludng them Erlng Berge Erlng Berge

5 Protecton aganst NON-NORMALE resduals Robust methods can help when the tals n the dstrbuton of the resduals are heavy,.e. when t s too many outlers compared to the normal dstrbuton Unusual X-values have leverage and may cause problems But for other causes of non-normalty robust methods wll not help Erlng Berge Estmaton methods for robust regresson M-estmaton (maxmum lkelhood) mnmzes a weghted sum of the resduals. Ths can be approxmated by the weghted least squares method (WLS) R-estmaton (based on rank) mnmzes a sum where a weghted rank s ncluded. The method s more dffcult to use than M- estmaton L-estmaton (based on quantles) uses lnear functons of the sample order statstcs (quantles) Erlng Berge Erlng Berge

6 IRLS- Iterated Reweghted Least Squares M-estmaton by means of IRLS needs 1. Start values from OLS. Save the resduals 2. Use OLS resduals to fnd weghts. Larger resduals gves less weght 3. Fnd new parameter values and resduals wth WLS 4. Go to step 2 and fnd new weghts from the new resduals, go on to step 3 and 4, untl changes n the parameters become small Iteraton: to repeat a sequence of operatons Erlng Berge IRLS IRLS s n theory equvalent to M- estmaton To use the method we need to compute Scaled resduals, u, and a Weght functon, w,that gves least weght to the largest resduals Erlng Berge Erlng Berge

7 Scalng of resduals I e u = s Scaled resdual u s s the scale factor and e resdual The scale factor n OLS s the estmate of the standard error of the resdual: nb! s e s not resstant A resstant alternatve s based on MAD, "medan absolute devaton" s e = RSS n K ( ) MAD = medan e medan e Erlng Berge Scalng of resduals II ( ) MAD = medan e medan e The scale factor (standard error of the dstrbuton) Usng a resstant estmate wll be s = MAD/ = 1.483MAD and the scaled resdual u =[e / s ] = (0.6745*e )/MAD In a normal dstrbuton s= MAD/ wll estmate the standard error correctly lke s e In case of non-normal errors s= MAD/ wll be better. Ths s a resstant estmate, s e s not resstant Erlng Berge Erlng Berge

8 Weght functons I Propertes s measured n relaton to OLS on normally dstrbuted errors. The method should be almost as good as OLS on normally dstrbuted errors and much better when the errors are nonnormal Propertes are determned by a calbraton constant (c n the formulas) Erlng Berge Weght functons II OLS-weghts: w = 1 for all Huber-weghts: weghts down when the scaled resdual s larger than c, c=1,345 gves 95% of the effcency of OLS on normally dstrbuted errors Tukey s b-weghted estmates get 95% of the effcency of OLS on normally dstrbuted errors by gradually weghtng down scaled errors untl u c = and by droppng cases where the resdual s larger Erlng Berge Erlng Berge

9 Huber-weghts w w = 1 u c c = u > c u =for alle Erlng Berge w w Tukey weghts 2 u = 1 u c = 0 u > c = for alle 2 c Tukey weghtng n IRLS s senstve for start values of the parameters (one may end up at local mnma) Erlng Berge Erlng Berge

10 Standard errors and tests n IRLS The WLS program cannot estmate standard errors and test statstcs correctly by IRLS A procedure that works s descrbed by Hamlton on page Erlng Berge Use of Robust Estmaton If OLS and Robust estmates are dfferent t means that outlers have nfluence on the OLS results makng them unrelable. Results cannot be trusted Robust predcted values wll better portray the bulk of the data Robust resduals wll better at dscoverng whch cases are unusual Weghts from the robust regresson wll show whch cases are outlers OLS and RR can support each other Erlng Berge Erlng Berge

11 Mortalty regressed on ar polluton Effect of hgh leverage Fg 6.9 Hamlton: OLS and RR on untransformed data AGE-ADJUSTED MORTALITY/100K 1100, ,00 900,00 SanDego SanFrancsco robust LosAngeles 800,00 SanJose R Sq Lnear = 0,031 0,00 100,00 200,00 300,00 400,00 500,00 600,00 700,00 RELATIVE HC POLLUTION POTENTIAL Erlng Berge Fg 6.10 Hamlton: OLS and RR on untransformed data when two outlers are removed Mortalty regressed on ar polluton AGE-ADJUSTED MORTALITY/100K 1100, ,00 900,00 robust SanDego 800,00 SanJose R Sq Lnear = 7,116E-4 0,00 30,00 60,00 90,00 120,00 150,00 RELATIVE HC POLLUTION POTENTIAL Erlng Berge Erlng Berge

12 RR do not protect aganst leverage RR wth M-estmaton protects aganst unusual y- values (outlers) but not necessarly aganst unusual x-values (leverage) Efforts to test and dagnose are stll needed (heteroscedastcty s stll a problem for IRLS) Studes of the data and transformaton to symmetry wll reduce the rsk of problems appearng No method s safe f t s used wthout forethought and dagnostc studes of data Erlng Berge Robust Multppel Regresjon X 1 RELATIVE HC POLLUTION POTENTIAL (natural log) X 2 AVG. YEARLY PRECIP. INCHES X 3 AVG. JANUARY TEMPERATURE, F X 4 MEDIAN EDUCATION OF POP 25+ X 5 % NON-WHITE (square root) X 6 POPULATION PER HOUSEHOLD X 7 % 65 AND OVER X 8 % SOUND HOUSING UNITS X 9 PEOPLE PER SQUARE MILE (natural log) X 10 AVG. JULY TEMPERATURE, F X 11 % WHITE COLLAR EMPLOYMENT X 12 % FAMILIES WITH INCOME<$3000 (negatve recprocal root) X 13 AVG RELATIVE HUMIDITY, % Erlng Berge Erlng Berge

13 Multple OLS regresson wth transformed varables: effect of transformaton 7,00 8,00 6,00 5,00 6,00 LN_hc_polluton 4,00 3,00 2,00 SQRT_pct_non_whte 4,00 2,00 1,00 0,00 0,00 0,00 100,00 200,00 300,00 400,00 500,00 600,00 700,00 RELATIVE HC POLLUTION POTENTIAL ln of ar polluton 0,00 10,00 20,00 30,00 40,00 % NONWHITE Square root of % non-whte Erlng Berge OLS wth backward elmnaton gves Dependent Varable: AGE-ADJUSTED MORTALITY/100K B Std. Error t Sg. (Constant) LN_hc_polluton AVG. YEARLY PRECIP. INCHES AVG. JANUARY TEMPERATURE, F MEDIAN EDUCATION OF POP 25+ SQRT_pct_non_whte 986,261 17,469 2,352-2,132-17,958 27,335 82,674 4,636,640,504 6,204 4,398 11,929 3,768 3,677-4,228-2,895 6,215,000,000,001,000,005,000 Robust regresson gves predcted y: Y= x x x x x 5 Erlng Berge Erlng Berge

14 Multple OLS regresson wth transformed varables Leverage plot of resdual from mortalty (y) and resdual of ln_ar_polluton (x) Los Angeles and San Francsco are no longer outlers Unstandardzed Resdual 100, , , , ,00000 FortWorth Dallas Wchta Albany NewOrleans Lancaster Boston LosAngeles SanFrancsco R Sq Lnear = 0,208-4, , , , , , ,00000 Unstandardzed Resdual Erlng Berge Four estmates of the relatonshp mortalty ar polluton Effect of ar polluton OLS 1 varable varables Robust Note that n RR the bvarate regresson comes pretty close to the result of the multvarate regresson In the fve-varable model there are new cases wth nfluence on the lne of regresson Removng the 5 cases that have the hghest leverage parameter (h ) do not gve substantal changes n the coeffcents Erlng Berge Erlng Berge

15 Robust Regresson vs Bounded Influence Regresson Robust Regresson protect aganst the effect of outlers (unusual y-values) f these do not go together wth unusual x- values Bounded Influence Regresson s desgned to protect aganst nfluence from unusual combnatons of x-values Erlng Berge BI - Bounded Influence Regresson BI-methods are made to lmt the nfluence of hgh leverage cases (large h = hgh leverage) The smplest way of dong ths s to modfy the Huber-weghts or Tukey-weghts n the IRLS procedure for RR (robust regresson) wth a factor based on the leverage statstc Erlng Berge Erlng Berge

16 Bounded nfluence: modfcaton of weghts Expand the weght functon wth a weght based on the leverage statstc h w H = 1 f h c H w H = (c H / h ) f h > c H c H s often set to the 90% percentle n the dstrbuton of h Then the IRSL weght becomes w w H where w s ether the Tukey- or Huber-weght that changes from teraton to teraton whle w H s constant Erlng Berge Bounded nfluence as a dagnostc tool Estmaton of standard errors and test statstcs becomes even more complcated than for the M-estmators mentoned above We can use BI estmates as a descrptve tool to check up on other estmates One (somewhat) extreme example: PCB polluton n rver mouths n 1984 and 1984 (Hamlton table 6.4) Erlng Berge Erlng Berge

17 Fg 6.15 and 6.16 Hamlton 800,00 800,00 Boston Harbor Boston Harbor 600,00 600,00 TOTAL PCBS 1985 (PPB) 400,00 Robust M- estmate TOTAL PCBS 1985 (PPB) 400,00 BI regresson 200,00 200,00 R Sq Lnear = 0,384 0,00 0,00 0, , , , ,00 TOTAL PCBS 1984 (PPB) 0, , , , ,00 TOTAL PCBS 1984 (PPB) Erlng Berge OLS and BI estmates wth transformed varables gve about the same result lnpcbs1985 Fg 6.17 Hamlton 7,00 6,00 5,00 Delaware Bay 4,00 3,00 BI lnja OLS Boston Harbor 2,00 1,00 R Sq Lnear = 0,878 0,00 0,00 2,00 4,00 6,00 8,00 10,00 lnpcbs1984 Erlng Berge Erlng Berge

18 Conclusons When data have many outlers robust methods wll have better propertes than OLS They are more effectve and gve more accurate confdence ntervals and tests of sgnfcance Robust regresson can be used as a dagnostc tool If OLS and RR agree we can have more confdence to the OLS results If they dsagree we wll Know that a problem exst Have a model that fts the data better and dentfes the outlers better Robust methods does not protect aganst problems that are due to curvlnear or non-lnear models, heteroscedastcty, and autocorrelaton Erlng Berge Erlng Berge