Individualized Self-learning Program to Improve Primary Education: Evidence from a Randomized Field Experiment in Bangladesh

Transcription

1 Individualized Self-learning Program to Improve Primary Education: Evidence from a Randomized Field Experiment in Bangladesh BEP x Kumon Project Research Team: Yasuyuki Sawada, An Le, Minhaj Mahmud, Mai Seki, and Hikaru Kawarazaki North American Summer Meetings of the Econometric Society June 15th, 2017

2 Disclaimer The views expressed are those of the authors. They do not necessarily reflect the views of the affiliated institutions.

3 Motivation: Improved Universal Primary Education MDGs (until 2015): Impressive progress in the primary completion rate!

4 Motivation: Bangladesh on MDG Goal 2 Figure 1: Planning commission Bangladesh, 2015

5 Motivation: Global Learning Crisis Global learning crisis (WDR 2018) 38% of primary school completers do not have adequate level of learning outcomes) SDGs (post 2015):

6 Research Questions What happens if we integrate one of the most successful non-formal education methods in the world to disadvantaged schools in very low income communities? To target poor students studying at BRAC primary schools (BPS) To examine the KUMON Method of Learning (Kumon), a globally popular non-formal education program To investigate both cognitive and non-cognitive abilities To adopt an RCT-based experiment

7 Preview Overall impact: Robust and better learning ability. Impact on teachers: Better assessment capacities of students performance, mitigating teachers stereotyping behaviors. Relevance to the literature: Pedagogical interventions that match teaching to students learning works (Kremer et al. 2011; USAID 2011; Banerjee et al. 2016, NBER No ; Banerjee et al. 2007; 3ie 2016) Policy contribution: Complementary to lecture-style education for solving learning crisis

8 Outline 1 BEP x Kumon Project 2 Research design 3 Empirical specification 4 Data 5 Results 6 Conclusion

9 BRAC Education Program (BEP) BRAC is one of the largest NGOs in the world, started in Bangladesh in 1972 and now operating across 11 countries [D]edicated to empowering people living in poverty (Source: BRAC website) BRAC works for... Economic Development (e.g. microfinance (Pitt and Khandker, 1998)) Targeting Ultra Poor (Barnerjee et al., 2015) Education (e.g. non-formal primary education especially for poor, rural, or disadvantaged children, and drop-outs) BRAC Education Program (BEP) Public Health (e.g. public healthcare focused on curative care) Disaster Relief (e.g. Distributing emergency relief after Cyclone 2007) Skills Development Program

10 BRAC Primary School BEP uses National Curriculum and Textbook Board (NCTB) textbooks from grade four to five and grade one to three have their own textbooks followed by the National Curriculum.. A BPS in Bangladesh during a Kumon session

11 Kumon A Kumon center in India. An example of a worksheet

12

13

14 Implementation of Kumon in BPS The Kumon Program was implemented in selected BPS (30 students per 1 school) For 30 min before regular classes everyday Kumon BEP joint project: 2 Marking Assistants were employed per 1 school to aid BPS Teachers in marking the worksheets. Field Staff were assigned to assist and follow-up on BPS Teachers. 3 days of preparatory training for BPS Teachers and Field Staff were held prior to launching the program. In addition, 3 follow-up training sessions were held during the implementation period.

15 Research Design RCT to evaluate the applicability of the Kumon Method of Learning in strengthening the cognitive and non-cognitive abilities of the disadvantaged students studying at BRAC primary schools (BPS): 34 schools are randomly selected out of 179 schools in 4 branches of BEP with grade specific strata. RCT (17 treatment & 17 control) Around 1,000 students and 34 teachers in the 3rd or 4th grades 8 months tracking data

16 Study Timeline c Yasuyuki Sawada, An Le, Minhaj Mahmud, Mai Seki, and Hikaru Kawarazaki. mimeographed, Tokyo, Japan, August 2016.

17 Homogenous Impacts on Cognitive and Non-Cognitive Abilities ANCOVA: Cross-sectional regression: Y it = α 0 + βy it 1 + δd i + ε it (1) Y i = α 0 + δd i + ε i (2) where Y = level of cognitive and non-cognitive outcomes

18 Heterogeneous Impacts on Cognitive and Non-Cognitive Abilities ANCOVA Y i = α 1 + βy it 1 + δ 0 d i + δ j d j X ij + ε i (3) Cross-sectional regression: Y i = α 1 + δ 0 d i + δ j d j X ij + ε i (4) where X = Student s baseline cognitive score; Student s baseline non-cognitive score

19 Summary Statistics for Grade 3 Grade 3 Summary Statistics on Students Performance Measurement Baseline Endline Control Mean Treatment Mean Difference Observations Control Mean Treatment Mean Difference Observations DT Score per Minute *** *** 445 (0.166) (0.895) PTS Score ** 473 (0.239) (3.739) Rosenberg Non-cognitive Score (0.472) (0.0811) CPCS Non-cognitive Score (0.451) (0.0773) GRIT Non-cognitive Score (0.271) (0.148) Demographics Male (0.0268) Wealth Index (0.323)

20 Summary Statistics for Grade 4 Grade 4 Summary Statistics on Students Performance Measurement Baseline Endline Control Mean Treatment Mean Difference Observations Control Mean Treatment Mean Difference Observations DT Score per Minute *** 366 (0.211) (1.336) PTS Score *** 364 (0.212) (2.529) Rosenberg Non-cognitive Score (0.174) (0.102) CPCS Non-cognitive Score (0.179) (0.0913) GRIT Non-cognitive Score (0.138) (0.0985) Demographics Male (0.0462) Wealth Index (0.316)

21 Sample Attrition The attrition rates are 11.3% (57/478) for the treated group and 15.6% (82/526) for the control group. To calculate attrition rates, we count a student as a Dropout from the sample if he/she didn t take either DT test or PTSII in endline. Non-systematic attrition: None of the baseline outcome variables predict the status of becoming the Dropouts by the endline Dep. Var Dropout Dropout Dropout Dropout Grade 3 Grade 3 Grade 4 Grade 4 OLS Probit OLS Probit (1) (2) (3) (4) Baseline DT Score ( ) (0.0356) (0.0117) (0.0494) Baseline PTS Score ( ) ( ) ( ) (0.0126) Baseline Teacher Evaluation (0.0224) (0.149) (0.0272) (0.117) Male (0.0207) (0.130) (0.0305) (0.129) Constant ** (0.0926) (0.675) (0.171) (0.797) Num of Obs R-squared Cluster standard errors in parentheses. *** p<0.01, ** p<0.05, * p<0.1

22 Missing Value Pattern in Survey Questions survey1 survey2 survey3 survey4 survey5 survey6 survey7 survey8 survey9 survey10 survey11 survey12 survey13 survey14 survey15 survey16 survey17 survey18 survey19 survey20 survey21 survey22 survey23 survey24 survey25 survey26 survey Students ID y z Baseline Survey survey1 survey2 survey3 survey4 survey5 survey6 survey7 survey8 survey9 survey10 survey11 survey12 survey13 survey14 survey15 survey16 survey17 survey18 survey19 survey20 survey21 survey22 survey23 survey24 survey25 survey26 survey Students ID y z Endline Survey c Yasuyuki Sawada, An Le, Minhaj Mahmud, Mai Seki, and Hikaru Kawarazaki. mimeographed, Tokyo, Japan, August 2016.

23 1-1. Significant Impacts on Cognitive Abilities ANCOVA Dep. Var Impacts of KUMON Program on Students Outcomes Grade 3 Grade 4 DT Score PTS Score DT Score PTS Score DT Score PTS Score DT Score PTS Score Treatment 2.048*** 0.736** 1.514** 0.870** 2.876*** 1.276*** 2.425*** 1.001*** (0) (0.0340) (0.0200) (0.0440) (0) (0) ( ) (0.322) Lag values *** *** *** ** * *** ** ( ) ( ) ( ) (0.0180) (0.0561) (0.213) (0.150) (0.286) Treatment*Initial Cog (0.138) (0.963) (0.935) (0.336) Treatment*Initial Noncog (0.132) (0.719) (0.503) (0.140) Constant 0.755* 1.088** ** (0.0921) (0.0120) (0.110) (0.0120) (0.266) (0.287) (0.284) (0.305) Num of Obs R-squared p-value in parentheses and is calculated using wild cluster bootstrap

24 1-2. Significant Impacts on Cognitive Abilities Endline on Treatment dummy Impacts of KUMON Program on Students Outcomes Dep. Var Grade 3 Grade 4 DT Score PTS Score DT Score PTS Score Treatment 2.251*** 0.675* 2.831*** 1.221*** ( ) (0.0661) (0) (0) Treatment*Initial Cog ** (0.126) (0.364) (0.844) (0.0250) Treatment*Initial Noncog (0.288) (0.519) (0.526) (0.491) Constant *** (0.248) ( ) (0.242) (0.100) Num of Obs R-squared p-value in parentheses and is calculated using wild cluster bootstrap

25 2-1. Insignificant Impacts on Non-Cognitive Abilities ANCOVA, homogenous effects Dep. Var Impacts of KUMON Program on Students Outcomes Grade 3 Grade 4 Rosenberg CPCS GRIT Rosenberg CPCS GRIT Treatment (0.953) (0.933) (0.649) (0.436) (0.276) (0.665) Lag values *** *** *** *** *** *** ( ) ( ) ( ) ( ) ( ) ( ) Constant ** *** (0.589) (0.545) (0.482) (0.0320) ( ) (0.370) Num of Obs R-squared p-value in parentheses and is calculated using wild cluster bootstrap

26 2-2. Insignificant Impacts on Non-Cognitive Abilities ANCOVA, heterogeneous effects Dep. Var Impacts of KUMON Program on Students Outcomes Grade 3 Grade 4 Rosenberg CPCS GRIT Rosenberg CPCS GRIT Treatment (0.837) (0.505) (0.889) (0.276) (0.372) (0.498) Lag values *** *** *** *** *** *** ( ) ( ) ( ) ( ) ( ) ( ) Treatment*Initial Cog * (0.569) (0.170) (0.625) (0.853) (0.865) (0.0701) Treatment*Initial Noncog ** (0.921) (0.376) (0.579) (0.0220) (0.488) (0.296) Constant ** *** (0.577) (0.623) (0.507) (0.0320) ( ) (0.340) Num of Obs R-squared p-value in parentheses and is calculated using wild cluster bootstrap

27 2-3. Insignificant Impacts on Non-Cognitive Abilities Endline on Treatment dummy Impacts of KUMON Program on Students Outcomes Dep. Var Grade 3 Grade 4 Rosenberg CPCS GRIT Rosenberg CPCS GRIT Treatment (0.933) (0.805) (0.573) (0.360) (0.242) (0.997) Treatment*Initial Cog (0.232) (0.130) (0.583) (0.385) (0.568) (0.257) Treatment*Initial Noncog * (0.144) (0.448) (0.304) (0.878) (0.782) (0.0791) Constant ** (0.721) (0.789) (0.901) (0.176) (0.0360) (0.406) Num of Obs R-squared p-value in parentheses and is calculated using wild cluster bootstrap

28 2-4. Significant Impacts on Individual Survey Questions Endline on Treatment dummy Dep. Var Impacts of KUMON Program on Students Outcomes Endline on Treatment dummy Question 7 Question 1 Question 2 Question 7 Question 1 Question 2 Grade 3 Grade 4 Grade 4 Grade 3 Grade 4 Grade 4 Treatment 0.322** 0.342** 0.349** 0.385* 0.232* 0.335** (0.0240) (0.0420) (0.0240) (0.0601) (0.0841) (0.0460) Treatment*Initial Cog (0.412) (0.944) (0.934) Treatment*Initial Noncog *** ( ) (0.295) (0.508) Constant *** *** *** *** ** *** ( ) ( ) ( ) ( ) (0.0220) ( ) Num of Obs R-squared p-value in parentheses and is calculated using wild cluster bootstrap

29 Conclusion Non-formal education has the potential to contribute to the enhancement of learning outcomes. This study finds robust improvements in students cognitive with mixed findings in non-cognitive outcomes by the Kumon Method of Learning among the students of BRAC Primary Schools. Through this intervention, teachers become better in predicting students performance. This could be one of the paths for the improved student learning outcomes (avoiding teachers stereotyping behaviors). This evidence contributes to the literature that finds pedagogical interventions that match teaching to students learning works (Kremer et al. 2011, AER; USAID 2011; Banerjee et al. 2016, NBER No ; Banerjee et al. 2007, QJE; 3ie 2016: 27 28).

30 Future Analysis Further investigation of the heterogeneous impacts: Relationship with parents interest to children s education, environment of household, and characteristics of teachers Relationship with household s other educational expenditure to children s education Analysis of the progress path among the treated: Quizzes (all), Report sheets and Grading books (the treated only) More robustness checking Controlling for dropouts

31 Thank you! c Yasuyuki Sawada, An Le, Minhaj Mahmud, Mai Seki, and Hikaru Kawarazaki. mimeographed, Tokyo, Japan, August 2016.

32 PTSII Survey Questions Non-Cognitive Ability Measurements Number Question in English CPCS RSES GritScale 1 I did well in this test. 2 I can do most things better than other people. x x 3 There are many things about myself I can be proud of. x x 4 I feel that I cannot do anything well no matter what I do. x x 5 I believe I can be someone great. x 6 I don t think I am a helpful person. x x 7 I can confidently express my opinion. x 8 I don t think I have that many good qualities. x x 9 I am always worried that I might fail. x x 10 I am confident about myself. x x 11 I am satisfied with myself. x x Even if I fail, I think I can get better and better at things 12 if I keep trying 13 I like to do calculations. x 14 I can calculate in my head when I go shopping. x 15 I think speed is important when solving problems. x When studying, I believe everything will go well if I 16 correctly follow instruction 17 I am more motivated when people praise me. 18 I always volunteer in class. 19 I enjoy studying. 20 School is fun. 21 I do things better when I have a goal. 22 There are many things I want to learn more about. a. I have a role model around me. 23 b. There is someone around me who I want to be like. I always have someone who I can go to for advice 24 when I am having trouble with my studies. a. There is someone around me who I don t want to lose against. 25 b. There is someone around me who I am always competing with. 26 I always try to do something when things don t go as expected. It doesn t matter whether I fail in the beginning because 27 I believe that things will eventually work out.

33 Children s Perceived Competence Scale: CPCS Developmental psychologist, Harter invented it in 1979 Japanese version was made and used widely in the field of social psychology Competence (White, 1959) Realising their own active work as I can do it! 28 contents, 4 choices. (1.Strongly Agree 2.Agree 3.Disagree 4.Strongly Disagree)

34 Rosenberg Self-Esteem Scale: RSES Rosenberg invented this in 1965; measuring generally and quantitatively the extent of self-esteem Used in many countries and also for comparing internationally 10 contents, 4 choices. (1.Strongly Agree 2.Agree 3.Disagree 4.Strongly Disagree) The higher the score is, the more positively they think themselves entirely and more highly they evaluate themselves

35 Grit Scale Ability for setting long-term goals and not-giving-up till they are achieved using any methods and trying as hard as possible Huge impact on children s growth and future success Angela Lee Duckworth, Grit: The power of passion and perseverance 12 contents, 5 choices. (1 Very much like me 2 Mostly like me 3 Somewhat like me 4 Not much like me 5 Not like me at all)