the lower secondary school cycle.
11
The second test was a vocabulary test based on picture recognition. This test asked respondents to identify the picture corresponding
to a word which the enumerator read out loud. For each word the respondent was then asked to select from a choice of four pictures. While the initial words are relatively
easy to identify “shoulder,” “arrow,” “hut” the test is structured such that items be- come increasingly diffi cult. Later words in the test we administered included, for
example, “speed,” “selecting,” “adjustable”. There were in total 72 words that each applicant was asked to identify. The third test is a test of puzzles and shapes loosely
based on the Ravens Progressive Matrices.
12
These tests are not linked to the curricu- lum taught in Cambodia’s lower secondary schools. We normalize the scores on all
three tests by subtracting the mean and dividing by the standard deviation of nonre- cipients. The Cronbach alpha values for all of our tests are reasonably high—0.68 for
the math test, 0.90 for the vocabulary test, and 0.65 for the test of puzzles and shapes.
13
The household survey also collected data on adolescent mental health using an ad- aptation of the Center for Epidemiological Studies Depression scale CESD, a widely
used measure of depression Radloff 1977.
14
Subjective social status was assessed using the “MacArthur ladders.”
15
Adolescents were shown a picture of a ladder with ten rungs, and were told that higher rungs correspond to higher socioeconomic status.
They were asked to place themselves on the ladder in relation to everyone in their communities, and in relation to everyone in Cambodia. Finally, we asked respondents
if they were married or had children.
III. Identifi cation Strategy
The identifi cation strategy we use in this paper is based on regression
discontinuity RD. These regressions take the following form: 1 Y
is
= α
s
+ f C
s
+ βIT
is
= 1 +
ε
is
where Y
is
is an outcome for example, school enrollment, or the grades of completed schooling for child i who applied to CSP school s;
α
s
is a set of fi xed effects for lower
11. The test questions ranged from simple addition and subtraction of two- and three- digit numbers, reading the numbers off a bar chart or a graph, interpreting the area of shapes on a grid, manipulating fractions, and
completing a progressive sequence of shapes. 12. The Ravens has been applied in many settings in the developing world. It is considered a test of cogni-
tive capacity or general intelligence, and it is therefore not clear whether we would expect scholarships to improve scores on this test.
13. Cronbach’s alpha is the average covariance across the item responses, divided by the sum of the average variance of all items plus the average covariance across all items. It is often used as a measure of the reli-
ability of a test, although it is best interpreted here as the extent to which the various items on the test capture a single underlying concept such as ability.
14. Specifi cally, the survey included 12 questions about mental health such as “Over the past month have you been feeling unhappy or depressed?”, which respondents answered as “always,” “sometimes,” or
“never,” which we code with values of 1, 2, or 3. The total score is then the sum of all 12 scores, normalized by subtracting the mean and dividing by the standard deviation of nonrecipients. Our results are robust to
different ways of aggregating the individual responses. 15. For a description and bibliography of papers that use MacArthur ladders, see the MacArthur Founda-
tion’s Network on SES and Health website: http:www.macses.ucsf.eduResearchPsychosocialnotebook subjective.html.
secondary schools; f C
s
is a fl exible formulation of the control function; IT
is
= 1 is
an indicator variable that takes on the value of one if a child was offered a CSP schol- arship; and
ε
is
is the regression error term. In this set- up, the coeffi cient β is a measure of the impact of the scholarship. To allow for different slopes, we interact the control
function with the indicator variable for children who were eligible for scholarships. Standard errors are clustered at the level of the primary feeder school.
Unusually, in our application of RD there are 57 school- specifi c cutoffs, rather than a single cutoff, and the cutoff falls at different values of the dropout- risk score in
different schools.
16
We return to this point in our discussion of heterogeneity below. This raises the question of how best to pool the results from what are, in effect, 57
separate applications of RD. One way to address this challenge would be to normal- ize the dropout- risk score by subtracting the value of the score at the school- specifi c
cutoff. However, this gives a score of exactly zero to at least one applicant in each school more if there are tied scores, and mechanically results in a piling of mass
at the cutoff. To avoid this, we normalize the cutoff in each school as the value of the score that is the midpoint between the scores of the last scholarship recipient and
the fi rst nonrecipient in that school.
17
Also, to ensure that identifi cation is based on comparisons of scholarship recipients and nonrecipients within schools, not across
schools, we include school fi xed effects. The dropout- risk score perfectly predicts whether an applicant was offered a schol-
arship, with the exception of a very small number of changes that resulted from the public complaint mechanism. This is therefore a case of sharp as opposed to fuzzy
RD. Also, because we focus on the impact of being offered a scholarship, rather than that of actually taking up a scholarship, these are Intent- to- Treat ITT estimates of
program impact. For convenience, in the paper we interchangeably refer to children with values of the dropout- risk score above the cutoff, all of whom were offered
scholarships, as children who were eligible for scholarships, recipients, or treated children.
As in other applications of RD, it is important to ensure that results are not driven by a particular parametrization of the control function. Following Lee and Lemieux
2010, we present results based on two ways of implementing RD. The fi rst approach uses data on all children in the household survey. In this approach, the estimation
choice revolves around the selection of the polynomial in f C
s
. We follow Lee and Lemieux 2010 and use both the Akaike information criterion AIC and bin re-
gressions to select the optimal order of polynomial. As it turns out, both approaches produce very similar results, and in virtually every case, no matter which dependent
variable, the optimal order of the polynomial is one meaning that the control func- tion is linear in the score, with slopes that can be different on each side of the cutoff .
Nevertheless, to ensure that our results do not depend on the choice of polynomial, we
16. Because the number of scholarship recipients is fi xed in “small” and “large” schools, as discussed above, the value of the dropout- risk score at the cutoff will on average be higher in schools that received more ap-
plications and in schools which serve poorer children. 17. We thank an anonymous referee for this suggestion. Pop- Eleches and Urquiola 2013 analyze the effects
of school quality in Romania in an RD setting with multiple cutoffs. They normalize the score by subtract- ing the value of the score at the school- specifi c cutoff but exclude the observation whose score is exactly
equal to zero to avoid the mechanical piling of mass at the cutoff. Our results are robust to this approach to normalizing the score.
also present results for polynomial order two with quadratic terms interacted with the indicator variable for scholarship recipients.
18
The second approach, based on local linear regressions LLRs, restricts the sample to those who are relatively “close to” the cutoff. In this approach, the estimation choice
revolves around the selection of the bandwidth around the cutoff. Here too we follow Lee and Lemieux 2010 and present results based on a rule- of- thumb ROT selection
of bandwidth, as well as a cross- validation or “leave- one- out” procedure. In the ROT approach we use a rectangular kernel and follow the estimation steps detailed in Imbens
and Kalyanaraman 2012. Based on these calculations, we estimate that the optimal bandwidth is between four and eight, depending on the outcome in question. To avoid
presenting different bandwidths in different tables, we report results from LLRs with a bandwidth of six for all outcomes, but generally also report results with a bandwidth of
three half the optimal and 12 twice the optimal. As we show below, our results are not sensitive to the selection of polynomial or the selection of bandwidth in the LLRs.
There are two main threats to our identifi cation strategy: attrition and manipulation of the dropout- risk score. Attrition is a potential source of concern because the fi rm that col-
lected the household survey was given a list of households to visit on the basis of the roster of children who had completed the application form in 2005, but not all of these
households could be located in 2010 for example, if the information regarding the name or place of residence on the application form was inaccurate or if a household had moved
to another province. In practice, almost fi ve years after students fi lled out their scholar- ship application forms as described above, 14.1 percent of children on the list given to
the fi rm could not be located. We note that these values are comparable to those for other surveys in developing countries—for example, Behrman, Parker, and Todd 2011 report
an attrition rate of 14 percent for their analysis of medium- term effects of PROGRESA transfers.
Figure 1 graphs the proportion attrited as a function of the normalized dropout- risk score.
19
Attrition is higher among scholarship recipients, who are poorer, but there is no evidence of a discrete jump in attrition at the cutoff that determines eligibility for
scholarships. We conclude that it is unlikely that attrition is a source of bias for our estimates of scholarship effects.
20
An identifying assumption for RD is that, conditional on a fl exible parametrization of the control function, there is no discrete jump at the cutoff in characteristics includ-
18. We also calculated results for polynomial order zero, which is equivalent to a simple comparison of mean outcomes for children who were offered scholarships and those who were turned down. We do not
report these results as they make no adjustment for differences in socioeconomic status between scholarship recipients and nonrecipients.
19. In this, as in other fi gures, the size of each circle is proportional to the number of observations at each value of the score. Lower values of the normalized dropout- risk score, on the lefthand side of the graph,
correspond to children with characteristics that make them less likely to drop out of school in the absence of scholarships. The regression line is based on polynomial order one, with school fi xed effects, and with the
cutoff centered at the midpoint between the score of the last scholarship recipient and the fi rst nonrecipient. 20. As a further robustness check, we ran separate regressions in which the indicator variable for attrited
children was regressed on an indicator for a given characteristic for example, age of the child and the relevant interaction with treatment for example, age interacted with treatment. We repeated this for each of
the 26 variables on the application form. In the polynomial order 1 regressions, the interaction term is only signifi cant at the 5 percent level or higher in one case corresponding to the interaction between treatment
and the availability of drinking water within the house. We take this as further confi rmation that attrition is unlikely to be a source of concern for our estimates.
ing unobservables that could be correlated with the regression error term. Although one cannot test this assumption directly, we provide two standard RD identifi cation
checks. See Imbens and Lemieux 2008; Lee and Lemieux 2010. The fi rst checks for unusual heaping of mass at the cutoff. Figure 2 graphs the proportion of observations
in 60 bins of the normalized dropout- risk score. The fi gure shows no unusual heaping of mass above or below the cutoff for scholarship eligibility.
21
21. McCrary 2008 proposes a formal test of the log difference in the heights of the bins corresponding to observations just to the right and to the left of the cutoff. On the basis of this test, we fail to reject the null of
equal mass on both sides of the cutoff. The regression coeffi cient is 0.118, with a standard error of 0.075.
.1 .2
.3
Probability of Attrition
-25 -15
-5 5
15 25
Normalized Dropout-risk Score
Figure 1 Analysis of Attrition
Notes: Dashed lines are from the linear, interacted regression of attrition dummy on recipient status, recipient status interacted with normalized dropout- risk score, normalized- dropout risk score, including
secondary school fi xed effects. Circles proportional to number of observations.
A second standard check on RD involves regressions of predetermined character- istics on the indicator value for treatment in our case, children who were offered
scholarships and the control function. In general, if the RD design is valid, we would expect the coeffi cient on the treatment indicator in these “placebo” regressions to be
small and not signifi cant.
Table 1 presents results of these regressions for a number of variables from the household survey that are arguably fi xed—including the gender and age of applicants,
education levels of all adults in the household, and various measures of household composition. In the polynomial order one regressions and the LLRs with a bandwidth
of six, none of the coeffi cients are signifi cant at the 5 percent level or higher. We conclude that the RD approach we take in this paper is valid, and is likely to provide
causal estimates of the medium- term effects of scholarships.
IV. Results
