Teacher Pay Reform and Productivity
Panel Data Evidence from Adoptions of Q- Comp in Minnesota
Aaron J. Sojourner Elton Mykerezi
Kristine L. West
Sojourner, Mykerezi, and West
abstract
This paper studies the impacts of teacher pay- for- performance P4P reforms adopted with complementary human resource management HRM
practices on student achievement and workforce fl ows. Since 2005, dozens of Minnesota school districts in cooperation with teachers’ unions implemented
P4P as part of the state’s Quality Compensation program. Exploiting district variation in participation status and timing, we fi nd evidence that
P4P- centered HRM reform raises students’ achievement by 0.03 standard deviations. Falsifi cation tests suggest that gains are causal. They appear
to be driven especially by productivity increases among less- experienced teachers.
I. Introduction
The potential to improve U.S. education through human resource man- agement HRM reforms centered on pay- for- performance P4P remains an open,
active question for economists and policymakers. After decades of paying teachers
Aaron J. Sojourner asojournumn.edu is an assistant professor at the Carlson School of Management at the University of Minnesota. Elton Mykerezi myker001umn.edu is an associate professor of Applied
Economics at the University of Minnesota. Kristine L. West klweststkate.edu is an assistant professor of Economics at St. Catherine University. Thanks to Avner Ben- Ner, Jen Brown, John Budd, Brian Cadena,
Caroline Hoxby, Paul Glewwe, Karthik Muralidharan, Michael Lovenheim, Morris Kleiner, Collen Manachester, Jason Shaw, Chris Taber, and Joel Waldfogel; to participants at the NBER Spring 2011 Eco-
nomics of Education meeting for comments; and to Qihui Chen, Paul Kristapovich, Qianyun Xie, Yingchun Wang, and Robert Vellella for able research assistance. Thanks to George Henley and Kristie Anderson of
the Minnesota Department of Education and the Kingsbury Center for help with restricted- data acquisition and to the Human Capital Research Collaborative and the University of Minnesota Center for Urban and
Regional Affairs for funding support. All errors are the authors’. Data on district Q- Comp participation can be obtained from the authors May 2015 through April 2018. Student achievement data and workforce
fl ows must be requested from the Minnesota Department of Education. [Submitted June 2013; accepted September 2013]
ISSN 0022-166X E- ISSN 1548-8004 © 2014 by the Board of Regents of the University of Wisconsin System
T H E J O U R N A L O F H U M A N R E S O U R C E S • 49 • 4
almost solely on their education and experience, hundreds of U.S. school districts have begun measuring teacher performance in various ways and incorporating these
measures into pay determination. State governments including Colorado, Florida, Minnesota, South Carolina, and Texas now encourage or require local school districts
to shift their teacher management systems to include performance evaluation and pay. The federal government also encourages districts to shift toward performance- based
HRM systems through its 1.6 billion Teacher Incentive Fund and 4.4 billion Race to the Top program. These reform efforts are driven by theories that P4P- centered HRM
reform will raise teacher productivity and student learning.
Despite strong evidence of P4P effects in production environments where output is relatively easy to measure Lazear 2000; Bandiera, Barankay, and Rasul 2007,
evidence of P4P and broader HRM reforms’ impact in more complex environments is limited. According to Bloom and Van Reenen 2011, in organizations in general,
“There is certainly a robust positive cross sectional association between bundles of ‘modern’ HRM practices and productivity, but with some exceptions for example,
Ichniowski, Shaw, and Prennushi 1997 these are not robust in the time- series dimen- sion.” In education, evidence of the effects of P4P in U.S. schools is mixed Podgursky
and Springer 2007; Prentice, Burgess, and Propper 2007; Neal 2011. P4P field ex- periments generated null or even negative effects Springer et al. 2010; Fryer 2011.
1
Aside from P4P, evidence is emerging that another kind of HRM reform, performance measurement and feedback, can generate positive effects through improvement of in-
cumbent teachers’ practice Taylor and Tyler 2012 and increasing low performers’ separation rates Rockoff et al. 2012.
This paper reports new evidence on how teacher HRM reforms centering on P4P contracts affect student achievement and workforce flows using panel data from Min-
nesota. A shock to local school districts’ HRM practices was induced by the state’s introduction of its Quality Compensation program Q- Comp in 2005. The state began
offering local districts additional funding in return for districts’ use of P4P contracts along with complementary HRM practices. Different districts applied for and joined
Q- Comp each year. Dozens of districts have since implemented reform and hundreds of thousands of student- years have been taught in participating districts. As one of
the nation’s largest and longest- standing programs encouraging P4P- centered HRM reform, Q- Comp attracts significant policy and political attention, yet little is known
about the effects of the reforms it spurred. This is the first study to use the Q- Comp program to study effects of reform on student achievement.
2
This study estimates the effects of P4P- centered HRM reform on achievement for students in Grades 3 to 8 and the mechanisms by which the effects operate by inte-
grating data from multiple sources. We use student- level panels from two different standardized achievement tests one state- mandated and the other optional but widely
used, the population of teachers linked to their district each year, Q- Comp program
1. Strong evidence of positive P4P effects has emerged in other countries Muralidharan and Sundararaman 2011; Lavy 2002, 2009; Glewwe, Ilias, and Kremer 2010.
2. A legislative auditor’s report Nobels 2009 and state commissioned external reports Hezel Associates 2009; Wahlstrom, Sheldon and Peterson 2006 provide evidence about Q- comp’s implementation but little
about resulting student achievement. Neither dealt with selection or covariates. Nadler and Wiswall 2011 studies Q- Comp participation but not impacts. Schwartz 2012 provides qualitative description based on
interviews with numerous stakeholders.
data coded from archives of offi cial documents, the U.S. Schools and Staffi ng Survey, as well as data on district characteristics and finance from the Minnesota Department
of Education MDE. This enables the study to make five main contributions. First, this statewide field study based on one of the nation’s largest teacher pay
programs is uniquely informative about longer- run, general- equilibrium effects. Pre- vious efforts in this direction have relied on cross- national variation Hanushek and
Woessmann 2011. Theory suggests that P4P will work by some mixture of 1 in- creasing incumbent employees’ productivity and 2 attracting more able employees
to the organization Lazear 2000. In order to operate fully, both mechanisms require belief that the reform is here to stay. Incumbent teachers may require a few years of
trial and error to increase their own productivity in response to P4P incentives. They will make these investments only if they believe the reform will endure. Regarding
sorting, people will not alter their choice of employer based on a pay system they expect to disappear. Only a permanent policy change gives teachers, administrators,
and families incentives and time to adjust to reform.
Second, despite the nonexperimental context, the study has several properties that make identification credible. Student achievement is measured in individual panels
covering the academic years starting 2003 to 2009, and different districts applied for and adopted Q- Comp in different years. A generalized difference- in- difference
framework identifies the effect of reform on districts’ productivity net of time effects and fixed differences between individual students. Further, we conduct a series of
falsification tests to rule out various alternative explanations. One placebo test uses multiple years of pre- adoption data to estimate Q- Comp “effects” prior to adoption,
which tests for the presence of differential trends in unobservables among adopting districts. A second placebo test estimates failed- application “effects” to assess whether
districts’ desire to reform is suffi cient to generate effects, absent actual reform. Q- Comp’s teacher- pay reforms produced an average 3 percent of a standard deviation
increase in reading achievement, with some evidence of a similar effect on math achievement in the full sample. Evidence from the appliers- only sample is weaker.
Third, we assess whether achievement effects reflect real skill gains or simply teaching- to- the- test by assessing Q- Comp’s impact using two distinct standardized
tests. Gaming is a central concern in the P4P literature generally Holmstrom and Milgrom 1991 and in the educational P4P literature particularly. According to Neal
2011, “Clean evidence on the generalizability of assessment gains is rare, and the existing literature does not speak with one voice.” The Minnesota Comprehensive
Achievement test MCA is state mandated and administered in all districts. Some districts also voluntarily contract with the Northwest Evaluation Association NWEA
to use its Measures of Academic Progress achievement test.
3
We use all existing data on Minnesota students in Grades 3 to 8 from the MCA and NWEA in both reading and
math. In each subject, we assess the estimates’ robustness across the tests. Further, we check whether students make larger gains on the higher stakes test than on the other
test. We do this in a unique way. Commonly, higher stakes are attached to one test for all students and a second test is lower stakes for everyone, meaning that stakes are
confounded with test. In this study, many students also take two tests but, in contrast,
3. Though the panels cover many of the same students in the same years, we cannot link a student across tests or to a particular teacher.
which test is higher stakes with respect to school- wide performance bonuses differs by district and sometimes even by school grade within district. Test and stakes are not
confounded. Fourth, we use data on the universe of Minnesota teachers tied to their employ-
ing district each year to examine if changes to districts’ teacher HRM policies affect teacher movements and the extent to which this accounts for achievement gains. Al-
though we cannot link particular teachers to particular students, we develop measures of district- year teacher workforce flows through various channels novice teachers
entering interdistrict transfers in and out, retention, and retirements out and school- year teacher experience profiles share with
≤ 5, 6–15, and ≥ 16 years experience. We study how these relate to changes in student achievement. There is no evidence of
significant effects of HRM reform on teacher workforce flows or experience profiles, and observed changes in flows and profiles do not explain coincident changes in stu-
dent achievement. Similarly, we test and reject the possibilities that changes in dis- tricts’ student populations or total expenditures drive achievement gains. In contrast,
we find evidence that the effect of reform interacts significantly with school- level measures of teacher experience; the effect is strongest among schools with more in-
experienced teachers.
Finally, Q- Comp provides an opportunity to examine the effects of a “grantor- grantee” structure for program design that mirrors recent U.S. Department of Educa-
tion efforts such as Race to the Top and the Teacher Incentive Fund. In these programs, the funder sets out guidelines and asks local entities to propose tailored plans within
them. The grantor delegates some design decisions in order to harness local knowl- edge about what will work and what is politically feasible. However, this comes with
risk that local grantees do not deliver improvements. Q- Comp is an opportunity to examine if decentralized design with centralized approval generated gains or produced
mostly rent- seeking behavior.
These features of the study enable contributions to the education, personnel, and la- bor economics literatures. Having long- term outcomes coupled with a real policy shift
permits study of how impacts unfold over a multiyear horizon, which is unusual and valuable. Further, having P4P- centered HRM reform sweeping across a large fraction
of an industry while observing output and worker flows for all firms in the industry, gives an unprecedented view into the mechanisms of change. This paper reports some
of the strongest evidence available that teacher HRM reform can increase teacher productivity and student learning in the United States. Though the magnitudes of the
achievement impacts are not large, a rough estimate suggests the social benefits of the gains exceed the costs of producing them.
II. Background and Process
