Does Training Matter Evidence from Performance Management Reforms (pages 411–420)

Management Reforms

Abstract: Training is much discussed but rarely studied in public management. Using multiple waves of survey data,

Alexander Kroll is assistant profes-

the authors examine the eff ects of training on the implementation of performance management reforms in the U.S. sor of public administration at Florida

International University. His research inter-

federal government, asking whether those exposed to training are more likely to use performance data and strategic

ests are in studying organizational effective-

goals when making decisions. Training is positively associated with reform implementation, but there is little evidence

ness, employee behavior, and particularly

that this association can be explained by the development of specifi c capacities to overcome performance management the roles of performance information,

strategy, leadership, and motivation.

challenges. Th e fi ndings off er two implications for the practice and study of training. Th e authors propose that training E-mail: akroll@fi u.edu is likely to succeed if it is designed and funded to close specifi c capacity gaps needed for successful reform implementa-

tion. However, it is also necessary to better understand alternative causal mechanisms by which training facilitates Donald P. Moynihan is professor in

the La Follette School of Public Affairs,

reform implementation, such as explaining and justifying reforms.

University of Wisconsin–Madison. He is fellow of the National Academy of Public

Administration, author of Practitioner Points The Dynamics of

Performance Management: Constructing

• Training can facilitate the implementation of new policies by providing information about the policies,

Information and Reform (Georgetown

justifying why they are needed, and giving employees the capacity to put the new policies in place.

University Press, 2008), and winner of the

• Using training to build specifi c employee capacities is more complex, resource intensive, and rare than using ASPA/NASPAA Distinguished Research

Award.

training to provide information and justify reforms.

E-mail: dmoynihan@lafollette.wisc.edu

• Eff ective training in government requires not just more resources but also better understanding of the specifi c capacities needed and how to create them—research can help by identifying the most signifi cant capacity gaps in policy implementation.

T capacity to do so. Training is a default solution to all sensitivity, sexual harassment, or ethics.

raining occupies a paradoxical place in man-

accusations of inattentiveness to undesirable employee

agement theory and practice. We ask it to do

behavior or to punish employees under suspicion of

great things even as we are cynical about its

wrongdoing. Th is kind includes training on racial

manner of managerial challenges ranging from ethical problems and rule violations to employees acquiring

Th ere is a middle-ground view, which is that train-

skills. It is credited with communicating organiza-

ing has high potential value but is off ered too little or

tional norms and new ideas and

in a manner that rarely helps

reinforcing organizational cul-

Training is a default solution

employees. U.S. federal employ-

ture. Th is optimistic account of

to all manner of managerial

ees ranked attention to training

training assumes that individual

challenges ranging from ethical

lower than almost any other

preferences and capacities are

aspect of their work experience.

mutable and subject to external

problems and rule violations to

One in two employees believed

infl uence.

employees acquiring skills.

that their training needs had been assessed or were satisfi ed

Th e pessimistic counternarra-

with the training they received

tive is that training does little. Th is view is held

(OPM 2014, 29, 32). As budget cuts squeeze spend-

by anyone who has counted the minutes during a

ing, employee frustration grows. Among those who

mandated training session, worried about taking time

off er training for federal employees whom we inter-

away from actual work and yearning to check e-mail.

viewed for this article, there was a uniform belief that

Pessimists say that training is just a symbolic or well-

resources available for training have declined as federal

intended eff ort to deal with problems too fundamen-

spending has come under pressure. 1 One trainer put

Public Administration Review ,

tal to be solved by a daylong session with slides and Vol. 75, Iss. 3, pp. 411–420. © 2015 by it like this: “From a department perspective, there are easel boards. Managers may use training to ward off The American Society for Public Administration. real needs, but training is a want, not a need.”

DOI: 10.1111/puar.12331.

Does Training Matter? Evidence from Performance Management Reforms 411

In this article, we off er evidence on the value of training by exam- Th e policy makers who passed the original and updated GPRA ining whether the exposure to training is associated with reform

identifi ed a role for training. In the GPRA (Public Law 103-62), the implementation, specifi cally performance management reforms in

ce of Personnel Management (OPM) was directed to “develop a the U.S. federal government. Th e variety of types and purposes of

Offi

strategic planning and performance measurement training compo- training makes it diffi cult to discern how it matters. Th e benefi t of

nent for its management training program and otherwise provide focusing on a single purpose of training (reform implementation)

managers with an orientation on the development and use of strate- in the area of performance management is that it largely controls

gic planning and program performance measurement” (sec. 9). Th e for diff erences in types of training. To better establish the valid-

GPRA Modernization Act (Public Law 111-352) called on the head ity of our analysis, we examine the eff ects of training across three

of the OPM and the Performance Improvement Council (a govern- time periods: 2000, 2007, and 2013. Each time period correlates

ment-wide panel of performance improvement offi cers) to identify with a diff erent reform: the Government Performance and Results

“the key skills and competencies needed by Federal Government Act (GPRA) passed in 1993 but not fully implemented until the

personnel for developing goals, evaluating programs, and analyzing late 1990s, the Program Assessment Rating Tool (PART) that was

and using performance information for the purpose of improving in place from 2002 to 2008, and the GPRA Modernization Act

Government effi ciency and eff ectiveness” (sec. 12a). Further, the passed in 2010.

OPM was asked to work with agencies to incorporate these skills into training. Table 1 summarizes the performance management

We use employee utilization of performance data and strategic goals competencies identifi ed by the OPM (2012). as measures of reform implementation. Th is approach is consist- ent with explicit expectations in the language of these reforms that

Th e Modernization Act recognized that while the OPM might emphasized performance information use and a focus on agency-

identify key skills, much of the actual training would come through wide strategic objectives as central goals (GAO 2013; Moynihan

agencies and other parties. Agencies were directed to identify “the and Lavertu 2012). We examine whether training fosters the use

operation processes, training, skills and technology, and the human, of performance data and strategic goals to make decisions by

capital, information, and other resources and strategies required” giving employees greater skills to overcome performance manage-

(sec. 3) to achieve performance goals. According to Government ment problems, such as diffi culties in measuring outcomes. Our

ce (GAO) surveys, in 2000, 62 percent of federal quantitative analysis fi nds support for a direct positive eff ect of

Accountability Offi

employees said they had had some training on performance manage- training, although we do not fi nd that training is associated with

ment in the last three years; the share was 70 percent in 2007 and the specifi c capacity gaps that we identify. Qualitative interviews

66 percent in 2013. But training occurs largely through a patchwork suggest that performance management training spent a good deal of of diff erent providers: the OPM, agencies themselves, and nonprofi t time explaining and justifying new reforms. Th is fi nding highlights

or private providers. Th e relative infl uence of each group is diffi cult alternative causal mechanisms for researchers to more systematically

to determine, even as nonprofi t and private providers have gained investigate. In short, while we fi nd that training matters, under-

more prominent roles.

standing how it matters remains a pressing practical and scholarly question.

Research on Training and Reform Implementation

One way to conceptualize reform implementation is to treat it as a Th is article unfolds as follows: First, we explain the context of

form of organizational change, thereby allowing us to draw insights our study—performance management reforms in the U.S. federal

from organizational development scholarship (e.g., see Gallos 2006). government and the role of training in these reforms. Second, we

Th eories of organizational change suggest that acceptance of change propose that training has a positive eff ect on reform implementa-

requires a variety of types of information and arguments, hinting at, tion. Th ird, we examine a particular mechanism by which training

although rarely explicitly identifying, the role that training might may matter: the development of capacity to overcome implemen-

play. Beckhard and Harris’s (1977) classic model identifi es common tation diffi culties. Next, we explain the data (government surveys

challenges to change. Th ey suggest that individuals must experi- and additional interviews) and methods we use. Finally, we explain

ence dissatisfaction with the status quo, observe the desirability and our fi ndings, discuss them, and point out avenues for further

practical applicability of an alternative, and have specifi c doubts research before identifying implications for the practice of public

administration.

Table 1 Performance Management Competencies Identifi ed by OPM Accountability

Attention to detail

Performance Management and Training in the U.S.

Customer service

Federal Government

Infl uencing/negotiating

In the last 20 years, the U.S. federal government has experienced Information management three waves of performance management reforms. Th Oral communication e GPRA

Organizational awareness

required agencies to measure performance and undertake strategic

Organizational performance analysis

planning. PART was an in-depth assessment of each federal pro-

Partnering

gram, ranking it on a scale from ineff ective to eff ective (Moynihan Performance measurement 2013a). Th Planning and evaluating e GPRA Modernization Act of 2010 updated the GPRA

Problem solving

by requiring additional reporting and review processes and institu-

Reasoning

tionalizing new performance management staff positions and leader- Technical competence ship expectations (Moynihan 2013b).

Written communications

412 Public Administration Review • May | June 2015 412 Public Administration Review • May | June 2015

“bigger picture” fosters coordination and successful collective action can help convey problems with the status quo and outline benefi ts

(Marks et al. 2002).

of an alternative. Details on the content of a reform and competen- cies to overcome specifi c challenges can illustrate the practicality of

A second explanation of the training eff ect is that it may alter the the reform and overcome specifi c concerns about implementation.

trajectory of reform implementation to the extent that it presents a reform as an appropriate norm to follow, thereby raising employee

Another infl uential model of organizational change identifi es three support for and investment in the reform. Th is could be based on a distinct strategies: empirical-rational, power-coercive, and norma-

legal requirement (“we are doing this because you are legally obliged tive-reeducative (Chin and Benne 1969; Quinn and Sonenshein

to do so”), but also on a normative imperative (“this change is the 2008). Th e empirical-rational strategy (“telling”) is based on the

right thing to do” or “all of our peers are doing it”). Institutional assumption that organizational members will adopt change if

isomorphism theory, for example, argues that the potential for managers can justify the change and demonstrate its benefi ts. Th e new organizational approaches to spread and embed themselves

power-coercive strategy (“forcing”) is about managers exercising depends greatly on training in educational or other professional set- coercion and using sanctions to ensure compliance with organiza-

tings. Such settings communicate these ideas as appropriate norms tional changes. Th e normative-reeducative (“reeducating”) strategy

relevant to defi ning the identity of the group (DiMaggio and Powell primarily focuses on the social dimension of organizations. To adopt 1983, 152). According to Feldman, “training has become the set- change, social norms, values, and habits need to be altered, which

ting where perceptions and expectations about the organization are requires mutual persuasion within collaborative relationships. Of

formed, where norms about social behavior are developed, where these three change strategies, training could be paired with at least

corporate values and ideology are communicated” (1989, 399). two. It could be used for “telling” and “reeducating,” although for the latter, training would have to be about challenging assumptions

Some studies of performance management emphasize that training rather than just learning about new practices.

can be important in changing normative beliefs about the value and appropriateness of the reform. Training can “reduce uncertainty,

Th e following two sections theorize about the eff ect of training on fear, and cynicism” (Yang and Hsieh 2006, 863) and help employ- reform implementation in greater detail. Th e fi rst section hypothe-

ees “understand, accept, and feel comfortable with the innovation” sizes a positive training eff ect and elaborates on diff erent explanatory instead of “feeling pressured or overwhelmed” (Cavalluzzo and mechanisms. We derive two mechanisms directly from the organi-

Ittner 2004, 249). Darnall and Kim (2012) suggest that technical zational change literature: conveying information about a change

changes designed to improve environmental performance become needed and providing normative justifi cation for behavioral change. less likely to succeed in the absence of training that generates shared We identify a third mechanism (the creation of capacity) that is less

norms and beliefs.

well-considered in organizational change research but may facilitate reform implementation by helping overcome specifi c objections

A third explanation of a positive training eff ect, which we discuss (Beckhard and Harris 1977). Indeed, Fernandez and Rainey (2006)

in greater detail in the next section, is that it creates capacity to identify training as one means that managers can use to overcome

implement reforms. Th at is, the adoption of performance manage- resistance to change. Th e second section theorizes more about

ment reforms requires competencies with regard to performance capacity eff ects and develops hypotheses about specifi c performance- measurement, analysis, and reporting, and training can be crucial management-related capacity.

for developing such capacity.

Hypothesis 1: Performance management training improves One explanation of why training can foster reform implementation

The Effect of Training

the implementation of performance management systems. is that it has an information eff ect. Training conveys basic informa- tion about the reform and its requirements, and simply knowing

Capacity Effects of Training

more about a reform may make employees more likely to imple- Reforms require employees to face new tasks or ways of thinking. ment it. Th is type of declarative knowledge (information about

Reform implementation is more likely to occur when training gives what) is usually seen as inferior to procedural or tacit knowledge

employees the skills to meet the new demands, which we refer to as (information about how or which, when, and why) (Aguinis and

a capacity eff ect. We borrow our defi nition of capacity from Huber Kraiger 2009). However, the provision of

and McCarty, who understand capacity information on important reform concepts,

as “skills and expertise” and “the ability to changes to central processes and routines,

accomplish intended actions” (2004, 481; for and expectations and rewards seems to

Th e dissemination of

overlapping defi nitions, see Ingraham, Joyce pertain to fundamental lower-order condi-

information through training

and Donahue 2003, 15; Ting 2011, 245). tions for behavioral change. Th e dissemina-

may foster shared mental

models, meaning that

tion of information through training may Th e importance of capacity to governance foster shared mental models, meaning that

employees understand their own elevates the value in understanding whether

employees understand their own and their

training can aff ect capacity. In models of gov- colleagues’ role(s), behavior(s), and linkages

and their colleagues’ role(s),

ernmental and third-party provider capacity, as parts of a larger system. Although this has

behavior(s), and linkages as

training is explicitly assumed to off er a means little to do with the development of specifi c

parts of a larger system.

of raising capacity (Brown 2012; Ingraham,

Does Training Matter? Evidence from Performance Management Reforms 413

Joyce, and Donahue 2003). Prominent scholars argue that capac- the development of more accurate logic models and meaningful sur- ity deserves attention equivalent to other concepts that animate

rogates for diffi cult-to-measure outcomes (Hatry 2006). formal theories of governance, such as information asymmetry or policy design (Huber and McCarty 2004; Ting 2011). Th e general

For learning processes and discretion, the connections are less treatment of training in public administration makes the assump-

direct but relate to a wider array of skills listed in table 1. Training tion that training can generate specifi c capacities. When we speak

could amplify the benefi ts of learning routines if regular reviews of of specifi c capacity, we refer to skills and expertise that are related

performance trends are informed by performance analysis, problem to a specifi c task (in our case, performance management), consist-

solving, and oral communication skills (Behn 2014). Training can ent with Ting’s (2011, 248) distinction between generalist and

also better leverage discretion by conferring capacity to diagnose and specialist capacity. Examples of specialist capacity are the use of

redesign processes or by facilitating the appropriate use of targets Federal Aviation Administration training to inculcate high-reliability and incentives to motivate employees. characteristics that reduce airplane crashes (O’Neil and Kriz 2013) or facilitating greater environmental performance as part of an envi- If training generates a capacity eff ect, we expect it to moderate the ronmental management system (Darnall and Kim 2012).

relationships between the variables we identifi ed earlier and reform implementation. Th at is, those who are trained will be better able to

A long-standing tradition of research on training in psychology use performance information and strategic goals in the face of meas- focuses on the eff ects on job performance or skills that serve as

urement diffi culties. We also hypothesize that more training will ena- antecedents of job performance (Aguinis and Kraiger 2009). Th is

ble managers to better leverage the discretion granted to them, benefi t research points to adaptive expertise as an important outcome

more from learning routines and performance feedback, and better variable for training, studying whether and when employees show

respond to contexts shaped by high accountability requirements. the capacity to match diff erent skills to the appropriate task rather than simply learning a certain competency (Kozlowski et al. 2001).

Hypothesis 2: Performance management training will weaken For the case of performance management, this research implies that

the negative eff ect of measurement problems for the imple- specifi c capacity is more than a checklist of additive competencies

mentation of performance management systems. and that it is important to develop the ability to adapt measurement and evaluation skills to diff erent situations and contexts.

Hypothesis 3: Performance management training will strengthen the positive eff ect of discretion for the implementa-

Management studies research also assumes that training can be used to tion of performance management systems. develop needed competencies to enable individuals to succeed in their specifi c organizational contexts (Burke and Hutchins 2007). Empirical

Hypothesis 4: Performance management training will research on the impact of training on skill development shows that

strengthen the positive eff ect of a learning routine for the this eff ect is stronger for technical training compared to more complex

implementation of performance management systems. managerial training programs, and more signifi cant for skills that can

be segmented into step-by-step routines compared to, for example, soft Hypothesis 5: Performance management training will skills (Hunt and Baruch 2003; Morrow, Jarrett, and Rupinski 1997).

strengthen the positive eff ect of accountability for the imple- mentation of performance management systems.

In the context of performance management, Yang and Hsieh (2006)

suggest that technical training facilitated the adoption and eff ective- Data and Methods

ness of performance systems in Taiwan because of the traditional

Sample and Measures

lack of capacity to analyze performance data. Th e need for such Th is article uses data from three surveys conducted by the GAO in analytical capacity has not lessened. For example, Behn’s (2014)

2000, 2007, and 2013. Th e surveys were addressed to a random, detailed account of performance reviews in the United States

nationwide sample of mid- and upper-level federal managers in describes the central importance of analytical capacity.

the agencies covered by the Chief Financial Offi cers Act of 1990, stratifi ed by agency and management level. Th e GAO used similar

Cumulatively, then, there is a good deal of support for the idea that sampling frames across years and drew a new random sample of training matters by improving the capacity of individual employees.

managers for each of the waves. 2 Th e overall response rate for each To assess whether training is associated with conveying capacity

of the surveys was about 70 percent (for more information, see skills for our study, we identify four well-established performance

GAO 2008, 2013). Because we do not hypothesize that the eff ect management implementation challenges that demand capacity: (1)

of training diff ers across reform initiatives, we do not run separate how to measure performance, (2) how to use discretion, (3) how to

models for the years 2000, 2007, and 2013. Instead, we pool the learn from performance data, and (4) how to use data for account-

data to increase the external validity of the study by examining ability purposes. Some of the capacity items we identify clearly map

whether the eff ects of training hold across diff erent points in time onto the performance management competencies identifi ed by the

and diff erent reform contexts. Other advantages of a pooled model OPM in table 1. Most obviously, accountability skills relate to our

are that it makes effi cient use of the data structure, as all three sur- measure of accountability context, and performance measurement

veys employed identical questions, and it estimates reliable standard skills relate to performance measurement challenges. In a high-

errors because all t-values are based on a constant sample size. accountability context, training could help create more eff ective reports and enable managers to better explain the limitations of data While our primary goal with this article is to empirically test the to stakeholders. For measurement problems, training could facilitate hypotheses laid out earlier, we employed some qualitative methods

414 Public Administration Review • May | June 2015 414 Public Administration Review • May | June 2015

performance data when making decisions.

who oversaw training in the GPRA, PART, and current eras, includ- ing offi cials who worked in agencies, for the OPM, for the Offi ce Hypotheses 2–5 are modeled as interaction eff ects because our of Management and Budget (OMB), and for private and nonprofi t

assumption is that the relationships between the independent trainers. Because the population of such individuals is not easily

variables measurement problems, discretion, learning routine, and identifi able, they were selected using a snowball sampling approach. accountability and the dependent variables performance informa- Th ey were asked questions on the nature, extensiveness, and content tion use and goal use are contingent on diff erent levels of employee of performance management training at diff erent time periods. We

training. To test each interaction eff ect between all four variables also examined actual training materials such as presentations and

and the training index, we regress each dependent variable on all of instructional manuals.

the 11 variables listed in table 3 and add each interaction term one at a time (see columns 3–10). Th is way, we can examine whether

In our quantitative analysis, we measure reform implementation adding the interactions improves the model fi t signifi cantly or not. using factor scores of the variables “use of strategic goals” and “use of Th e interaction terms were generated as the multiplied scores of performance information” when setting program priorities; allocat-

the mean-centered individual variables, and the interpretation of ing resources; adopting new program approaches or changing work

their unstandardized regression coeffi cients is as follows: for every processes; and developing or refi ning program performance meas-

one-unit increase in the moderator variable (training), the slope ures. Both proxies measure two important elements of performance

relating the predictor to the outcome variable increases or decreases management reform implementation: defi ning strategic priorities

by the reported coeffi cient while holding all other variables constant and collecting data on results and achievements as well as using both (Whisman and McClelland 2005). in managerial decision making (Moynihan 2013a, 2013b). Th e vari- ables are moderately positively correlated (r = 0.51), but there still

Because our dependent variables are underlying factors rather than is a good portion of unshared variation worth studying. Indicators

ordinal scales, we employ ordinary least squares regression analysis of the convergent (see Cronbach’s alpha) and discriminant validity

to estimate the direct and interactive eff ects of training on reform (see factor loadings) of both dependent variables as well as the exact

implementation. To account for the possibility that responses measures of all variables and their scale reliability coeffi cients are

from the same agency might be related to one another, we cluster reported in the appendix.

standard errors at the agency level. Th e usual shortcomings of cross- sectional survey research should be kept in mind when general-

Training was measured as an additive index of six planning- and izing from our study. One is that our fi ndings represent statistical measurement-related activities, ranging from zero (no training) to

associations rather than causal eff ects. Another is that the possibility six. We also included nine control variables, most of which were

that those who had higher interest in and support for performance identifi ed as relevant in a recent review of empirical studies on

systems were more likely to select into training cannot be fully ruled the implementation of performance management systems (Kroll

out because our data do not come from a repeatedly surveyed panel

forthcoming). Th e measures of these variables are widely established, of employees. Our interviews suggest, however, that there is not a and other studies that used survey data collected by the GAO

clear and direct selection pattern that skews the results in a certain utilized these measures (see, e.g., Cavalluzzo and Ittner 2004; Dull

direction. 3

2009; Moynihan and Lavertu 2012). More descriptive information on all variables and their bivariate correlation coeffi cients—which

As noted earlier, our test of specifi c capacity eff ects is not compre- do not exceed a value of 0.55—can be found in table 2.

hensive. We examined four specifi c interactions (we also tested a fi fth interaction with “missing link to action,” which generated

Modeling and Limitations

equivalent fi ndings of the other interactions). It is plausible that To model the diff erent eff ects of training, we use the following

further capacity measures worth examining exist but are not in our strategy: Hypothesis 1 focuses on a general training eff ect. Here, we

data set and that some capacity eff ect occurs in the direct measure simply examine the direct eff ect of training on the two dependent

of training. Th at said, our eff orts to measure specifi c capacity eff ects variables while controlling for 10 variables (see table 3, models 1

are more comprehensive than prior work, and the fi ndings are and 2). Th e expectation is that more exposure to training will make

consistent. We use 10 statistical models to examine fi ve interactions

Table 2 Descriptive Information and Correlation Matrix Variable

(8) (9) (10) 1. Performance information use

2. Strategic goal use

4. Measurement problems

6. Learning routine

1.00 9. Leadership commitment

8. Missing link to action

0.46 0.43 0.43 0.48 1.00 10. Political confl ict

–0.16 0.37 1.00 11. Senior Executive Service

0.19 0.39 0.11 0.17 0.08 0.04 0.21 0.09 0.09 0.14 –0.05 0.17 Note: All information is based on the pooled data from 2000, 2007, and 2013.

Does Training Matter? Evidence from Performance Management Reforms 415

Table 3 Regression Models Direct Effects

Moderated Effects

PI Use Goal Use Training

PI Use

Goal Use

PI Use

Goal Use

PI Use

Goal Use

PI Use

Goal Use

(0.02) (0.02) Learning routine

(0.02) (0.03) Missing link to

(0.02) (0.01) Leadership commit-

(0.02) (0.02) Political confl ict

(0.02) (0.02) Senior Executive

(0.03) (0.04) Training x Measure-

ment problems

Training x Discretion

Training x Learning

Training x 0.01 0.01 Accountability

(0.01) (0.01) N

3,774 3,774 Adjusted R 2 0.303

0.304 0.187 Notes: Standardized coeffi cients and t-statistics are reported for the direct effect models, while unstandardized coeffi cients and standard errors, which are adjusted for 16 agency clusters, are reported for the moderated effects models. This analysis is based on the pooled data from 2000, 2007, and 2013.

*p < .05; **p < .01 (two-tailed tests).

(see table 3), as well as another 10 models featuring only the 2007 eff ects. For the moderated eff ects, unstandardized coeffi cients are and 2013 samples that included data from additional eight agencies

shown, which makes it possible to interpret them in a straightfor- only identifi ed in these waves (results not reported). Of these 20

ward way, as we explain in the Data and Methods section. models, only two found support for a signifi cant capacity interac- tion between training and discretion, whereas 18 showed null

Given the number of variables in table 3, we will not examine eff ects. 4

each in detail but instead center our discussion on (1) the direct eff ects of training; (2) fi ndings related to the use of strategic goals,

Common source bias might generate artifi cial correlations between as this variable is less well examined than forms of performance items, especially when items are highly perceptual and may have a

information use; and (3) the interaction eff ects between train- strong social desirability bias. However, Meier and O’Toole (2013)

ing and management challenges. For the other variables (missing fi nd that measures of performance information use seem less sus-

link to action, leadership commitment, political confl ict, and ceptible to common source bias than measures with a higher social

senior-level position), it is suffi cient to note that our fi ndings are desirability bias, such as estimations of organizational performance.

consistent with prior research (e.g., Dull 2009; Moynihan and In addition, our primary independent variable—whether one par-

Lavertu 2012).

ticipated in training—is a relatively straightforward query about an objective event that lacks a strong social desirability bias, making it

Across all models, we fi nd that training has a signifi cant, posi- less prone to common source bias. Th is follows Meier and O’Toole’s tive, and direct eff ect on the use of performance information and (2013, 447) recommendation to focus on independent variables

strategic goals. Th e magnitude of this eff ect seems to be slightly that are “more specifi c and less likely to generate spurious results.” 5 larger in the “PI use” compared to the “goals use” model, and it is of moderate strength in comparison to the slopes of other variables.

Results

A one-standard-deviation increase in training can be associated with Table 3 portrays the results of the statistical analysis, providing

a 0.10- and a 0.08-standard-deviation increase in the use of perfor- information on the direct eff ects of training (fi rst two columns) as

mance data and strategic goals, respectively. Managers who experi- well as all four interaction eff ects (following columns). For the direct enced more training are more likely to report paying attention to eff ects, table 3 reports standardized regressions coeffi cients, mainly

performance information and to strategic goals for decision making, to provide a better overview of the relative magnitude of the training

a fi nding that supports hypothesis 1.

416 Public Administration Review • May | June 2015

Much of the recent literature on performance management has generally unfamiliar with the content of reforms and the broader focused on the use of performance information, giving little atten-

federal performance framework. Although one might expect that tion to the role of strategic goals. But performance management

familiarity with the GPRA Modernization Act would be greater systems generate strategic goals just as they do other types of per-

because it built on prior reforms, one trainer said, “Managers are formance information, such as measures or targets. Th erefore, there not really familiar with Modernization Act—surprisingly so. Th ey is value in examining patterns associated with the use of strategic

have heard of it, but actual application and use of it are pretty low.” goals and comparing them to the patterns associated with perfor-

Th is observation is in line with descriptive fi ndings in our data: mance information use. Th e results presented here suggest that these although 84 percent of employees said they had heard of the GPRA patterns are broadly similar—variables that predict performance

in 2000, only 64 percent had heard of the Modernization Act in information use also tend to predict the use of strategic goals.

2013, and only 58 percent knew about PART in 2007. One concern is that the strong parallels between the use of strategic

Training materials also tended to convey normative claims about goals and performance information might be driven by an inability

the virtues of performance management, building on the intuitive of respondents to discern between the two. However, closer exami-

appeal of the idea of performance. PART-era training centered on nation of two independent variables that do not produce equivalent

conveying information about PART and on justifying it as a legal fi ndings (measurement problems, senior executive service) undercuts requirement rather than sharing normative arguments as to its ben- this concern. It is intuitively obvious why the variable measurement

efi ts. “With PART, it was more about just education,” one trainer problems limits the use of performance data in decision making,

said. “We did not run it like the workshop for GPRA where we but there is little ex ante reason to assume that such problems would examined the details and challenges of the programs that individuals aff ect the use of strategic goals. It is also plausible that attention

were managing . . . We mostly just made people aware of PART and to broad strategic goals might be greater for more senior managers

what was being asked, so when OMB came to you, you were able to relative to attention to performance data. Th e results follow these

answer questions.” Agency-level training on PART was similar, with expected patterns—diffi culties in measurement undermine the use

a stronger emphasis on how to answer the PART questions rather of performance information but do not matter to the use of strategic than how to use it as a tool to improve performance. goals, and more senior managers devote more attention to the use of strategic goals.

Th e most prominent reason for the limited attention to capacity gaps was that building competencies through training took time and

With respect to the interaction eff ects we theorized in hypotheses resources that were not available. Enabling employees to develop

2, 4, and 5, little evidence supports the possibility that training better measures or make better use of performance feedback is provided capacity skills to overcome some of the challenges of per-

more diffi cult to realize than broadly directing attention to chang- formance management. Th e interactions of training with measure-

ing requirements and new expectations. “To many people training ment problems, learning routines, and accountability turn out to

confi rmed that it [performance management] was a compliance

be insignifi cant. Managers with more training are not better able to exercise,” one interviewee said. “Training was not much more than handle measurement problems and do not get more out of learning

‘here is what GPRA requires.’” Th is perception was partly due to the routines and accountability mechanisms.

diffi culty of dealing with task-specifi c challenges in contexts where trainees came from diff erent backgrounds. A trainer at one non-

Th e one exception is that training seems to help managers use their profi t company noted, “One limitation of training is time and the discretion to facilitate performance information use (hypothesis 3).

cross-agency nature of the group, which limits specifi c discussion of We can see that a one-unit increase on the training scale leads to a

problems in individual agencies.”

signifi cant increase in the eff ect of discretion on performance infor- mation use and goals use (0.02 and 0.01, respectively). Put diff er-

Discussion

ently, the slope relating discretion to data and goals use will be 0.12 We studied performance management reforms in the U.S. federal and 0.06 factor score points steeper for managers who have experi-

government and found that training is positively associated with enced the maximum amount of training (25 percent of the respond- the implementation of these reforms. Managers who were exposed ents) when compared with those who did not receive any training at to more training reported higher levels of use of performance all (34 percent of the respondents). Th is pattern implies that the use

information and strategic goals when making decisions. Such a of data and goals will be further stimulated

positive eff ect aligns with several theoretical when training complements discretion.

explanations, but our interviews pointed out

Managers who were exposed to that most likely “information” and “justifi ca-

tion” mechanisms were at play. Th at is, train- port the fi nding that training had a positive

Our qualitative interview data also sup-

more training reported higher

ing mainly disseminated information about eff ect on reform implementation but did not

levels of use of performance

information and strategic goals how performance management works and

develop performance-management-related

what is expected of managers, and it tried to capacity. Th e offi cials we interviewed were

when making decisions.

explain why these eff orts are necessary and skeptical that capacity gaps could be fi lled in

why they can help achieve better outcomes. relatively short training sessions. Th ey reported that training allotted more time to explaining and justifying reforms than to developing

Th ere are two possible interpretations of these fi ndings. Th e posi- performance management-specifi c capacity. Employees, even those

tive one is that training can have a signifi cant eff ect on reform who cared about performance metrics for their programs, were

implementation, even if it fails to develop reform-specifi c capacity.

Does Training Matter? Evidence from Performance Management Reforms 417

418 Public Administration Review • May | June 2015

federal managers has been told, since the implementation of the GPRA, to consider strategic goals and performance information as connected parts of the same overall system.

Conclusion

Th is article has studied the role of training in reform implementa- tion using the three most recent performance management reform initiatives in the U.S. government as a case study. Th e fi ndings contribute evidence on how training relates to public sector reform implementation. Despite the potential importance of training iden- tifi ed by scholars and practitioners, actual evidence about training is scarce in public sector settings in general and on the impacts on reforms in particular. We fi nd that training is positively associated with reform implementation. Managers who received more training on performance management also paid more attention to perfor- mance data and strategic goals when making decisions. However, training did not lead to the development of specifi c capacities needed to cope with measurement or management challenges. We did not fi nd diff erences in the eff ects of measurement problems, learning, and accountability on reform implementation when com- paring more and less trained managers—a signifi cant interaction between training and discretion was the only exception.

Our fi ndings fall between the negative and positive narratives of training noted in the introduction. Training directed attention to reform implementation, but it failed to equip managers to deal with implementation problems. Based on the case we studied, we can draw the following conclusions for the practice of public adminis- tration more broadly. First, training matters, but an understanding how training matters remains elusive. As organizations undertake large-scale change, their managers should outline what they want training to do and how it will reach its goals. What is it that training is supposed to accomplish: inform employees, establish a new cul- ture, develop reform-specifi c capacity, or a combination of all three? Based on the focus, training programs can be designed and have to

be evaluated diff erently. Second, higher-order needs, such as creating performance-manage-

ment-related capacity, require more resources than other training objectives. Although our quantitative models did not off er evidence on the eff ect of resources on training outcomes, our interviewees pointed out that not having enough funds and time was one of the major drawbacks for developing specifi c capacities. Interviewees also saw resources for training declining since the fi rst training programs for the GPRA were initiated. Th e Barack Obama administration has acknowledged the issue, proposed more spending on training, and directed agencies to target their training needs (OMB 2014).

Th ird, the potential of training depends not just on adequate resources, but also on addressing specifi c capacity gaps. Th e types of performance management challenges we examine—dealing with measurement problems, making the most of performance feedback, and responding to a context with a strong accountability empha- sis—are well established in prior research but appear to be mostly unaddressed by training eff orts. Th is disjunction between what research identifi es as implementation challenges and the actual con- tent of training intended to facilitate implementation suggests an opportunity for better connecting scholarship and practice. Research can identify implementation challenges. Training can address those

Explaining and justifying reforms, which are lower-order condi- tions for change, may improve implementation, suggesting that any training is better than no training. Th e more negative reading is that training could have done much more, if it had aimed at creating performance-management-specifi c capacity. However, to better compare the eff ects of lower- and higher-order change conditions, and to disentangle the eff ects of information, justifi cation, and capacity, we clearly need further research. Th e most robust design for this purpose would require a randomized quasi-experiment with three treatment groups: an organization off ers three training programs, each of which only focuses on information, norma- tive justifi cation, or capacity. Managers are randomly selected into programs, and reform implementation can be compared before and after the training programs, as well as among groups, and compared to a control group.

Our results also did not provide strong evidence that training created performance-management-related capacity to help managers better deal with known challenges. Th e interviews suggest that this defi cit was attributable to the limited time and resources devoted to per- formance management training and the diversity of training groups, which limited discussion to generic issues common to multiple agencies. However, to better determine the extent to which resources aff ect training outcomes, or whether diff erent training purposes require diff erent levels of funding, we need further research.

One exception is that we found modest evidence that training helped managers get more out of the discretion granted to them. While we are cautious about overstating this fi nding, as adding this interaction term only marginally improved model fi t, it points toward an interesting implication for our understanding of the relationship between discretion and performance information use. Empirical studies suggest that performance management has often not met expectations because increasing output control was not complemented by the provision of greater operational discretion (Moynihan 2008; Nielsen 2014). Indeed, our results provide further support for the direct role of discretion in facilitating the use of per- formance data. But even if managers have discretion, they may still lack a strong sense of how to make use of performance information. Our fi ndings off er some evidence that training may help managers better understand how to utilize their discretion to use performance information and strategic goals.

Our research employed an indirect measure of capacity using inter- action eff ects—we considered capacity to be created if managers who experienced more training were better able to handle known performance-management-related challenges than those with less training. Another way to learn more about capacity would be to measure it more directly based on surveys through supervisor ratings or to observe it as a part of an experimental study before and after managers’ participation in training programs. Th ese methods would better examine the direct eff ect of training on capacity and sort out issues of causality with which correlational research often struggles.

For readers interested in performance management issues, this arti- cle is the fi rst of which we are aware that compares the use of strate- gic goals and the use of performance information. Apart from some logical exceptions, there are strong parallels between the two behav- iors. One plausible reason for these parallels is that a generation of For readers interested in performance management issues, this arti- cle is the fi rst of which we are aware that compares the use of strate- gic goals and the use of performance information. Apart from some logical exceptions, there are strong parallels between the two behav- iors. One plausible reason for these parallels is that a generation of

Social Sciences, edited by Warren G. Bennis, Kenneth D. Benne, and Robert

initial eff ort to conceptualize training, so it facilitates such dialogue

Chin, 32–59. New York: Holt, Rinehart and Winston.

between research and practice and off ers guidance for training in the Darnall, Nicole, and Yongsung Kim. 2012. Which Types of Environmental implementation of performance management reforms.

Management Systems are Related to Greater Environmental Improvements? Public Administration Review 72(3): 351–65.

Notes

e Iron Cage Revisited: 1. We turned to interviews in part because documentation on basics such as overall

DiMaggio, Paul J., and Walter W. Powell. 1983. Th

Institutional Isomorphism and Collective Rationality in Organizational Fields. spending on training and how funds are divided among training providers is not