Directory UMM :Data Elmu:jurnal:A:Accounting, Organizations and Society:Vol25.Issue4-5.May2000:
Accounting, Organizations and Society 25 (2000) 483±496
www.elsevier.com/locate/aos
Budgetary criteria in performance evaluation: a critical
appraisal using new evidence
David Otley a,*, Raili M. Pollanen b
a
Department of Accounting and Finance, Lancaster University, Lancaster LA1 4YX, UK
b
Faculty of Administration, Regina University, Regina SK, S45 0A2, Canada
Abstract
Previous studies on the use of budgetary criteria in performance evaluation have used a wide range of variables,
measures and models, but have concentrated upon extending rather than repeating previous work. This study includes
replications of parts of ®ve previous studies and obtains results that dier in many respects from previous ®ndings.
Because of the use of similar measures and models, it can be concluded that the dierent results stem primarily from
the dierent sample of managers studied. More generally, it is suggested that control practices dier across organizations, cultures and time; universal ®ndings should not be expected. Rather, a more coherent programme of work is
required to explicate the varied ways in which budgetary techniques can be used in performance evaluation and management. # 2000 Elsevier Science Ltd. All rights reserved.
1. Introduction
The past 25 years have seen a small but continuing stream of studies that have sought to
explore the in¯uence of budgetary controls on
managerial performance. The seminal work in this
®eld was the study by Hopwood (1972) which
explored the role of accounting data in managerial
performance evaluation. It was quickly followed
by a study which, using similar measures, found
con¯icting results (Otley, 1978). This apparent
con¯ict stimulated a stream of work, much of it
from Australian authors (e.g. Brownell, 1982,
1985; Dunk, 1989; Harrison, 1992), which introduced additional variables in order to provide
contingent explanations of the observed dierences in managerial behaviour.
* Corresponding author.
In one sense, this stream of work was clearly
successful. A variety of exploratory studies produced a range of ®ndings and statistically signi®cant results. Indeed, Brownell and Dunk (1991)
stated that ``the continuing stream of work devoted to this issue constitutes, in our view, the only
organized critical mass of empirical work in management accounting at present''. Even so, the
work is characterised by a number of de®ciencies.
Dierent authors have tackled dierent aspects of
the issue; dierent studies use dierent subsets of
variables; in many cases, the same variable is
measured in importantly dierent ways. The samples of managers studied have been drawn from
dierent types of organization in dierent countries and the sample size has been generally quite
small. Some studies use samples of managers
drawn from a single organization; others use
managers from dierent organizations. Most
0361-3682/00/$ - see front matter # 2000 Elsevier Science Ltd. All rights reserved.
PII: S0361 -3 682(98)00031 -2
484
D. Otley, R.M. Pollanen / Accounting, Organizations and Society 25 (2000) 483±496
surprisingly, no single study has been precisely
replicated; each new piece of work has chosen to
vary some aspect of the research design so that it
is impossible to gauge the general validity of any
of the reported work.
The motivation behind the study reported here
is to repeat, in a dierent context, several of the
major previous studies to assess the robustness of
their ®ndings and to form the basis of further
work that is truly cumulative in nature. Such an
endeavour requires attention to be paid to the
conceptualisation of relevant variables, their measurement and the type of sample drawn. However,
this is a substantial exercise and the work reported
here concentrates upon repeating some of the
more important previous studies. This has led to
each previous study being replicated using (as far
as possible) exactly the same measures as originally used, despite the known problems with some
of the measurement instruments, especially evaluative style. The issues involved in conceptualizing evaluative style is the subject of another paper
(Otley and Fakiolas, 1998). A reasonably large
sample (n=ca. 125) for this body of literature was
used, consisting of managers drawn from a single
sector having common control practices in order
to reduce the confounding eects of using a sample of managers drawn from a variety of dierent
enterprises and industries.
The results indicate a range of outcomes. Some
results are con®rmed; some ®ndings are insignificant statistically; but other results are contrary to
those previously reported. At the very least, this
indicates the need for a more careful and benchmarked approach to research design in this ®eld.
At present, we cannot judge the extent to which
statistically signi®cant reported results are likely to
be generalizable to other contexts. It is hoped that
this paper represents one small step along the path
towards the accumulation of more reliable results
in this important area.
2. Hypotheses
A set of interaction studies were selected, all of
which made use of Hopwood's original instrument
to measure evaluative style or what has now
become known as Reliance on Accounting Performance Measures (RAPM). These were Brownell (1982), Brownell and Hirst (1986), Dunk
(1989), Brownell and Dunk (1991) and Harrison
(1992). All of these studies included budgetary
participation as an explanatory variable, and all
considered managerial performance as the dependent variable, with the exception of Harrison
(1992) where job-related tension (JRT) was used
as a surrogate. However, as JRT was also used as
an additional dependent variable in Brownell and
Hirst, it seemed appropriate to include JRT as an
additional dependent variable. Two studies also
used job satisfaction as a dependent variable, and
two used job satisfaction as a moderating variable,
so this was also included. Finally, two studies
identi®ed task uncertainty as a moderating variable, but each used dierent measures. We chose
to include task uncertainty, but to use an alternative measure better suited to our context. The
above selection gave a reasonable spread of studies and hypotheses to be replicated within the
constraint of using a questionnaire of reasonable
length that would enable a high response rate to
be obtained.
The study therefore attempts to replicate some
important parts of previous studies by testing the
following hypotheses, stated in their null form:
1. Brownell (1982) H1: There is no interaction
between supervisory evaluative style and
budgetary participation aecting performance or job satisfaction.
2. Brownell and Hirst (1986) H2: There is no
interaction between budget emphasis, budgetary participation and task uncertainty
aecting performance or JRT.
3. Dunk (1989) H3: There is no interaction
between budget emphasis and budgetary
participation aecting performance.
4. Brownell and Dunk (1991) H4: There is no
interaction between budget emphasis, budgetary participation and task uncertainty
aecting performance.
5. Harrison (1992) H5: There is no interaction
between budget emphasis and participation
aecting job satisfaction and job-related
tension.
D. Otley, R.M. Pollanen / Accounting, Organizations and Society 25 (2000) 483±496
3. Measurement of variables
3.1. Budget emphasis [RAPM]
Much of the preceding work, and all the studies
being replicated, based their measurement of budget emphasis on Hopwood's (1972) instrument.
The eight-item instrument developed by Hopwood
was subsequently modi®ed and used by Otley
(1978), Brownell (1982), Brownell and Hirst (1986)
and others. The ten-item variant used by Brownell
and Hirst was used here, but with two minor variations. First, the measure used here includes one
attitude item instead of the two items into which
Brownell and Hirst had split the original scale
item. Second, an additional item regarding cooperation with peers was added. Neither of these
modi®cations aected the two essential scale
items, namely `my long-term concern with costs
and revenues' and `my ability to meet budgets in
the short-run' that Hopwood contrasted to
develop his measure of evaluative style.
Both an importance rating and a ranking scale
were used, as in Hopwood's original study. However, in order to help ensure that the ranking
question was properly answered following reported problems in its use by Brownell (1982), the
question was asked twice, once with rankings
required and once for importance scores. This
proved successful in obtaining complete responses.
The information obtained from these two sets of
scores enable all variants of the budget emphasis
measure used in the major studies being replicated
to be precisely derived.
485
importance scale was used, which is identical to all
the previous studies except Dunk (1989) who
appears to have used a ®ve-point scale. The
descriptive statistics are shown in Table 1.
3.3. Task uncertainty
This proved to be the most dicult variable to
replicate as the two previous studies that incorporated this variable used dierent measures. As
each measure used multi-item scales, it was not felt
desirable to include a variety of dierent measures
into the questionnaire for reasons of length. The
measure developed by Sathe (1974), which itself
slightly modi®ed Duncan's (1972) measure, was
selected because of its broad scope and likely
applicability to the research site chosen. This
measure incorporates three main dimensions of
task uncertainty: lack of information about environmental factors; inability to attach probabilities
to environmental factors; and not knowing the
eects of incorrect decisions. The broader scope of
this measure made it more suitable for use in the
public sector organizations being studied. The
original scale had 12 items which yielded a Cronbach alpha of 64%, but reliability analysis indicated that this would improve to 70% if two items
were omitted. As neither previous study being
replicated had used the same measure, these two
items were removed. Factor analysis indicated
three underlying factors, as expected, accounting
for 50% of the total variance. A single measure
comprising of the ten selected items was therefore
used.
3.2. Participation
3.4. Job satisfaction
The six-item measure developed by Milani
(1975) was used as it had been used in almost
identical form in all the previous studies. These
items were incorporated into a larger, nine-item
scale that included three extra items developed by
Pope and Otley (1996). However, these extra items
did not correlate well with the Milani items and
were dropped from the further analysis. When the
scale was factor analysed all six items loaded onto
a single factor and the six-item Milani scale
showed a Cronbach alpha of 83%. A seven-point
The 20-item short version of the Minnesota
Satisfaction Questionnaire (MSQ) (Weiss, Davis,
England, & Lofquist, 1967) was used to measure
job satisfaction, where each single item corresponds to the 20 sub-dimensions of the longer 100item version of the questionnaire. This measure
was used by Brownell (1982, 1983) and Harrison
(1992). Minor changes in wording were made to
re¯ect the organizations being studied (e.g. `company' is replaced by `organization') but these are
not seen as making a signi®cant change. Factor
486
D. Otley, R.M. Pollanen / Accounting, Organizations and Society 25 (2000) 483±496
Table 1
Descriptive statistics of measures
Variables
No. of
items
Role of accounting inf. (Brownell, 1985)
2
Relative role of accounting inf. (Harrison, 1992)
2/8
Budget participation (Milani, 1975)
6
Task uncertainty (Sathe, 1974)
10
Job satisfaction (Weiss, Davis, England, & Lofquist, 1967) 20
Job-related tension (Kahn et al., 1964)
15
Performance (Mahoney et al., 1963)
1
Sample Mean
size
123
121
121
121
123
127
126
1.99
1.00
2.25
2.49
2.02
2.50
4.02
SD
0.93
0.50
0.95
0.33
0.41
0.45
0.65
Theoretical range Actual range
Min
Max
Min
Max
1
0.2
1
1
1
1
1
5
5
7
5
5
5
5
1
0.3
1
1.8
1.2
1.2
2.0
5
4.0
6.2
3.8
3.3
3.7
5.0
Distribution of evaluative styles
Evaluative style
BC (%)
BP (%)
PC (%)
NA (%)
This study
Hopwood (1972)
Otley (1978)
Brownell (1982)
Brownell and Hirst (1986)
6
20
13
21
17
8
10
56
24
41
30
26
28
24
7
56
44
3
31
35
analysis indicated ®ve factors, accounting for 59%
of the total variance, but with a predominant
initial factor which accounted for 29% of the variance. It therefore seemed reasonable to use a single scale comprising of the 20 items which had a
Cronbach alpha of 85%.
3.5. Job-related tension
Job-related tension [JRT] was measured using
the 15-item instrument developed by the Institute
for Social Research at the University of Michigan
(Kahn, Wolfe, Quinn, Snoek, & Rosenthal, 1964).
This has been widely used with consistently reliable results. Two items on budget-related tension
were added to the instrument for other purposes,
but not used in the analyses reported here. A few
minor changes in wording were made as with previous scales. Factor analysis indicated four factors
accounting for 58% of the total variance, but with
one predominant factor accounting for 30%. Thus
a single 15-item scale was used and an overall
Cronbach alpha of 81% obtained. The average
JRT score of 2.50 is comparable to that found in
other studies.
3.6. Performance
The nine-item scale developed by Mahoney,
Jerdee, and Carroll (1963) has been widely used. It
consists of eight sub-dimensions and one overall
question. However, in the literature being replicated, only the overall measure was ever used after
it was demonstrated that the overall measure
explained a substantial proportion of the variance
in the other eight items. Thus, only the overall
measure was selected for use in this study, to keep
the questionnaire length within bounds.
4. Research site
In selecting a research site, two con¯icting
requirements became apparent. Although drawing
managers from a wide range of dierent organizations may appear to give representativeness and
allow results to be generalized, the restricted range
of control practices observed in a narrowly based
questionnaire survey does not assist in controlling
for the impact of unobserved practices. It may,
therefore, be more meaningful to select managers
D. Otley, R.M. Pollanen / Accounting, Organizations and Society 25 (2000) 483±496
from a single organization where such practices
might both be observed and have a more restricted
range of variation. However, the results would not
necessarily be generalizable beyond the organization concerned, which would also have to be quite
large in order to allow a sample of adequate size to
be drawn. Previous studies have used both types of
sample. In the event, it was decided to restrict the
scope of the study to a single sector, but to sample
from as complete a range of organizations as possible within that sector. At the very least, the results
obtained ought to be generalizable to that sector.
The chosen sector was the administration of
universities and colleges in the province of
Ontario, Canada. This allowed a relatively homogeneous population of institutions large enough
for budgetary control techniques to be an important control mechanism, and subject to common
control procedures. Although the sector contains
dierent types of organization, because of reporting and control requirements determined by the
Ontario Ministry of Education and Training they
face a relatively uniform control environment, at
least in terms of upward accountability. Access
was negotiated through the Deputy Minister, by
writing to the presidents of 35 publicly funded
colleges and universities, of whom 16 agreed to
participate. These presidents supplied lists of
senior administrators having signi®cant budgetary
responsibilities and 176 such people were selected
for the ®nal mailing list (with titles such as vicepresident for academic, administrative, ®nancial
or service operations, or dean of an academic
programme area). Such people typically reported
to the president either directly or via just one level
of management.
A substantial questionnaire, consisting of the
questions already outlined plus an approximately
equal number of additional items relating to other
budgetary behaviours was sent to each of the 176
selected respondents. A total of 134 replies (76%)
were obtained after some follow-up by telephone,
of which 127 were usable (72% net response rate).
These replies appeared to be representative of the
institutions sampled, with no evidence of nonresponse bias being apparent.1 A few questionnaires were not fully completed in all respects,
so the sample size is not 127 for all analyses, but it
487
always exceeds 120 and is reported for each set of
results. Individual responses were kept anonymous, but a report of the aggregate results was
prepared which was sent back to the presidents of
participating colleges. Summary statistics of the
variables relevant to this paper are given in
Table 1. These indicate a wide range of responses
and considerable variation between respondents
on the independent and dependent variables.
5. Results
The results will be presented as a series of comparisons between the ®ndings of this study, and
those found in the individual studies replicated, as
one of the main purposes of this paper is to draw
attention to the similarities and dierences that
occur.2 Similarities in ®ndings may indicate that
underlying results are persistent and stable features of behaviour across a range of organizations
and situations; dierences, by contrast, indicate
that the previous results should not be treated as
universal, and that more attention needs to be
paid to the circumstances under which greater
generalisation is likely to be possible. It should be
noted that the measures used are adjusted to
match those used in the original studies wherever
possible; this means that the reported results cannot be compared across studies, except where indicated, as dierent scales may be used in each case.
5.1. Brownell (1982)
This study examined the eect of budget
emphasis and participation on job satisfaction and
managerial performance, in particular hypothesising an interaction between budget emphasis and
1
For reasons of con®dentiality, it was not possible to determine response rates by institution; however, descriptive data
relating to respondents appeared to appropriately re¯ect the
characteristics of the sector.
2
A more coherent analysis of the results relating to the sector
will be provided in a future paper. In this paper, the variables
and measures used are dierent in each replication, to ensure
as a precise a replication as possible in each case. This explains
the dierences in the descriptive statistics reported in Tables 1,
2, 5 and 7.
488
D. Otley, R.M. Pollanen / Accounting, Organizations and Society 25 (2000) 483±496
participation. The sample was 48 managers holding cost centre responsibility in a US manufacturing company. Budget emphasis was measured
using Hopwood's method of comparing the ranking of the two relevant items from the 10-item
measure. Brownell then combined Hopwood's
Budget-Constrained [BC] and Budget-Pro®t [BP]
categories into a high budget emphasis category
(coded ÿ1), and his Pro®t Conscious [PC] and
Non-Accounting [NA] categories into a low budget emphasis category (coded +1). The descriptive
statistics for each study are shown in Table 2, with
this study's variables recoded to exactly match
those of Brownell (1982).3
There are two main dierences. First, there was
a considerably higher level of budgetary participation in this study compared with Brownell's.
Therefore analyses were re-run coding participation as high or low using both this study's mean
and Brownell's mean. Although this causes some
dierences in the results, the main interaction
eect is virtually unchanged. Second, the distribution of budget emphasis scores is rather dierent,
with only 14% [compared to 45% in Brownell
(1982)] being categorised as high budget emphasis.4 Nevertheless this still represents some 17
respondents, the same absolute number as Brownell, and does not aect the comparisons by
category.
The results of regressing the data in the interaction model are shown in Table 3, for both job
satisfaction and managerial performance. These
clearly indicate dierent results. The interaction
term is on the borderlines of statistical signi®cance
at the 5% level for job satisfaction in this study,
but is not signi®cant for managerial performance.
These results are thus quite dierent to those found
by Brownell (1982). With regard to job satisfaction,
3
In all cases, re-coding was performed to give exact comparibility between all the reported regression coecients. This goes
some way towards countering Southwood's (1978) assertion
that only the highest order interaction coecient is meaningful
because other coecients can change if Likert-scaled variables
are re-coded. We would contend that where measures of variables are precisely replicated, and the same coding is used in
two studies, then it is both valid and useful to compare all
reported coecients.
4
These dierences most likely re¯ect the dierent sector studied and also cultural dierences between countries.
the supposed incompatible combinations of participation and budget emphasis (high±high and low±
low) are associated with high job satisfaction and
the compatible combinations (low±high and high±
low) are associated with low job satisfaction. This
result is con®rmed by means tests and graphical
analysis, and by using two alternative measures of
budget emphasis. With regard to performance,
where Brownell found a signi®cant and positive
result, this study found a negative, albeit insignificant, result. If an alternative measure of budget
emphasis is used [combining all the accounting
styles (BC+BP+PC) into one category and nonaccounting into the other] the negative result
becomes signi®cant at the 2% level.
In summary, the results of this study are not consistent with Brownell's ®ndings; the relationships
found tend to be in the opposite direction, although
not always achieving statistical signi®cance. This is
unlikely to be an artefact of the dierent distribution of budget emphasis that was observed, nor of
the dierent absolute level of participation.
Rather, it has to be laid at the door of the dierent
context in which the study was conducted. What is
clear is that the Brownell (1982) result is not universal; the limits of its validity need further
exploration.
5.2. Brownell and Hirst (1986)
Part of Brownell and Hirst's (1986) study replicated Brownell (1982) but using job-related tension (JRT) as a dependent variable in place of
performance and job satisfaction. This will be
considered before the main part of their study that
added task uncertainty as a further explanatory
variable.
One relatively minor dierence between the two
studies is that budget emphasis is not measured by
the relative rankings of the top 3 items on the
Hopwood scale, but uses the top 4 items. This
procedure is followed here for our own results.5 A
direct comparison of the eect of budget emphasis
and participation on JRT with Brownell and Hirst
(1986) cannot be made as they do not present this
restricted equation (although it can be compared
with the 3-way interaction equation presented
later). However, the 2-way interaction coecient
489
D. Otley, R.M. Pollanen / Accounting, Organizations and Society 25 (2000) 483±496
Table 2
Descriptive statistics for continuous variablesÐcomparison with Brownell (1982)
Variables
No. of items
Sample size
Mean
SD
Theoretical range
Actual range
Min
Max
Min
Max
This study
Budget participation
Job satisfaction
Managerial performance
6
20
1
121
123
126
34.48
79.59
7.03
5.69
8.18
1.29
6
20
1
42
100
9
11
54
3
42
97
9
Brownell (1982)
Budget participation
Job satisfaction
Managerial performance
6
20
1
38
38
38
25.79
76.50
6.84
7.10
7.25
0.95
6
20
1
42
100
9
13
58
5
39
93
8
Note: Consistent with Brownell (1982), budget participation was reverse-coded and the total (rather than the average) score used.
Similarly, job satisfaction was reverse-coded and converted to a 20±100 scale. Managerial performance was re-coded to a 9-point scale.
Table 3
Results of regressing job satisfaction and managerial performance on budget emphasisÐcomparison with Brownell (1982)
Variables
Job satisfaction
Intercept
Budget participation
Budget emphasis
Participation
budget emphasis
This study
Brownell (1982)
Value
p
Value
78.4
ÿ0.41
1.17
ÿ0.28
0.0000
0.0050
0.2448
0.0515
76.5
ÿ0.19
ÿ0.79
0.05
R2=0.19
Managerial performance
Intercept
Budget participation
Budget emphasis
Participation
budget emphasis
7.08
0.03
ÿ0.06
ÿ0.04
0.0000
0.2302
0.7352
0.1533
R2=0.02
p
www.elsevier.com/locate/aos
Budgetary criteria in performance evaluation: a critical
appraisal using new evidence
David Otley a,*, Raili M. Pollanen b
a
Department of Accounting and Finance, Lancaster University, Lancaster LA1 4YX, UK
b
Faculty of Administration, Regina University, Regina SK, S45 0A2, Canada
Abstract
Previous studies on the use of budgetary criteria in performance evaluation have used a wide range of variables,
measures and models, but have concentrated upon extending rather than repeating previous work. This study includes
replications of parts of ®ve previous studies and obtains results that dier in many respects from previous ®ndings.
Because of the use of similar measures and models, it can be concluded that the dierent results stem primarily from
the dierent sample of managers studied. More generally, it is suggested that control practices dier across organizations, cultures and time; universal ®ndings should not be expected. Rather, a more coherent programme of work is
required to explicate the varied ways in which budgetary techniques can be used in performance evaluation and management. # 2000 Elsevier Science Ltd. All rights reserved.
1. Introduction
The past 25 years have seen a small but continuing stream of studies that have sought to
explore the in¯uence of budgetary controls on
managerial performance. The seminal work in this
®eld was the study by Hopwood (1972) which
explored the role of accounting data in managerial
performance evaluation. It was quickly followed
by a study which, using similar measures, found
con¯icting results (Otley, 1978). This apparent
con¯ict stimulated a stream of work, much of it
from Australian authors (e.g. Brownell, 1982,
1985; Dunk, 1989; Harrison, 1992), which introduced additional variables in order to provide
contingent explanations of the observed dierences in managerial behaviour.
* Corresponding author.
In one sense, this stream of work was clearly
successful. A variety of exploratory studies produced a range of ®ndings and statistically signi®cant results. Indeed, Brownell and Dunk (1991)
stated that ``the continuing stream of work devoted to this issue constitutes, in our view, the only
organized critical mass of empirical work in management accounting at present''. Even so, the
work is characterised by a number of de®ciencies.
Dierent authors have tackled dierent aspects of
the issue; dierent studies use dierent subsets of
variables; in many cases, the same variable is
measured in importantly dierent ways. The samples of managers studied have been drawn from
dierent types of organization in dierent countries and the sample size has been generally quite
small. Some studies use samples of managers
drawn from a single organization; others use
managers from dierent organizations. Most
0361-3682/00/$ - see front matter # 2000 Elsevier Science Ltd. All rights reserved.
PII: S0361 -3 682(98)00031 -2
484
D. Otley, R.M. Pollanen / Accounting, Organizations and Society 25 (2000) 483±496
surprisingly, no single study has been precisely
replicated; each new piece of work has chosen to
vary some aspect of the research design so that it
is impossible to gauge the general validity of any
of the reported work.
The motivation behind the study reported here
is to repeat, in a dierent context, several of the
major previous studies to assess the robustness of
their ®ndings and to form the basis of further
work that is truly cumulative in nature. Such an
endeavour requires attention to be paid to the
conceptualisation of relevant variables, their measurement and the type of sample drawn. However,
this is a substantial exercise and the work reported
here concentrates upon repeating some of the
more important previous studies. This has led to
each previous study being replicated using (as far
as possible) exactly the same measures as originally used, despite the known problems with some
of the measurement instruments, especially evaluative style. The issues involved in conceptualizing evaluative style is the subject of another paper
(Otley and Fakiolas, 1998). A reasonably large
sample (n=ca. 125) for this body of literature was
used, consisting of managers drawn from a single
sector having common control practices in order
to reduce the confounding eects of using a sample of managers drawn from a variety of dierent
enterprises and industries.
The results indicate a range of outcomes. Some
results are con®rmed; some ®ndings are insignificant statistically; but other results are contrary to
those previously reported. At the very least, this
indicates the need for a more careful and benchmarked approach to research design in this ®eld.
At present, we cannot judge the extent to which
statistically signi®cant reported results are likely to
be generalizable to other contexts. It is hoped that
this paper represents one small step along the path
towards the accumulation of more reliable results
in this important area.
2. Hypotheses
A set of interaction studies were selected, all of
which made use of Hopwood's original instrument
to measure evaluative style or what has now
become known as Reliance on Accounting Performance Measures (RAPM). These were Brownell (1982), Brownell and Hirst (1986), Dunk
(1989), Brownell and Dunk (1991) and Harrison
(1992). All of these studies included budgetary
participation as an explanatory variable, and all
considered managerial performance as the dependent variable, with the exception of Harrison
(1992) where job-related tension (JRT) was used
as a surrogate. However, as JRT was also used as
an additional dependent variable in Brownell and
Hirst, it seemed appropriate to include JRT as an
additional dependent variable. Two studies also
used job satisfaction as a dependent variable, and
two used job satisfaction as a moderating variable,
so this was also included. Finally, two studies
identi®ed task uncertainty as a moderating variable, but each used dierent measures. We chose
to include task uncertainty, but to use an alternative measure better suited to our context. The
above selection gave a reasonable spread of studies and hypotheses to be replicated within the
constraint of using a questionnaire of reasonable
length that would enable a high response rate to
be obtained.
The study therefore attempts to replicate some
important parts of previous studies by testing the
following hypotheses, stated in their null form:
1. Brownell (1982) H1: There is no interaction
between supervisory evaluative style and
budgetary participation aecting performance or job satisfaction.
2. Brownell and Hirst (1986) H2: There is no
interaction between budget emphasis, budgetary participation and task uncertainty
aecting performance or JRT.
3. Dunk (1989) H3: There is no interaction
between budget emphasis and budgetary
participation aecting performance.
4. Brownell and Dunk (1991) H4: There is no
interaction between budget emphasis, budgetary participation and task uncertainty
aecting performance.
5. Harrison (1992) H5: There is no interaction
between budget emphasis and participation
aecting job satisfaction and job-related
tension.
D. Otley, R.M. Pollanen / Accounting, Organizations and Society 25 (2000) 483±496
3. Measurement of variables
3.1. Budget emphasis [RAPM]
Much of the preceding work, and all the studies
being replicated, based their measurement of budget emphasis on Hopwood's (1972) instrument.
The eight-item instrument developed by Hopwood
was subsequently modi®ed and used by Otley
(1978), Brownell (1982), Brownell and Hirst (1986)
and others. The ten-item variant used by Brownell
and Hirst was used here, but with two minor variations. First, the measure used here includes one
attitude item instead of the two items into which
Brownell and Hirst had split the original scale
item. Second, an additional item regarding cooperation with peers was added. Neither of these
modi®cations aected the two essential scale
items, namely `my long-term concern with costs
and revenues' and `my ability to meet budgets in
the short-run' that Hopwood contrasted to
develop his measure of evaluative style.
Both an importance rating and a ranking scale
were used, as in Hopwood's original study. However, in order to help ensure that the ranking
question was properly answered following reported problems in its use by Brownell (1982), the
question was asked twice, once with rankings
required and once for importance scores. This
proved successful in obtaining complete responses.
The information obtained from these two sets of
scores enable all variants of the budget emphasis
measure used in the major studies being replicated
to be precisely derived.
485
importance scale was used, which is identical to all
the previous studies except Dunk (1989) who
appears to have used a ®ve-point scale. The
descriptive statistics are shown in Table 1.
3.3. Task uncertainty
This proved to be the most dicult variable to
replicate as the two previous studies that incorporated this variable used dierent measures. As
each measure used multi-item scales, it was not felt
desirable to include a variety of dierent measures
into the questionnaire for reasons of length. The
measure developed by Sathe (1974), which itself
slightly modi®ed Duncan's (1972) measure, was
selected because of its broad scope and likely
applicability to the research site chosen. This
measure incorporates three main dimensions of
task uncertainty: lack of information about environmental factors; inability to attach probabilities
to environmental factors; and not knowing the
eects of incorrect decisions. The broader scope of
this measure made it more suitable for use in the
public sector organizations being studied. The
original scale had 12 items which yielded a Cronbach alpha of 64%, but reliability analysis indicated that this would improve to 70% if two items
were omitted. As neither previous study being
replicated had used the same measure, these two
items were removed. Factor analysis indicated
three underlying factors, as expected, accounting
for 50% of the total variance. A single measure
comprising of the ten selected items was therefore
used.
3.2. Participation
3.4. Job satisfaction
The six-item measure developed by Milani
(1975) was used as it had been used in almost
identical form in all the previous studies. These
items were incorporated into a larger, nine-item
scale that included three extra items developed by
Pope and Otley (1996). However, these extra items
did not correlate well with the Milani items and
were dropped from the further analysis. When the
scale was factor analysed all six items loaded onto
a single factor and the six-item Milani scale
showed a Cronbach alpha of 83%. A seven-point
The 20-item short version of the Minnesota
Satisfaction Questionnaire (MSQ) (Weiss, Davis,
England, & Lofquist, 1967) was used to measure
job satisfaction, where each single item corresponds to the 20 sub-dimensions of the longer 100item version of the questionnaire. This measure
was used by Brownell (1982, 1983) and Harrison
(1992). Minor changes in wording were made to
re¯ect the organizations being studied (e.g. `company' is replaced by `organization') but these are
not seen as making a signi®cant change. Factor
486
D. Otley, R.M. Pollanen / Accounting, Organizations and Society 25 (2000) 483±496
Table 1
Descriptive statistics of measures
Variables
No. of
items
Role of accounting inf. (Brownell, 1985)
2
Relative role of accounting inf. (Harrison, 1992)
2/8
Budget participation (Milani, 1975)
6
Task uncertainty (Sathe, 1974)
10
Job satisfaction (Weiss, Davis, England, & Lofquist, 1967) 20
Job-related tension (Kahn et al., 1964)
15
Performance (Mahoney et al., 1963)
1
Sample Mean
size
123
121
121
121
123
127
126
1.99
1.00
2.25
2.49
2.02
2.50
4.02
SD
0.93
0.50
0.95
0.33
0.41
0.45
0.65
Theoretical range Actual range
Min
Max
Min
Max
1
0.2
1
1
1
1
1
5
5
7
5
5
5
5
1
0.3
1
1.8
1.2
1.2
2.0
5
4.0
6.2
3.8
3.3
3.7
5.0
Distribution of evaluative styles
Evaluative style
BC (%)
BP (%)
PC (%)
NA (%)
This study
Hopwood (1972)
Otley (1978)
Brownell (1982)
Brownell and Hirst (1986)
6
20
13
21
17
8
10
56
24
41
30
26
28
24
7
56
44
3
31
35
analysis indicated ®ve factors, accounting for 59%
of the total variance, but with a predominant
initial factor which accounted for 29% of the variance. It therefore seemed reasonable to use a single scale comprising of the 20 items which had a
Cronbach alpha of 85%.
3.5. Job-related tension
Job-related tension [JRT] was measured using
the 15-item instrument developed by the Institute
for Social Research at the University of Michigan
(Kahn, Wolfe, Quinn, Snoek, & Rosenthal, 1964).
This has been widely used with consistently reliable results. Two items on budget-related tension
were added to the instrument for other purposes,
but not used in the analyses reported here. A few
minor changes in wording were made as with previous scales. Factor analysis indicated four factors
accounting for 58% of the total variance, but with
one predominant factor accounting for 30%. Thus
a single 15-item scale was used and an overall
Cronbach alpha of 81% obtained. The average
JRT score of 2.50 is comparable to that found in
other studies.
3.6. Performance
The nine-item scale developed by Mahoney,
Jerdee, and Carroll (1963) has been widely used. It
consists of eight sub-dimensions and one overall
question. However, in the literature being replicated, only the overall measure was ever used after
it was demonstrated that the overall measure
explained a substantial proportion of the variance
in the other eight items. Thus, only the overall
measure was selected for use in this study, to keep
the questionnaire length within bounds.
4. Research site
In selecting a research site, two con¯icting
requirements became apparent. Although drawing
managers from a wide range of dierent organizations may appear to give representativeness and
allow results to be generalized, the restricted range
of control practices observed in a narrowly based
questionnaire survey does not assist in controlling
for the impact of unobserved practices. It may,
therefore, be more meaningful to select managers
D. Otley, R.M. Pollanen / Accounting, Organizations and Society 25 (2000) 483±496
from a single organization where such practices
might both be observed and have a more restricted
range of variation. However, the results would not
necessarily be generalizable beyond the organization concerned, which would also have to be quite
large in order to allow a sample of adequate size to
be drawn. Previous studies have used both types of
sample. In the event, it was decided to restrict the
scope of the study to a single sector, but to sample
from as complete a range of organizations as possible within that sector. At the very least, the results
obtained ought to be generalizable to that sector.
The chosen sector was the administration of
universities and colleges in the province of
Ontario, Canada. This allowed a relatively homogeneous population of institutions large enough
for budgetary control techniques to be an important control mechanism, and subject to common
control procedures. Although the sector contains
dierent types of organization, because of reporting and control requirements determined by the
Ontario Ministry of Education and Training they
face a relatively uniform control environment, at
least in terms of upward accountability. Access
was negotiated through the Deputy Minister, by
writing to the presidents of 35 publicly funded
colleges and universities, of whom 16 agreed to
participate. These presidents supplied lists of
senior administrators having signi®cant budgetary
responsibilities and 176 such people were selected
for the ®nal mailing list (with titles such as vicepresident for academic, administrative, ®nancial
or service operations, or dean of an academic
programme area). Such people typically reported
to the president either directly or via just one level
of management.
A substantial questionnaire, consisting of the
questions already outlined plus an approximately
equal number of additional items relating to other
budgetary behaviours was sent to each of the 176
selected respondents. A total of 134 replies (76%)
were obtained after some follow-up by telephone,
of which 127 were usable (72% net response rate).
These replies appeared to be representative of the
institutions sampled, with no evidence of nonresponse bias being apparent.1 A few questionnaires were not fully completed in all respects,
so the sample size is not 127 for all analyses, but it
487
always exceeds 120 and is reported for each set of
results. Individual responses were kept anonymous, but a report of the aggregate results was
prepared which was sent back to the presidents of
participating colleges. Summary statistics of the
variables relevant to this paper are given in
Table 1. These indicate a wide range of responses
and considerable variation between respondents
on the independent and dependent variables.
5. Results
The results will be presented as a series of comparisons between the ®ndings of this study, and
those found in the individual studies replicated, as
one of the main purposes of this paper is to draw
attention to the similarities and dierences that
occur.2 Similarities in ®ndings may indicate that
underlying results are persistent and stable features of behaviour across a range of organizations
and situations; dierences, by contrast, indicate
that the previous results should not be treated as
universal, and that more attention needs to be
paid to the circumstances under which greater
generalisation is likely to be possible. It should be
noted that the measures used are adjusted to
match those used in the original studies wherever
possible; this means that the reported results cannot be compared across studies, except where indicated, as dierent scales may be used in each case.
5.1. Brownell (1982)
This study examined the eect of budget
emphasis and participation on job satisfaction and
managerial performance, in particular hypothesising an interaction between budget emphasis and
1
For reasons of con®dentiality, it was not possible to determine response rates by institution; however, descriptive data
relating to respondents appeared to appropriately re¯ect the
characteristics of the sector.
2
A more coherent analysis of the results relating to the sector
will be provided in a future paper. In this paper, the variables
and measures used are dierent in each replication, to ensure
as a precise a replication as possible in each case. This explains
the dierences in the descriptive statistics reported in Tables 1,
2, 5 and 7.
488
D. Otley, R.M. Pollanen / Accounting, Organizations and Society 25 (2000) 483±496
participation. The sample was 48 managers holding cost centre responsibility in a US manufacturing company. Budget emphasis was measured
using Hopwood's method of comparing the ranking of the two relevant items from the 10-item
measure. Brownell then combined Hopwood's
Budget-Constrained [BC] and Budget-Pro®t [BP]
categories into a high budget emphasis category
(coded ÿ1), and his Pro®t Conscious [PC] and
Non-Accounting [NA] categories into a low budget emphasis category (coded +1). The descriptive
statistics for each study are shown in Table 2, with
this study's variables recoded to exactly match
those of Brownell (1982).3
There are two main dierences. First, there was
a considerably higher level of budgetary participation in this study compared with Brownell's.
Therefore analyses were re-run coding participation as high or low using both this study's mean
and Brownell's mean. Although this causes some
dierences in the results, the main interaction
eect is virtually unchanged. Second, the distribution of budget emphasis scores is rather dierent,
with only 14% [compared to 45% in Brownell
(1982)] being categorised as high budget emphasis.4 Nevertheless this still represents some 17
respondents, the same absolute number as Brownell, and does not aect the comparisons by
category.
The results of regressing the data in the interaction model are shown in Table 3, for both job
satisfaction and managerial performance. These
clearly indicate dierent results. The interaction
term is on the borderlines of statistical signi®cance
at the 5% level for job satisfaction in this study,
but is not signi®cant for managerial performance.
These results are thus quite dierent to those found
by Brownell (1982). With regard to job satisfaction,
3
In all cases, re-coding was performed to give exact comparibility between all the reported regression coecients. This goes
some way towards countering Southwood's (1978) assertion
that only the highest order interaction coecient is meaningful
because other coecients can change if Likert-scaled variables
are re-coded. We would contend that where measures of variables are precisely replicated, and the same coding is used in
two studies, then it is both valid and useful to compare all
reported coecients.
4
These dierences most likely re¯ect the dierent sector studied and also cultural dierences between countries.
the supposed incompatible combinations of participation and budget emphasis (high±high and low±
low) are associated with high job satisfaction and
the compatible combinations (low±high and high±
low) are associated with low job satisfaction. This
result is con®rmed by means tests and graphical
analysis, and by using two alternative measures of
budget emphasis. With regard to performance,
where Brownell found a signi®cant and positive
result, this study found a negative, albeit insignificant, result. If an alternative measure of budget
emphasis is used [combining all the accounting
styles (BC+BP+PC) into one category and nonaccounting into the other] the negative result
becomes signi®cant at the 2% level.
In summary, the results of this study are not consistent with Brownell's ®ndings; the relationships
found tend to be in the opposite direction, although
not always achieving statistical signi®cance. This is
unlikely to be an artefact of the dierent distribution of budget emphasis that was observed, nor of
the dierent absolute level of participation.
Rather, it has to be laid at the door of the dierent
context in which the study was conducted. What is
clear is that the Brownell (1982) result is not universal; the limits of its validity need further
exploration.
5.2. Brownell and Hirst (1986)
Part of Brownell and Hirst's (1986) study replicated Brownell (1982) but using job-related tension (JRT) as a dependent variable in place of
performance and job satisfaction. This will be
considered before the main part of their study that
added task uncertainty as a further explanatory
variable.
One relatively minor dierence between the two
studies is that budget emphasis is not measured by
the relative rankings of the top 3 items on the
Hopwood scale, but uses the top 4 items. This
procedure is followed here for our own results.5 A
direct comparison of the eect of budget emphasis
and participation on JRT with Brownell and Hirst
(1986) cannot be made as they do not present this
restricted equation (although it can be compared
with the 3-way interaction equation presented
later). However, the 2-way interaction coecient
489
D. Otley, R.M. Pollanen / Accounting, Organizations and Society 25 (2000) 483±496
Table 2
Descriptive statistics for continuous variablesÐcomparison with Brownell (1982)
Variables
No. of items
Sample size
Mean
SD
Theoretical range
Actual range
Min
Max
Min
Max
This study
Budget participation
Job satisfaction
Managerial performance
6
20
1
121
123
126
34.48
79.59
7.03
5.69
8.18
1.29
6
20
1
42
100
9
11
54
3
42
97
9
Brownell (1982)
Budget participation
Job satisfaction
Managerial performance
6
20
1
38
38
38
25.79
76.50
6.84
7.10
7.25
0.95
6
20
1
42
100
9
13
58
5
39
93
8
Note: Consistent with Brownell (1982), budget participation was reverse-coded and the total (rather than the average) score used.
Similarly, job satisfaction was reverse-coded and converted to a 20±100 scale. Managerial performance was re-coded to a 9-point scale.
Table 3
Results of regressing job satisfaction and managerial performance on budget emphasisÐcomparison with Brownell (1982)
Variables
Job satisfaction
Intercept
Budget participation
Budget emphasis
Participation
budget emphasis
This study
Brownell (1982)
Value
p
Value
78.4
ÿ0.41
1.17
ÿ0.28
0.0000
0.0050
0.2448
0.0515
76.5
ÿ0.19
ÿ0.79
0.05
R2=0.19
Managerial performance
Intercept
Budget participation
Budget emphasis
Participation
budget emphasis
7.08
0.03
ÿ0.06
ÿ0.04
0.0000
0.2302
0.7352
0.1533
R2=0.02
p