Manajemen | Fakultas Ekonomi Universitas Maritim Raja Ali Haji 08832320309599095

Journal of Education for Business

ISSN: 0883-2323 (Print) 1940-3356 (Online) Journal homepage: http://www.tandfonline.com/loi/vjeb20

Evaluating and Improving Student Achievement
in Business Programs: The Effective Use of
Standardized Assessment Tests
H. Tyrone Black & David L. Duhon
To cite this article: H. Tyrone Black & David L. Duhon (2003) Evaluating and Improving Student
Achievement in Business Programs: The Effective Use of Standardized Assessment Tests,
Journal of Education for Business, 79:2, 90-98, DOI: 10.1080/08832320309599095
To link to this article: http://dx.doi.org/10.1080/08832320309599095

Published online: 31 Mar 2010.

Submit your article to this journal

Article views: 97

View related articles

Citing articles: 23 View citing articles

Full Terms & Conditions of access and use can be found at
http://www.tandfonline.com/action/journalInformation?journalCode=vjeb20
Download by: [Universitas Maritim Raja Ali Haji]

Date: 12 January 2016, At: 23:00

Downloaded by [Universitas Maritim Raja Ali Haji] at 23:00 12 January 2016

Evaluating and Improving Student
Achievement in Business
Programs: The Effective Use of
Standardized Assessment Tests

zyxwv
zyxwvu

H. TYRONE BLACK
DAVID L. DUHON

The University of Southern Mississippi
Hattiesburg, Mississippi

0

ur purpose in writing this article is
to improve assessment of student
learning by (a) reviewing issues related
to the appropriateness of any particular
standardized test of student achievement and (b) providing examples of
how test results can be used for evaluation and performance improvement.
Allen and Bycio (1997), using data
from administration of the Educational
Testing Service’s (ETS) Major Field
Test in Business, addressed test appropriateness, and Mirchandani, Lynch,
and Hamilton (2001) compared the use
of grade point averages versus scores on
the ETS business field test for assessment of student learning. We use data
from administering the ETS business
field test to extend these studies, particularly in the area of applications of standardized test results. Before addressing

standardized tests and their uses, however, we provide some background on
educational accountability and learning
assessment.

Accountability Background
In the 1970s, a few state governments
and institutional governing boards
attempted to hold institutions of higher
learning more accountable for the funds
that they received. These embryonic
90

Journal of Education for Business

ABSTRACT. Having made “continuous improvement” the theme of its
accreditation process a decade ago,
the Association to Advance Collegiate
Schools of Business (AACSB) is
poised to make “assurance of learning” the current focus of the
facultyhtudent exchange. In this article, the authors discuss how to use

standardized assessment tests appropriately and constructively for compliance with this requirement. They provide an overview of accountability in
higher education and then focus on
assessing student achievement and
development. They use results from
the Educational Testing Service’s
Major Field Test in Business to illustrate verification of test reliability and
validity. Finally, they suggest ways
that standardized test results can be
used for evaluating and enhancing student performance.

accountability efforts included performance budgets, performance audits, and
program reviews. Folger (1977) underscored the output/effectiveness aspect of
these early activities in the following
descriptions: Performance budgets base
“at least some funding on the outcomes
or quality of the budgeted activity ”;
performance audits are “an assessment
of the effectiveness of an activity or
organization in achieving its goals and
objectives”; and program reviews assess

“the need for and the effectiveness of a
proposed or existing program” (pp. viii

and ix). Yet according to a 1976 survey
of state higher education agencies, only
six states were collecting and reporting
outcome measures in their budgeting
systems (Peterson, Erwin, & Wilson,
1977); similarly, Berdahl (1977) found
just seven states that were engaged in
intensive performance audits. Finally,
Barak (1977) reported that only a few
states were conducting comprehensive
program reviews.
Thus, by 1980, rather than focusing
on performance, academic assessment
still emphasized structure and
resources. Troutt (198 1) criticized
regional accrediting bodies for their
lack of attention to outcomes, arguing

that “accreditation standards suggest
checking the cumculum is more reliable
than checking the student” and adding
that “concerns about the outcomes of
the educational process generally
appear only as appendages to accreditation standards” (p. 47).
Nonetheless, accountability was about
to move to center stage. The catalyst was
a series of studies and reports pointing to
a deterioration in educational quality.
Perhaps the most influential of these critiques was A Nation at Risk: The Imperative for Educational Reform (National
Commission on Excellence in Education, 19831, which recited evidence of a

zyxwv
zyxw
zyxw

Downloaded by [Universitas Maritim Raja Ali Haji] at 23:00 12 January 2016

zyxw

zyxwvutsrq
zyxwvutsrq

decline in the abilities of high school
and college graduates. In the meantime,
state governments, institutional governing boards, education commissions, and
the public began asking for evidence
that educational dollars were producing
qualified students and worthwhile services. Responding to these calls for
more accountability, U.S. Secretary of
Labor William Bennett issued an executive order in 1988 requiring all federally
approved regional accreditation organizations to include evidence of institutional effectiveness in their accreditation criteria (Palomba & Banta, 1999).
Consequently, the six regional accrediting agencies began giving more weight
to outcomes in their accreditation
reviews. In this regard, the Southern
Association of Colleges and Schools
was the initial leader, but Kimmell,
Marquette, and Olsen (1998) identified
the North Central Association of Colleges and Schools as the association
most insistent that the candidate institution document student achievement and

overall institutional effectiveness.
Likewise, at the program level many
accrediting agencies have reoriented
standards to focus more on results and
accompanying processes, although
structure still plays a significant role in
some areas such as curriculum. The
AACSB began a review of business
and accounting standards in the late
1980s in response to criticisms that
business graduates lacked writing,
interpersonal, and other skills critical
to career success. Addressing such criticisms, an AACSB-commissioned
study, Management Education and

input to learning outcome. It asks for
evidence, rather than intent” (p. 5).

InstitutionaVProgram
Effectlveness

Nichols and Nichols (2000b) distinguished between strategic planning and
institutional effectiveness. Strategic
planning focuses on implementing
action plans designed to align organizational structure, resources, activities,
and related processes with institutional
mission, vision, and strategic goals. On
the other hand, institutional efjrectiveness is concerned with the extent to
which intended outcomes are being
achieved. Thus, a strategic plan may
include an objective of obtaining additional resources for altering a particular
program, and success would be judged
by whether the resources were secured
and used to modify the program.
Assuming that funds were obtained and
the program was modified, institutional
effectiveness would involve evaluation
of success according to evidence of
improved program outcomes.
The following description of educational program evaluation within the

institutional effectiveness framework is
an adaptation of the Nichols and
Nichols (2000a) model:
The institution’s mission, vision, and
goals are the basis for a program’s intended outcomes, which should concentrate
on student achievement and development.
Programs must decide on means of
assessment, including the data to be gathered and how to process it, as well as criteria for success. Program assessment
should focus on concrete, verifiable
results, such as how much students have
learned and whether they find employment upon graduation. Results, then,
determine program effectiveness, thereby
indicating whether changes are needed to
achieve intended outcomes and better
conformance with mission, vision, and
goals. Finally, institutional effectiveness
structure, processes, and results should be
reviewed regularly with the aim of continuous improvement.

literature deals with institutional and

program accountability with related
work on measuring and improving student achievement. Assessment manuals
by Nichols (1995) and Palomba and
Banta (1999) contain comprehensive
institutional effectiveness models with
detailed implementation guidelines. At
the program level, works by Barak and
Mets (1995) and Nichols and Nichols
(2000a) outline procedures for effective
evaluation. In the business area,
accounting appears to be the program
most involved in assessment, possibly
because of the outgrowth of experience
with the certified public accountant
(CPA) examination. DeMong, Lindgren, and Perry (1994) examined issues
related to the design of assessment
processes and presented a matrix of
techniques for assessing student outcomes. Ackers and Giacomino (1997)
discussed details of the design and
implementation of their school’s system
of accounting program assessment, and
Apostolou (1999) argued that accounting program assessment can be
improved by using proven practices
from other disciplines. A case study by
Stivers, Campbell, and Hermanson
(2000) demonstrated how their accounting program has been dealing with
design and implementation issues and
the use of assessment results for program improvements.
For student achievement assessment
and development, works by Erwin
(1991) and Halpern (1987) contain
information on design and implementation. In addition, the following studies
are examples of ways to use student
achievement data to strengthen programs: Staik and Rogers (1996), who
discussed how results on the ETS Major
Field Test in Psychology led to curriculum modification; Magruder and Young
(1996), who reported general-education
curriculum changes based on outcomes
on standardized examinations; Hanson
(1996), who described a regression
model used to place freshman students
in an appropriate introductory writing
class; Young (1996), who explained
how her institution has used standardized examinations administered to
seniors in all majors to make program
changes; and Franklin and Tolbert
(1993, who described how assessment

zyxwvutsrqpo

Development: Drift o r Thrust Into the
21st Century? (Porter & McKibbin,

1988), made recommendations for
changes in the content and delivery of
management education, thereby setting
the stage for new AACSB accreditation
standards. Subsequently, in 1992
AACSB replaced resource-oriented
standards for business program accreditation with requirements emphasizing
“continuous improvement” (AACSB,
1994-1995). The AACSB (2002) has
drafted new accreditation standards
with a curriculum component that
would emphasize “assurance of learning.” According to the draft, “This is a
shift of perspective from structural

Hence, true program accountabilitywithin the institutional effectiveness framework focuses on results and improvements, particularly in terms of student
achievement and development rather
than level of resources or structure.
A considerable body of assessment

November/December 2003

91

Downloaded by [Universitas Maritim Raja Ali Haji] at 23:00 12 January 2016

results have been the catalyst for
improvements in their nursing program.
These examples may give the impression that accountability is the modus
operandi of higher education. However,
assessment is still not the rule. For
instance, in a recent survey of accounting administrators, Kimmell et al.
(1998) reported that only 42% indicated
the existence of a comprehensive institutional assessment program at their
universities. Even more focused learning evaluation is far from universal, as
demonstrated by the following assessment rates found by Kimmell et al.
among the institutions in their survey:
tests of general education, 46%; major
field tests, 36%; tests of reading, writing, and/or listening ability, 21%; and
critical-thinking tests, 12%.
Even when assessment occurs, the
results all too frequently are ignored.
Franklin and Tolbert (1995, p. 37)
argued: “Traditionally, assessment data
have primarily been used to report program outcomes for accrediting bodies.
Unfortunately these valuable data have
been underutilized in curriculum development and individual student enrichment.” To some extent, the relatively few
applications reflect the considerable
investment of time and effort required for
effective use of assessment results. Moreover, faculty members are often skeptical
about the value of accountability, especially when they observe the extent to
which organizational interest in assessment waxes and wanes with accreditation
and other review cycles. In this article,
we attempt to help readers use standardized tests appropriately for gathering student achievement data that become the
basis for making improvements.

next” (Ball State University, 1992, section IV). Standardized tests can be
developed internally or acquired from
testing organizations. Local tests
require considerable faculty effort and
other resources for test development,
scoring, reporting, and improving.
Nonetheless, local instruments can be
tailored to a specific course or program
so that scores accurately reflect the
extent to which learning objectives are
being met. On the other hand, the testing organizations that offer standardized
tests generally provide valuable comparative data on other schools; moreover, external tests usually are developed by test experts who thoroughly
address desirable test properties such as
reliability and validity (Ewell, 1987;
Nichols & Nichols, 2000a). Consequently, the standardized tests offered
by testing organizations have become a
popular vehicle for determining whether
learning objectives are being met.

course. Any student taking the ETS
major field test who scored at the national 50th percentile or better was given an
extra-point bonus, which was used for
calculating the final course grade. If this
or a similar incentive is not used, some
students will not take the test seriously
and the results therefore may be misleading (Allen & Bycio, 1997).
Students who took the test typically
were in their last or next-to-last semester, and the majority of examinees had
completed the business core or were
currently enrolled in their last core
courses. We did not include in our study
students who had more than 6 hours of
core course work remaining after the
end of the semester. In addition, we
wanted to determine the association
between ETS test scores and performance on the ACT and therefore
excluded students who had not taken the
ACT. As a result, our analysis included
297 students out of the 456 who took
the ETS major field test during the
study period.

zyxwvut

The ETS Major Field Test in Business
The Educational Testing Service
describes the ETS Major Field Test in
Business as a “test of achievement” that
provides a measure of the “mastery of
the concepts, principles, and knowledge”
within the student’s major field of study,
in this case business (Educational Testing Service, 1997b, p. 1). The business
field test is a 120-item, 2-hour multiplechoice examination with possible scores
ranging from 120 to 200 and with associated percentiles. The subject matter
covered is typical of the “business core,”
or what was referred to as the “common
body of knowledge” in pre-1992
AACSB accreditation standards. Thus,
the test’s eight components are accounting, economics, management, quantitative business analysis, finance, marketing, legal and social environment, and
international business. To collect the data
reported in this study, we used version I
of the ETS business field test, which
more than 300 institutions used to test
over 45,000 examinees (Educational
Testing Service, 1997a). The major field
tests are revised on a 5-year cycle.
We collected data during 3 semesters
in 1996-1997 when the ETS Major Field
Test in Business was administered to 456
seniors enrolled in a senior business-core

Reliability Issues
Schools using standardized test
scores for evaluation and decision making should determine that the scores are
reliable and valid. Test scores are reliable when they provide consistent (stable) rank-order results. Reliable tests,
among other desirable properties, contain questions that are unambiguous.
Although there are various methods of
measuring test reliability, ETS employs
an indicator of internal reliability
known as Kuder-Richardson Formula
20. ETS’s version I business field test
reliability coefficient is 0.89. According
to Ebel and Frisbie (1991), schools can
make decisions about individual students when the reliability coefficient is
0.85 and above and can make decisions
or draw conclusions about groups when
the coefficient is 0.65 and above.

zyxwvuts
zyxwvutsrq
zyxwvutsrqp
zy

Standardized Achievement
Tests-The ETS Major Field Test
in Business
Standardized Tests

Standardized tests can be an effective
tool for assessing student achievement:
“A standardized test is one in which the
initial construction, as well as conditions for administration and scoring,
have a uniform procedure so that the
scores may be interpreted in a consistent
manner from one administration to the

92

Journal of Education for Business

Validity Issues
Test-score reliability is a necessary
but not sufficient condition for validity.
Validity exists when scores on the
instrument (a) reflect achievement on
the dimensions that the school wishes to
evaluate and (b) are used appropriately

Downloaded by [Universitas Maritim Raja Ali Haji] at 23:00 12 January 2016

zyxwvutsrqp
zy
zyxwvutsrqpo

for evaluation and decision making
(Ebel & Frisbie, 1991). Validity has several dimensions, including content and
context validity. For a particular program, content validity exists when test
coverage is consistent with the curriculum, whereas context validity addresses
the extent to which scores are correlated
logically with other variables expected
to be associated with student achievement. Test reliability and validity were
addressed by Herring and Izard (1992),
who analyzed scores from the AICPA
Level II Achievement Test, and by Allen
and Bycio (1997), who examined the
ETS Major Field Test in Business.
ETS addresses content validity by
using faculty committees and expert
consultants to construct and evaluate the
items on the business field test (Educational Testing Service, 1997a); nevertheless, any school considering the
instrument should thoroughly review
content to assure that scores reflect the
extent to which learning objectives are
being met. Our faculty members
reviewed the business field test and
determined that scores would measure
achievement appropriately in our business core curriculum. To enhance content validity, schools may include up to
50 locally written questions that ETS
will score and report; however, it is the
school’s responsibility to assure the reliability of scores on these items.
Criterion validity is the extent to
which scores on the standardized test
are correlated with other variables that
the school expects to be associated with
test performance (Ebel & Frisbie,
1991). Such expectations are formed to
a large extent by the school’s previous
experience in assessing the performance
of its students. Furthermore, for a particular standardized test, other users of
the test may have reported their findings
related to criterion validity, and this
information can be useful in forming
expectations about variables most likely
to be associated with student performance. Finally, ETS reports collective
data that also can be helpful in assessing
criterion validity. Based on these considerations, Table 1 lists the variables
analyzed in our consideration of criterion validity. The first item in the table is
the dependent variable, ETS Major
Field Test in Business scores, and the

remaining items are variables that we
expected to be related to ETS business
test scores for the following reasons:

Grade point averages (GPAs).GPA
is an indicator of student ability and
motivation and therefore should be
associated positively with ETS test performance. We evaluated three specific
grade point averages: (a) business core
GPA (given the similarity of the subject
matter covered by the ETS Major Field
Test in Business and our school’s business core, this GPA should be and was
the one most closely associated with
ETS test performance), (b) overall GPA,
and (c) a 9-hour GPA that included
grades in Principles of Economics I and
I1 and Introduction to Financial
Accounting. We chose this 9-hour GPA
because we wanted to develop a predictive model of ETS test performance (see
the applications section of this article).
Allen and Bycio (1997) considered several GPAs in their study of ETS business field test criterion validity and
found the strongest association between
business core GPAs and ETS test
scores. Mirchandani, Lynch, and Hamilton (2001) found the strongest association between a group of quantitative
courses and ETS test scores.
ACT scores. ACT scores are an
indication of academic ability and preparedness and should be associated positively with ETS test scores (we analyzed composite scores as well as scores
on the English and mathematics components). Allen and Bycio found a positive
association between SAT and ETS test

scores, as did Mirchandani, Lynch, and
Hamilton.
Gender. Because ETS reports that
the national mean score of men is 3.41
points higher than women’s mean
scores, a similar pattern would be
expected for our students. Moreover, the
Allen and Bycio and the Mirchandani,
Lynch, and Hamilton studies reported
that male students performed better than
female students. (We could not explain
the disparity between male and female
test performance by controlling for ACT
scores and GPA differences. We surmise
that work experience may be a contributing factor; however, we did not
have detailed work-related information
to test this hypothesis.)
Age. Given our experience with
nontraditional students, we believe that
age is a proxy for work experience and
thus should be positively related to ETS
test performance.
Majorfield of study. Given the differences in coverage of business topics
in course work specific to our various
majors, performance on the ETS major
field test should be influenced by a student’s major. Allen and Bycio reported a
statistically significant variation of ETS
test scores among the various majors
offered by their school.

zyxwvut
zyx
In Table 2, we provide descriptive
data for all 297 students in our study, in
a format that follows Allen and Bycio
(1997). In addition, we selected management majors ( n = 38) from among
our six undergraduate programs to illustrate how additional descriptive infor-

TABLE 1. Variables Analyzed
Variable
ETS Major Field Test in Business scores
GPAs (business core, 9-hour: and overall)
ACT scores (composite, English, math)
Age
Gender
Major

Explanation
Indicator of student learning in the
business core
Indicator of student motivatiodability
Indicator of student intellectual ability/
preparedness
Indicator of student experiencehaturity
Dummy variable: Men = 1
Dummy variable: Management
majors = 1

zyxw

aThe 9-hour GPA includes grades in Principles of Economics I and I1 and Inrroduction to Financial Accounting.

NovembedDecember 2003

93

zyxw
zyxw
zyx
zy
zyxw
zyxwvutsrq
zyxw

TABLE 2. Descriptive Statistics: Correlations, Means, and Standard Deviations (All: N = 297; Mgt.: n = 38)

Variable

Group

ETS test

All
M%.
All
Mgt.

ACT Corn.
ACT Eng.

All
Mgt.

ACT Mat.

All
Mgt.
All
Mgt.
All
Mgt.
All
Mgt.
All
Mgt.
All
Mgt.
All
Mgt.

B. core GPA

Downloaded by [Universitas Maritim Raja Ali Haji] at 23:00 12 January 2016

9-hr.GPA
Overall GPA

Age
Gender
(Men = 1)
Major
(Mgt. = 1)

B. core

Eng.

4
ACT
Mat.

GPA

GPA

r

r

r

r

+0.54
+0.63
+0.86
+0.86

+0.40
+0.16*
+0.77
+0.46
+0.58
+0.29*

2
ACT
Corn.

3
ACT

r

+0.62
+0.51

5

+0.63

+0.58

+0.54
+0.56
+0.49
+0.59
+0.45
+0.22*

6
7
9-hr. Overall

8

9

10

GPA

Age

Gender

Major

r

r

r

r

+0.23
+0.04*
-0.19
-0.22”
-0.21
-0.07*
-0.21
+0.21*

+ 0.16

+0.53 +0.56
+0.34 +0.53
+0.46 +0.53
+0.40
+0.66
+0.40 +0.48
+0.36
+0.60
+0.42 +0.43
+0.31* +0.21
+0.79 +0.91
+0.62
+0.92
+0.66
+0.53

+0.21
+0.19*
+0.05* -0.09*
+0.23*
-0.06* -0.09*
+0.06*
+o.o1* -0.14
+0.20*
+0.15
-0.08* -0.14
+0.19* +0.07*
+0.09* -0.07* -0.15
-0.04” +0.03*
+0.11* -0.11*
-0.04*
+0.29* +0.05*
+0.04* +0.06*
+0.29*
-0.02”

M

SD

154.30
147.55
21.38
20.45
21.47
20.39
20.46
18.84
2.82
2.61
2.87
2.57
2.86
2.81
24.45
23.68
0.47
0.45
0.13

12.57
11.12
3.90
3.11
4.48
3.80
4.57
3.40
0.56
0.49
0.75
0.75
0.49
0.43
4.50
3.23
0.50
0.50
0.33

-

-

Note. Figures for all majors are in standard font; figures for management majors are in bold font. All correlations are significant at p < .01 except those
indicated with an asterisk (*).

mation can be generated for any major
area of study. Furthermore, in the multivariate analysis below, we estimated the
impact of majoring in this area on ETS
test scores. (Other majors also could be
singled out for similar treatment.) In
Table 2, the mean ETS test score is
154.30 for all majors (46th percentile
for ETS student-total-score distribution)
and 147.55 for management majors
(27th percentile). Compared with all
majors, management students had the
lowest mean ACT scores and grade
point averages and were slightly
younger. For all majors, the strongest
ETS test score correlations were with
business-core GPA (r = +0.63) and with
the ACT composite score (r = +0.62).
All ETS test score correlation coefficients listed for “All” and “Mgt.” in
Table 2 (first two data rows) were statistically significant at the .01 level except
for the mathematics ACT, age, and gender correlations for management
majors.
Allen and Bycio (1 997) used regression analysis to study the association of
grade point averages and SAT scores
94

Journal of Education for Business

with ETS business field test scores;
however, they employed analysis of
variance when considering the association of gender and major with ETS test
scores. Mirchandani, Lynch, and Hamilton (2001) also used a regression model
that explained 37% of the variation in
the ETS field test scores of their native
students for whom they had SAT scores.
Our model explains 58% of the variation in ETS scores (see the RZ adjusted
in Table 3 below) and includes the following variables: business core GPA,
composite ACT score, age, gender, and
major. (We treated gender and major as
dummy variables, with men = 1 and
management majors = 1.) The multivariate regression results in Table 3
reveal the following:
A business core GPA that is one
point higher is associated with an ETS
test score that is 7.49 points higher.
A composite ACT score that is one
point higher is associated with an ETS
test score that is 1.51 points higher.
A l-year positive age differential is
associated with an ETS test score that is
0.71 points higher.

z

Men have an ETS test-score advantage of 3.79 points over women.
After controlling for GPA, ACT
scores, age, and gender, we found that
management majors were at a 3.57point disadvantage relative to other
business majors on the ETS test.
A high correlation (at least +/-0.70)
between any pair of independent variables would indicate the presence of
collinearity and its attendant statistical
distortions (Anderson, Sweeney, &
Williams, 2002). Because the correlation of +0.54 between ACT composite
scores and business core GPAs is the
highest correlation for any pair of independent variables (see Table 2), we
were satisfied that our regression coefficients provide credible, useful information. In addition, the value of R2
adjusted (0.58) in Table 3 and the high
significance levels of R2 and the independent-variable coefficients indicate
that ETS Major Field Test in Business
scores have criterion validity for our
program. Having addressed the issues
of score reliability and validity satisfac-

zyxwvuts
zyxwvuts
zyxwvut
zyxwvuts
zyxwvut
1

I

TABLE 3. Multivariate Regression Results ( N = 297)

Variable

Coefficient

t value

Constant

82.04
7.49
1.51
0.71
3.79
-3.57

20.12
7.14

Business core GPA

ACT composite score
Age
Gender (men = 1; women = 0)
Major (mgt. = 1; other = 0)
Total model R = 0.77.
Total model adjusted R 2 = 0.58.

10.00
6.41

4.02
-2.54

P’t

.oo
.oo
.oo
.oo
.oo

.o1

Total model F value = 84.15.
Total model p > F .OO.

1

I

Downloaded by [Universitas Maritim Raja Ali Haji] at 23:00 12 January 2016

TABLE 4. Standardized Test Score Uses

Learning outcomes information
Program
Student
assessment
assessment
1. Program

7. Examinee
comparisons
8. Course grade or
graduation
requirements
9. Resumes,
employment
applications,
and graduate
school admission
applications

Continuous improvement
Program
Student
development
development
10. Curriculum
1 1. Pedagogy
12. Resource
allocation

13. Identification
of at-risk
students
14. Proficiency
tests and
related
requirements
15. Additional
course
requirements
16. Work-study
participation
17. Mentoring

sample of such schools (Educational
Testing Service, 1997a). Whatever the
relative program performance level for
any particular administration of the test,
longitudinal test results are also important in evaluating student achievement
progress (see Table 4, item 2).
A school can use standardized tests
in accreditation and other program
reviews to relate learning outcomes,
thereby providing information on the
success of its students in achieving
learning objectives (see Table 4, item
3). For institutions that use multifaceted performance measures to assess
unit productivity, student-learning outcomes and improvements can serve as
unit performance indicators (see Table
4, item 4).
Schools using the ETS Major Field
Test in Business receive percentile
scores for each of the eight subject-matter areas covered by the test. In Table 5,
we show the mean national percentile
subject area ratings for our school for the
3 semesters covered by this study. This
information can be useful in evaluating
program initiatives (see Table 4, item 5 ) .
For instance, international business is a
high priority for our college: We (a)
offer an international business major, (b)
require all students to take an international economics course, (c) have an
international component in several core
courses, (d) require an international
course in most majors, (e) offer an
extensive array of study-abroad courses
and programs, and (f) have administered
three federal grants for enhancing international programming. The relatively
good performance in the international
business area, as reported in Table 5,
indicates that this particular initiative
has produced positive results.
Finally, for cases in which standardized test performance is relatively
strong or significant improvement is
occurring, the school may want to use
the results for public relations (e.g., in
newsletters and press releases; see Table
4, item 6).

zyxwvutsrqp
zyxwvut

comparisons
2. Achievement
trends
3. Program reviews
4. unit
performance
5. Initiatives
6. Public relations

torily, we conclude that schools can
turn to using standardized test results
for evaluation and enhancement.

Using Student Achievement Test
Results

In Table 4, we classify uses of standardized test results as either learning
outcomes information or continuous
improvement. We further divide learning information uses into program
assessment and student assessment, and
continuous improvement applications
into program development and student
development. The learning information
provided by standardized tests can indicate if actual learning outcomes are consistent with intended outcomes. Continuous improvement involves using test
results to enhance student achievement
by making program changes and inter-

vening in the individual student learning
process.

Uses of Learning Outcomes
Information

Program assessment, The percentile
level of the mean score of the cohort
who took the test (see Table 4, item 1) is
probably the most useful statistical comparison among groups of test takers. For
example, a cohort mean score at the
60th percentile indicates that this cohort
performed as well as or better than 60%
of the groups that have taken the test.
ETS provides percentile levels for comparing group (and individual) scores but
maintains that the percentiles are not
“norms” because the schools that participate in the ETS testing process are neither the entire population of schools
offering such programs nor a random

Student assessment evaluation. Comparisons can be made between the score of
a particular examinee and the mean
score of all students who have taken the
test (see Table 4, item 7). ETS provides
each examinee a percentile level indicatNovember/December 2003

95

zyxwvutsr
zyxwvutsrqp
such means as a discipline-focused lecture series.

TABLE 5. Subject Area Mean Percentiles

Subject area

Mean percentile ranking
(3 semesters: 1996-1997)

Accounting
Economics
Management
Quantitative business analysis
Finance
Marketing
LegaVsocial environment
International business

41.5
39.0
30.0
42.5
32.3
31.0
40.6
53.1

zyxwvuts
zyxwvuts
zyxwvuts
zyxwvut
zyxwvut

Downloaded by [Universitas Maritim Raja Ali Haji] at 23:00 12 January 2016

TABLE 6. Multivariate Regression Results ( N = 297)
Variable

Coefficient

t value

P>t

87.21
1.35
5.5 1
0.8 1
4.91

20.84
10.62
7.37
6.97
4.94

.oo
.oo
.oo
.oo
.oo

Constant
ACT English score
9-hour GPA
Age
Gender (men = 1; women = 0)
Total model R = 0.73.
Total model adjusted R 2 = 0.53.

Total model F value = 83.48.
Total model p > F: .OO.

ing the examinee’s standing in the array
of all students who have taken the test.
Schools can use scores on standardized tests in determining course grades
or as a condition for graduation (see
Table 4, item 8). For example, our college currently is using business field test
scores as a component of the grade in
the business policy/strategy course.
Schools could make further use of
scores by requiring a minimum standard
of performance-for example, a score
within the 25th percentile-as a condition of graduation. Livingston and
Zieky (1982) provided guidance on
methods for using standardized test
scores in making decisions on matters
such as graduation, course completion,
the need for remedial instruction, and
advanced placement.
Students should be made aware that
evidence of good performance on a
standardized test may be included on a
resume, employment application, or
graduate school admissions application
(see Table 4, item 9). Particularly for a
student whose grade point average is
not outstanding, a high score on the
ETS Major Field Test in Business or
96

Student development. Information from
administering standardized achievement
tests can be used for development of a
predictive model of expected learning.
Predictions from such a model can identify at-risk students whose characteristics are associated with a low level of
student learning (see Table 4, item 13).
In Table 6, we provide such a model: We
developed it to predict ETS test scores
(and the level of learning indicated by
the scores) when students have completed lower-division course work. The
independent variables are English ACT
score, 9-hour GPA for Introduction to
Financial Accounting and Principles of
Economics I and 11, age, and gender. We
used the English ACT score instead of
the composite ACT score because the
composite score does not lend itself to
identification of deficiencies that can be
addressed with specific course work.
(We also evaluated the math ACT score
as a predictor of learning in the business
core, but its coefficient was not statistically significant.) The information in
Table 6 indicates that a 19-year-old
sophomore male student with an English
ACT score of 18 and a 9-hour GPA of
2.00 has an expected ETS test score of
142.83 (18th percentile), whereas a 22year-old sophomore female student with
an English ACT score of 25 and a 9-hour
GPA of 3.61 has a predicted score of
159.00 (59th percentile). When predicted ETS test scores are at or below a certain percentile (e.g., the 25th), students
might be required or advised to take
remedial courses or be involved in specified activities either as a condition of
admission to upper-division business
courses or as a postadmission program
requirement. For example, these individuals could be required or encouraged to
take the following steps:

other standardized test can shed a more
favorable light on his or her ability or
learning achievement.
Uses for Continuous Improvement
Program development. Scores on standardized tests can be a basis for curriculum changes (see Table 4, item 10). The
data in Table 5 indicate a wide range of
relative performance among the eight
ETS business subject areas at our school
and suggest priorities for considering
curriculum changes. Course content may
be modified or additional course work
may be added in areas with relatively
weak performance. As for pedagogy (see
Table 4, item 1 l), areas with weak learning outcomes may benefit from experimentation with more cases, inclusion of
more team work, and requirement of
more written assignments and oral presentations. Last, schools may want to use
test results as a basis for resource allocation (see Table 4, item 12) by rewarding
faculty members and departments whose
students do well or improve. Alternatively, resources can be directed at improving performance in a particular area by

Take an English proficiency examination, followed by additional course
work for those whose proficiency test
results are below a specified standard
(Table 4, item 14);
Retake any course in the 9-hour
accounting/economics block until a
minimum of a C grade is achieved
(Table 4, item 15; in our study group

zyxwvutsr

Journal of Education for Business

Downloaded by [Universitas Maritim Raja Ali Haji] at 23:00 12 January 2016

zyxwvutsrqp
zyxwvutsr
zyxwvuts
zyxwvuts
zyxw
zyxwvutsrq

there were 88 students with ETS test
scores at or below the 25th percentile
who made a total of 25 Ds in the 9
hours, whereas the 63 students at or
above the 75th percentile had no Ds
whatsoever):
Participate in work-study programs
to better integrate classroom learning and
real-world applications(Table 4, item 16;
as noted earlier, we surmised that the
positive association of age/gender and
ETS test scores is related to exposure to
the business environment): and
Work with mentors such as upperdivision students, designated faculty
members, or managers who would help
develop good study habits and attitudes
conducive to success (Table 4,item 17).
Of course, all of the actions listed above
could be undertaken independently of
predicted performance on the ETS
examination. However, the advantage of
linking the remediation process to the
ETS test is that actual ETS test scores
can be compared with earlier predictions to help determine if the remediation program is beneficial.

Summary and Conclusions

More accountabilityfor higher education in the United States means that institutions and programs are being asked to
demonstrate that resources are generating intended outcomes. Accountability
has therefore shifted the focus from program inputs and structure to results, particularly student achievement. In this
article, we have outlined steps for using
standardized student-achievement test
scores to draw conclusions about program and student performance and to
make improvements. The process begins
with careful test development or selection, including the assurance that scores
are reliable and are valid for their intended purposes.
There are many uses of standardized
achievement test results. Informational
uses include making comparisons of
individual examinees and groups of test
takers, as well as documenting student
achievement for program reviews. Test
scores also can be used by schools for
course and graduation requirements or
by students on resumes and applications. In the continuous improvement
area, test results can be used as a cata-

lyst for curriculum or pedagogy
changes. Schools also can develop predictive models to identify at-risk students whose curricula and experiences
can be enriched to improve learning
outcomes.
In conclusion, assessment of student
achievement as a component of a system
of accountability can pay dividends in
terms of better programs and more qualified students.Although such results cannot be expected if the school approaches
assessment haphazardly or unenthusiastically, we believe that addressing the
issues outlined in this article will make
assessment of student achievement a
more rewarding endeavor.
REFERENCES

AACSB-The International Association for Management Education. (1994-1995). Achieving
quality and continuous impmvement through
self-evaluation and peer review: Standards for
accreditation, business administration and
accounting. St. Louis, MO: Author.
AACSB-The Association to Advance Collegiate
Schools of Business. (2002). Eligibility pmcedures and standards for business accreditation
(second working draft). St. Louis, MO: Author.
Ackers, M. D., & Giacomino, D. E. (1997).
Designing and implementing an accounting
assessment program. Issues in Accounting Education, 12, 259-280.
Allen, J. S., & Bycio, P. (1997). An evaluation of
the Educational Testing Service Major Field
Achievement Test in Business. Journal of
Accounting Education, 15, 503-514.
Anderson, D. R., Sweeney, D. J., & Williams, T.
A. (2002). Statistics for business and economics (8th ed.). Cincinnati: South-Western.
Apostolou, B. A. (1999). Outcome assessment.
Issues in Accounting Education, 14, 177-197.
Ball State University, Offices of Academic
Assessment and Institutional Research. (1992).
Assessment workbook. Muncie, IN: Ball State
University.
Barak, R. J. (1977). Program reviews by statewide
higher education agencies. New Directions for
Institutional Research, N(4). 67-90.
Barak, R. J., & Mets, L. A. (Eds.). (1995). Using
academic program review. San Francisco:
Jossey-Bass.
Berdahl, R. 0. (1977). Legislative program evaluation. New Directions for Institutional
Research, IV(4). 35-66.
DeMong, R. F., Lindgren, J. H., Jr., & Perry, S. E.
(1994). Designing an assessment program for
accounting. Issues in Accounting Education, 9,
11-27.
Ebel, R. L., & Frisbie, D. A. (1991). Essentials of
educational measurement (5th ed.). Englewood
Cliffs, NJ: Prentice-Hall.
Educational Testing Service. (1997a). Comparative data guide and descriptions of reportsMajorfield tests. Princeton, NJ: Author.
Educational Testing Service. (1997b). Test administration manual-Major field tests. Princeton,
NJ: Author.
Erwin, T. D. (1991). Assessing student learning
and development: A guide to the principles,

goals, and methods of determining college outcomes. San Francisco: Jossey-Bass.
Ewell, P. T.(1987). Establishing a campus-based
assessment program. In D. F. Halpern (Ed.).
Student outcomes assessment: What institutions
stand to gain (pp. 9-24). San Francisco: JosseyBass.
Folger, J. K. (Ed.). (1977). Increasing the public
accountability of higher education. New Directions for Institutional Research, IV(4). San
Francisco: Jossey-Bass.
Franklin, D. A,, & Tolbert, R. P. (1995). Maximizing the use of pre-existing assessment results for
improvement in higher education. Assessment
and Evaluation in Higher Education, 7, 37-44,
Halpern, D. F. (Ed.). (1987). Studenr outcomes
assessment: What institutions stand to gain.
San Francisco: Jossey-Bass.
Hanson, L. K. (1996). Placement for success: The
writing program placementkredit portfolio. In
T .W. Banta, J. P. Lund, K. E. Black, & F. W.
Oblander (Eds.), Assessment in pracrice:
Putting principles to work on college campuses
(pp. 199-203). San Francisco: Jossey-Bass.
Herring, H. C. 111, & Izard, C. D. (1992). Outcomes assessment of accounting majors. Issues
in Accounting Education, 7, 1-17.
Kimmell, S. L., Marquette, R. P., & Olsen. D. H.
(1998). Outcomes assessment programs: Historical perspective and state of the art. Issues in
Accounting Education, 13, 85 1-868.
Lawrence, J. K., & Solomon, L. C. (1981). Reputational ratings and undergraduate education
assessment. New Directions for Institutional
Research, VIlI(l), 1-14.
Livingston, S . A., & Zieky, M. (1982). Passing
scores, a manual for setting standards of performance on educational and occupational tests.
Princeton, NJ: Educational Testing Service.
Magruder, W. J., & Young, C. C. (1996). Valueadded talent development in general education.
In T.W. Banta, J. P. Lund, K. E. Black, & F. W.
Oblander (Eds.), Assessment in practice:
Putting principles to work on college campuses
(pp. 169-171). San Francisco: Jossey-Bass.
Mirchandani, D., Lynch, R., & Hamilton, D.
(2001). Using the ETS Major Field Test in
Business: Implications for assessment. Journal
of Education for Business, 77(I), 51-56.
National Commission on Excellence in Education. (1983). A nation at risk: The imperative
for educational reform. Washington, DC: U.S.
Government Printing Oftice.
Nichols, J. 0. (1995). A practitioner's handbook
for institutional effectiveness and student outcomes assessment implementation (3rd ed.).
New York: Agathon Press.
Nichols, J. 0..& Nichols, K. W. (2000a). The
departmental guide and record book for student
outcomes assessment and institutional effectiveness. New York: Agathon Press.
Nichols, J. O., & Nichols, K. W. (2000b). The
department head's guide to assessment implementation in administrative and educational
support units. New York: Agathon Press.
Palomba, C. A., & Banta, T.W. (1999). Assessment essentials. San Francisco: Jossey-Bass.
Peterson, M. W., Erwin, J. M., & Wilson, R.
(1977). State-level performance budgeting.
New Directions for Institutional Research,
IV(4). 1-34.
Porter, L. W., &. McKibbin, L. E. (1988). Management education and development: Drifr or
thrust into the 2Ist century? New York:
McGraw-Hill.
Staik, I. M., & Rogers, J. S. (1996). Listening to
your students. In T. W. Banta, J. P. Lund, K. E.

zy

NoveinbedDecember 2003

97

zyxwvutsrqpo
zy
zyxwvu
zyxwvut
zyxw
ing: Design, implementation, and reflection.
Issues in Accounting Education, 15, 553-581.
Troutt, W. E. (1981). Relationships between
regional accrediting association standards and
educational quality. New Directions for Institutional Research, VIll(l), 45-60.

Young, C. C. (1996). Triangulated assessment of
the major. In T. W. Banta, J. P. Lund, K. E.
Black, & F. W. Oblander (Eds.), Assessment in
practice: Putting principles to work on college
campuses (pp. 101-104). San Francisco:
Jossey-Bass.

Downloaded by [Universitas Maritim Raja Ali Haji] at 23:00 12 January 2016

Black, and F. W. Oblander (Eds.), Assessmenr
in practice: Putting principles to work on college campuses (pp. 132-135). San Francisco:
Jossey-Bass.
Stivers, B. P., Campbell, J. E., & Hermanson, H.
M. (2000).An assessment program for account-

98

zyxwvutsrqpo
zyxwvutsr
Journal of Education for Business