08832323.2010.480988
Journal of Education for Business
ISSN: 0883-2323 (Print) 1940-3356 (Online) Journal homepage: http://www.tandfonline.com/loi/vjeb20
The Use of the ETS Major Field Test for Assurance
of Business Content Learning: Assurance of
Waste?
Paul M. Mason , B. Jay Coleman , Jeffrey W. Steagall , Andres A. Gallo &
Michael M. Fabritius
To cite this article: Paul M. Mason , B. Jay Coleman , Jeffrey W. Steagall , Andres A. Gallo &
Michael M. Fabritius (2011) The Use of the ETS Major Field Test for Assurance of Business
Content Learning: Assurance of Waste?, Journal of Education for Business, 86:2, 71-77, DOI:
10.1080/08832323.2010.480988
To link to this article: http://dx.doi.org/10.1080/08832323.2010.480988
Published online: 23 Dec 2010.
Submit your article to this journal
Article views: 62
View related articles
Citing articles: 10 View citing articles
Full Terms & Conditions of access and use can be found at
http://www.tandfonline.com/action/journalInformation?journalCode=vjeb20
Download by: [Universitas Maritim Raja Ali Haji]
Date: 11 January 2016, At: 22:12
JOURNAL OF EDUCATION FOR BUSINESS, 86: 71–77, 2011
C Taylor & Francis Group, LLC
Copyright
ISSN: 0883-2323
DOI: 10.1080/08832323.2010.480988
The Use of the ETS Major Field Test for Assurance
of Business Content Learning: Assurance of Waste?
Paul M. Mason, B. Jay Coleman, Jeffrey W. Steagall, and Andres A. Gallo
University of North Florida, Jacksonville, Florida, USA
Downloaded by [Universitas Maritim Raja Ali Haji] at 22:12 11 January 2016
Michael M. Fabritius
Centre College, Danville, Kentucky, USA
Exit exams have become the currency of choice for both institutions and accrediting bodies
seeking to demonstrate student learning. Most researchers have ignored the opportunity costs
of these tests and the fundamental question of whether the exams add value to the assessment
process already in place on college campuses (course testing and GPA). A statistical model
that uses a student’s GPA, SAT score, and demographic characteristics predicts the student’s
Major Field Test (ETSB) score quite well. Furthermore, the opportunity cost of administering
these exams at the institution in question exceeds $25,000 annually. The authors argue that the
cost-benefit tradeoff of the ETSB exit exam is unfavorable.
Keywords: accreditation, assurance of learning, Educational Testing Service, exit exams,
logistic regression
Educational assessment is a major topic on college campuses as programs strive to achieve or maintain accreditation
and to demonstrate their efficacy to legislative and executive branches that oversee their funding. Accrediting bodies,
including the Association to Advance Collegiate Schools of
Business (AACSB), require evidence that business schools
fulfill their stated missions relative to student outcomes (Association to Advance Collegiate Schools of Business, 2003).
Regional accrediting bodies, such as the Southern Association of Colleges and Schools (SACS), have also begun to
mandate that educators prove that they accomplish their missions and achieve their expected learning outcomes (Southern
Association of Colleges and Schools, 2008).
Assurance of learning models in business schools virtually always contain a learning objective associated with
overall business content knowledge: learning fundamental
material across all major areas of business. This goal is separate from but typically included with objectives associated
with learning skills such as communication (oral, written, and
electronic), team building, critical thinking, and ethical reasoning. One increasingly common method schools employ
Correspondence should be addressed to Paul M. Mason, University of
North Florida, Department of Economics and Geography, 1 UNF Drive,
Jacksonville, FL 32223, USA. E-mail: pmason@unf.edu
to assess learning outcomes and measure students’ progress
in content learning is the administration of standardized exit
examinations at or near the end of students’ programs of
study. In many cases, a test such as the Educational Testing Service’s Major Field Test in Business (ETSB) is employed for this purpose. ETSB attempts to cover both depth
and breadth in assessing students’ levels of achievement and
is designed to measure a student’s knowledge and ability
to apply significant facts, concepts, theories, and analytical
methods (Educational Testing Service, 2006).
Because of this stated objective, the ETSB is viewed as
a viable instrument by institutions seeking to measure student progress vis-`a-vis business content knowledge. Some
accrediting bodies even recommend the ETSB as an evaluation instrument. For example, in a presentation to the administrative group of the institution studied herein, an SACS
accreditation representative specifically cited the ETSB as a
content knowledge assessment tool and inquired whether the
institution was using it.
In the present article we address the research question of
whether the administration of the ETSB is in effect wasteful
as an assurance of learning tool and, by extension, whether
the reliance or implicit insistence on the use of the ETSB
by accreditation bodies is appropriate. To pursue this question, we address two more specific lines of inquiry. First,
is the use of the ETSB efficient (or even necessary) as a
Downloaded by [Universitas Maritim Raja Ali Haji] at 22:12 11 January 2016
72
P. M. MASON ET AL.
measure of content knowledge? That is, does it add valuable information to the set of assessments of student learning that colleges routinely undertake (e.g., course grades)?
Several researchers have identified those demographic and
performance characteristics that generate the highest ETSB
scores. However, we seek not to determine how to maximize
ETSB scores, but rather to determine how well the score
a given student achieves can be predicted. A fundamental
hypothesis is that ETSB scores can be predicted accurately
from traditional grading criteria (e.g., GPAs), measures of
general academic preparation upon college entry (SAT or
ACT scores), and student demographic factors. This methodology advances previous literature on modeling ETSB results primarily through a more advanced form of analysis. Logistic regressions demonstrate that whether a student
passes the exam at a given threshold with a high degree of
accuracy (83–96%) using information known prior to the
student’s taking the exam can be predicted. Therefore, we
question the incremental value of the ETSB for assessing
learning. For present or prospective users of the ETSB, it appears that present and historical grading rubrics are sufficient
to capture very similar information about student content
knowledge.
Our second line of inquiry addresses whether the value
of any incremental information from the ETSB exceeds the
financial and opportunity costs of purchasing and administering the exams and analyzing the results. Such costbenefit analysis is typically absent from accrediting body
recommendations regarding the use of exit exams in general as well as in the literature. As we demonstrate subsequently, based on the data it can be estimated that the
direct and indirect costs associated with administration of
the ETSB at the institution in question exceed $25,000 annually. Given the limited marginal value of the ETSB that
we found in our first line of inquiry, combined with reduced funding for higher education, it is pertinent to question whether funds expended on the ETSB would be better
reallocated to hiring an additional faculty member, or research, etc.
METHOD
Several recent empirical articles dealt with assessment and
assurance of learning (Black & Duhon, 2003; Bycio & Allen,
2007; Marshall, 2007; Mirchandani, Lynch, & Hamilton,
2001; Pritchard, Potter, & Saccucci, 2006; Stoloff & Feeney,
2002). For brevity, we do not discuss their results except to
make reference to them in the subsequent analysis. Suffice it
to say that all of these researchers focused on what exit exams
reveal about demographic characteristics without discussing
the degree to which the scores can be accurately predicted.
They did not assess whether exit exams provide sufficient
incrementally valuable information to justify their cost.
To test the hypothesis regarding the accuracy with which
ETSB results can be predicted, most of the variables analyzed
in these studies were employed to assess the implications of
student performance on the ETSB. This data set is larger than
those in previous studies, consisting of 1,411 students who
took the ETSB as an exit exam in the capstone business policy
class at a comprehensive, urban institution with a total student
enrollment of between 15,000 and 20,000 students, and a
total undergraduate enrollment in the college of business of
approximately 3,000 students. ETSB results were collected
each semester from summer 2005 to fall 2007. The ETSB
counted as 10% of the student’s course grade in business
policy, so students had some incentive to perform well on the
exam. Thus, the sample represents a near census of incentivedriven students who graduated (or came close to graduating)
during the time frame examined.
Also collected were data on age, ethnicity, sex, term of
matriculation, overall GPA at graduation, GPA in business
courses only, GPA in nonbusiness courses, SAT (or equivalent ACT) score, major, transfer student status, and term
of graduation. Although ETSB scaled scores were available
for all 1,411 students, data anomalies on other factors, such
as missing graduating GPA values, missing ACT and SAT
scores, incorrect identifiers, and omitted responses relative
to ethnicity, lowered the useable sample size to 873. Many
students at the institution studied were older than traditional
college-aged students given the institution’s status as an urban university, but the remainder of the demographic and
performance characteristics of students and programs were
typical. The studied institution’s mean ETSB score for each
term in the data set was between the 75th and 90th percentiles
(inclusive) when compared to the mean ETSB scores at other
institutions using the exam.
For modeling purposes, the data were converted by the
matriculation and graduation dates into a value reflecting
degree completion time (expressed in number of terms).
Then separate completion time variables were constructed
for transfer students and for native (or nontransfer) students.
The term of administration was employed to construct binary variables reflecting fall, spring, and summer semesters,
with summer as the omitted category, and the year of administration (2005, 2006, and 2007, with 2005 as the omitted
category). Finally, the term variables were interacted with
the business GPA factor in order to assess whether the relationship between business GPA (BUSGPA) and the ETSB
scaled score (EXITEXAM) changed over time.
In order to compare the results to those from previous
studies, the analysis first produced bivariate correlations and
developed ordinary least squares (OLS) models employing
EXITEXAM as the dependent variable. Employing the demographic data and the GPA scores, a logistic model was
produced that predicts whether a student would get an exit
exam score above a given threshold (e.g., the 70th percentile)
based on the student’s specific characteristics. Our first line
of inquiry concerned whether a student passing the ETSB
ASSURANCE OF (CONTENT) LEARNING: ASSURANCE OF WASTE?
73
TABLE 1
OLS Results: Parameters
Downloaded by [Universitas Maritim Raja Ali Haji] at 22:12 11 January 2016
Variable
INTERCEPT
BUSGPA
NONBUSGPA
SATCONV
AGE
MALE
TRANSFER
BLACK
HISPANIC
ASIAN
AM INDIAN
MAR
ACC
FIN
IBS
TRL
BFS
ECN
COMPLETE TIME FOR NATIVES
COMPLETE TIME FOR TRANSFERS
FALL
SPRING
YR2006
YR2007
FAL2005XBUSGPA
SPR2006XBUSGPA
SUM2006XBUSGPA
FAL2006XBUSGPA
SPR2007XBUSGPA
SUM2007XBUSGPA
FAL2007XBUSGPA
Parameter estimate
SE
t
Pr > |t|
76.86
7.80
0.27
0.04
0.34
4.90
0.89
0.03
–1.77
–2.13
–0.95
0.00
6.62
5.27
3.80
2.05
6.76
7.66
0.43
0.09
2.56
–9.13
–2.10
–2.95
–1.05
3.12
0.65
–0.44
3.81
0.30
–0.09
7.04
2.29
1.22
0.00
0.12
0.59
1.64
1.14
1.26
1.05
4.09
0.87
0.91
0.80
1.17
1.49
1.55
2.11
0.33
0.22
5.30
5.75
6.25
6.19
1.83
2.58
2.15
2.89
2.49
2.13
2.80
10.92
3.40
0.22
18.91
2.79
8.36
0.54
0.02
–1.40
–2.03
–0.23
0.00
7.29
6.62
3.24
1.37
4.35
3.63
1.28
0.41
0.48
–1.59
–0.34
–0.48
–0.57
1.21
0.30
–0.15
1.53
0.14
–0.03
ISSN: 0883-2323 (Print) 1940-3356 (Online) Journal homepage: http://www.tandfonline.com/loi/vjeb20
The Use of the ETS Major Field Test for Assurance
of Business Content Learning: Assurance of
Waste?
Paul M. Mason , B. Jay Coleman , Jeffrey W. Steagall , Andres A. Gallo &
Michael M. Fabritius
To cite this article: Paul M. Mason , B. Jay Coleman , Jeffrey W. Steagall , Andres A. Gallo &
Michael M. Fabritius (2011) The Use of the ETS Major Field Test for Assurance of Business
Content Learning: Assurance of Waste?, Journal of Education for Business, 86:2, 71-77, DOI:
10.1080/08832323.2010.480988
To link to this article: http://dx.doi.org/10.1080/08832323.2010.480988
Published online: 23 Dec 2010.
Submit your article to this journal
Article views: 62
View related articles
Citing articles: 10 View citing articles
Full Terms & Conditions of access and use can be found at
http://www.tandfonline.com/action/journalInformation?journalCode=vjeb20
Download by: [Universitas Maritim Raja Ali Haji]
Date: 11 January 2016, At: 22:12
JOURNAL OF EDUCATION FOR BUSINESS, 86: 71–77, 2011
C Taylor & Francis Group, LLC
Copyright
ISSN: 0883-2323
DOI: 10.1080/08832323.2010.480988
The Use of the ETS Major Field Test for Assurance
of Business Content Learning: Assurance of Waste?
Paul M. Mason, B. Jay Coleman, Jeffrey W. Steagall, and Andres A. Gallo
University of North Florida, Jacksonville, Florida, USA
Downloaded by [Universitas Maritim Raja Ali Haji] at 22:12 11 January 2016
Michael M. Fabritius
Centre College, Danville, Kentucky, USA
Exit exams have become the currency of choice for both institutions and accrediting bodies
seeking to demonstrate student learning. Most researchers have ignored the opportunity costs
of these tests and the fundamental question of whether the exams add value to the assessment
process already in place on college campuses (course testing and GPA). A statistical model
that uses a student’s GPA, SAT score, and demographic characteristics predicts the student’s
Major Field Test (ETSB) score quite well. Furthermore, the opportunity cost of administering
these exams at the institution in question exceeds $25,000 annually. The authors argue that the
cost-benefit tradeoff of the ETSB exit exam is unfavorable.
Keywords: accreditation, assurance of learning, Educational Testing Service, exit exams,
logistic regression
Educational assessment is a major topic on college campuses as programs strive to achieve or maintain accreditation
and to demonstrate their efficacy to legislative and executive branches that oversee their funding. Accrediting bodies,
including the Association to Advance Collegiate Schools of
Business (AACSB), require evidence that business schools
fulfill their stated missions relative to student outcomes (Association to Advance Collegiate Schools of Business, 2003).
Regional accrediting bodies, such as the Southern Association of Colleges and Schools (SACS), have also begun to
mandate that educators prove that they accomplish their missions and achieve their expected learning outcomes (Southern
Association of Colleges and Schools, 2008).
Assurance of learning models in business schools virtually always contain a learning objective associated with
overall business content knowledge: learning fundamental
material across all major areas of business. This goal is separate from but typically included with objectives associated
with learning skills such as communication (oral, written, and
electronic), team building, critical thinking, and ethical reasoning. One increasingly common method schools employ
Correspondence should be addressed to Paul M. Mason, University of
North Florida, Department of Economics and Geography, 1 UNF Drive,
Jacksonville, FL 32223, USA. E-mail: pmason@unf.edu
to assess learning outcomes and measure students’ progress
in content learning is the administration of standardized exit
examinations at or near the end of students’ programs of
study. In many cases, a test such as the Educational Testing Service’s Major Field Test in Business (ETSB) is employed for this purpose. ETSB attempts to cover both depth
and breadth in assessing students’ levels of achievement and
is designed to measure a student’s knowledge and ability
to apply significant facts, concepts, theories, and analytical
methods (Educational Testing Service, 2006).
Because of this stated objective, the ETSB is viewed as
a viable instrument by institutions seeking to measure student progress vis-`a-vis business content knowledge. Some
accrediting bodies even recommend the ETSB as an evaluation instrument. For example, in a presentation to the administrative group of the institution studied herein, an SACS
accreditation representative specifically cited the ETSB as a
content knowledge assessment tool and inquired whether the
institution was using it.
In the present article we address the research question of
whether the administration of the ETSB is in effect wasteful
as an assurance of learning tool and, by extension, whether
the reliance or implicit insistence on the use of the ETSB
by accreditation bodies is appropriate. To pursue this question, we address two more specific lines of inquiry. First,
is the use of the ETSB efficient (or even necessary) as a
Downloaded by [Universitas Maritim Raja Ali Haji] at 22:12 11 January 2016
72
P. M. MASON ET AL.
measure of content knowledge? That is, does it add valuable information to the set of assessments of student learning that colleges routinely undertake (e.g., course grades)?
Several researchers have identified those demographic and
performance characteristics that generate the highest ETSB
scores. However, we seek not to determine how to maximize
ETSB scores, but rather to determine how well the score
a given student achieves can be predicted. A fundamental
hypothesis is that ETSB scores can be predicted accurately
from traditional grading criteria (e.g., GPAs), measures of
general academic preparation upon college entry (SAT or
ACT scores), and student demographic factors. This methodology advances previous literature on modeling ETSB results primarily through a more advanced form of analysis. Logistic regressions demonstrate that whether a student
passes the exam at a given threshold with a high degree of
accuracy (83–96%) using information known prior to the
student’s taking the exam can be predicted. Therefore, we
question the incremental value of the ETSB for assessing
learning. For present or prospective users of the ETSB, it appears that present and historical grading rubrics are sufficient
to capture very similar information about student content
knowledge.
Our second line of inquiry addresses whether the value
of any incremental information from the ETSB exceeds the
financial and opportunity costs of purchasing and administering the exams and analyzing the results. Such costbenefit analysis is typically absent from accrediting body
recommendations regarding the use of exit exams in general as well as in the literature. As we demonstrate subsequently, based on the data it can be estimated that the
direct and indirect costs associated with administration of
the ETSB at the institution in question exceed $25,000 annually. Given the limited marginal value of the ETSB that
we found in our first line of inquiry, combined with reduced funding for higher education, it is pertinent to question whether funds expended on the ETSB would be better
reallocated to hiring an additional faculty member, or research, etc.
METHOD
Several recent empirical articles dealt with assessment and
assurance of learning (Black & Duhon, 2003; Bycio & Allen,
2007; Marshall, 2007; Mirchandani, Lynch, & Hamilton,
2001; Pritchard, Potter, & Saccucci, 2006; Stoloff & Feeney,
2002). For brevity, we do not discuss their results except to
make reference to them in the subsequent analysis. Suffice it
to say that all of these researchers focused on what exit exams
reveal about demographic characteristics without discussing
the degree to which the scores can be accurately predicted.
They did not assess whether exit exams provide sufficient
incrementally valuable information to justify their cost.
To test the hypothesis regarding the accuracy with which
ETSB results can be predicted, most of the variables analyzed
in these studies were employed to assess the implications of
student performance on the ETSB. This data set is larger than
those in previous studies, consisting of 1,411 students who
took the ETSB as an exit exam in the capstone business policy
class at a comprehensive, urban institution with a total student
enrollment of between 15,000 and 20,000 students, and a
total undergraduate enrollment in the college of business of
approximately 3,000 students. ETSB results were collected
each semester from summer 2005 to fall 2007. The ETSB
counted as 10% of the student’s course grade in business
policy, so students had some incentive to perform well on the
exam. Thus, the sample represents a near census of incentivedriven students who graduated (or came close to graduating)
during the time frame examined.
Also collected were data on age, ethnicity, sex, term of
matriculation, overall GPA at graduation, GPA in business
courses only, GPA in nonbusiness courses, SAT (or equivalent ACT) score, major, transfer student status, and term
of graduation. Although ETSB scaled scores were available
for all 1,411 students, data anomalies on other factors, such
as missing graduating GPA values, missing ACT and SAT
scores, incorrect identifiers, and omitted responses relative
to ethnicity, lowered the useable sample size to 873. Many
students at the institution studied were older than traditional
college-aged students given the institution’s status as an urban university, but the remainder of the demographic and
performance characteristics of students and programs were
typical. The studied institution’s mean ETSB score for each
term in the data set was between the 75th and 90th percentiles
(inclusive) when compared to the mean ETSB scores at other
institutions using the exam.
For modeling purposes, the data were converted by the
matriculation and graduation dates into a value reflecting
degree completion time (expressed in number of terms).
Then separate completion time variables were constructed
for transfer students and for native (or nontransfer) students.
The term of administration was employed to construct binary variables reflecting fall, spring, and summer semesters,
with summer as the omitted category, and the year of administration (2005, 2006, and 2007, with 2005 as the omitted
category). Finally, the term variables were interacted with
the business GPA factor in order to assess whether the relationship between business GPA (BUSGPA) and the ETSB
scaled score (EXITEXAM) changed over time.
In order to compare the results to those from previous
studies, the analysis first produced bivariate correlations and
developed ordinary least squares (OLS) models employing
EXITEXAM as the dependent variable. Employing the demographic data and the GPA scores, a logistic model was
produced that predicts whether a student would get an exit
exam score above a given threshold (e.g., the 70th percentile)
based on the student’s specific characteristics. Our first line
of inquiry concerned whether a student passing the ETSB
ASSURANCE OF (CONTENT) LEARNING: ASSURANCE OF WASTE?
73
TABLE 1
OLS Results: Parameters
Downloaded by [Universitas Maritim Raja Ali Haji] at 22:12 11 January 2016
Variable
INTERCEPT
BUSGPA
NONBUSGPA
SATCONV
AGE
MALE
TRANSFER
BLACK
HISPANIC
ASIAN
AM INDIAN
MAR
ACC
FIN
IBS
TRL
BFS
ECN
COMPLETE TIME FOR NATIVES
COMPLETE TIME FOR TRANSFERS
FALL
SPRING
YR2006
YR2007
FAL2005XBUSGPA
SPR2006XBUSGPA
SUM2006XBUSGPA
FAL2006XBUSGPA
SPR2007XBUSGPA
SUM2007XBUSGPA
FAL2007XBUSGPA
Parameter estimate
SE
t
Pr > |t|
76.86
7.80
0.27
0.04
0.34
4.90
0.89
0.03
–1.77
–2.13
–0.95
0.00
6.62
5.27
3.80
2.05
6.76
7.66
0.43
0.09
2.56
–9.13
–2.10
–2.95
–1.05
3.12
0.65
–0.44
3.81
0.30
–0.09
7.04
2.29
1.22
0.00
0.12
0.59
1.64
1.14
1.26
1.05
4.09
0.87
0.91
0.80
1.17
1.49
1.55
2.11
0.33
0.22
5.30
5.75
6.25
6.19
1.83
2.58
2.15
2.89
2.49
2.13
2.80
10.92
3.40
0.22
18.91
2.79
8.36
0.54
0.02
–1.40
–2.03
–0.23
0.00
7.29
6.62
3.24
1.37
4.35
3.63
1.28
0.41
0.48
–1.59
–0.34
–0.48
–0.57
1.21
0.30
–0.15
1.53
0.14
–0.03