08832323.2011.576279
Journal of Education for Business
ISSN: 0883-2323 (Print) 1940-3356 (Online) Journal homepage: http://www.tandfonline.com/loi/vjeb20
Linking Course-Embedded Assessment Measures
and Performance on the Educational Testing
Service Major Field Test in Business
Gustavo A. Barboza & James Pesek
To cite this article: Gustavo A. Barboza & James Pesek (2012) Linking CourseEmbedded Assessment Measures and Performance on the Educational Testing Service
Major Field Test in Business, Journal of Education for Business, 87:2, 102-111, DOI:
10.1080/08832323.2011.576279
To link to this article: http://dx.doi.org/10.1080/08832323.2011.576279
Published online: 15 Dec 2011.
Submit your article to this journal
Article views: 90
View related articles
Citing articles: 7 View citing articles
Full Terms & Conditions of access and use can be found at
http://www.tandfonline.com/action/journalInformation?journalCode=vjeb20
Download by: [Universitas Maritim Raja Ali Haji]
Date: 11 January 2016, At: 21:57
JOURNAL OF EDUCATION FOR BUSINESS, 87: 102–111, 2012
C Taylor & Francis Group, LLC
Copyright
ISSN: 0883-2323 print / 1940-3356 online
DOI: 10.1080/08832323.2011.576279
Linking Course-Embedded Assessment Measures
and Performance on the Educational Testing Service
Major Field Test in Business
Gustavo A. Barboza and James Pesek
Downloaded by [Universitas Maritim Raja Ali Haji] at 21:57 11 January 2016
Clarion University of Pennsylvania, Clarion, Pennsylvania, USA
Assessment of the business curriculum and its learning goals and objectives has become a
major field of interest for business schools. The exploratory results of the authors’ model using
a sample of 173 students show robust support for the hypothesis that high marks in courseembedded assessment on business-specific analytical skills positively affect performance on
overall business disciplinary competence proxy by results in the Major Field Test in Business
(MFT-B) examination, while controlling for SAT score, GPA, major, and gender differences.
This particular result provides useful and relevant information to advance the assessment
process in schools of business as a valuable tool to enhance the overall learning experience.
The authors also found a marked difference across majors.
Keywords: assessment, business disciplinary competence, learning goals and objectives
Assessment of learning goals and objectives across the business curricula is increasingly receiving significant attention
as an important task conducive to successful Assurance of
Learning (AoL) and a driver for accreditation processes. In
this regard, assessment of learning goals plays a predominant
role in determining and evaluating students’ performance and
effectiveness in achieving overall business competence. Not
surprisingly, in recent years, a growing interest has been observed in the literature on learning outcomes assessment for
business students, in part, due to the emphasis that the Association to Advance Collegiate Schools of Business (AACSB)
International has placed on AoL since 2003. For example,
based on surveys conducted at several AACSB-accredited
schools in 2004 and 2006, Martell (2007) confirmed the dramatic progress those institutions made in developing and
supporting their AoL activities, including spending an average of $20,000 on AoL activities in 2006—a fivefold increase
over 2004. Using the 2006 survey results, Martell listed the
most popular direct methods of assessment as the following:
evaluating written and oral assignments with rubrics, courseembedded assessments, evaluating cases with a rubric, using the Educational Testing Service (ETS) Major Field Test
Correspondence should be addressed to Gustavo A. Barboza, Clarion
University of Pennsylvania, Department of Administrative Science, 319
Dana Still Hall, Clarion, PA 16214, USA. E-mail: [email protected]
in Business (MFT-B; Educational Testing Service, 2011),
and systematically evaluating teamwork. However, Martell
posed the question as to whether certain populations of students demonstrate different levels of achievement on learning goals, and added that uncovering those differences may
identify groups of students that may require remediation or
represent model practices or delivery systems that others may
emulate.
In this regard, a common practice across the assessment literature indicates a strong preference for using the
ETS MFT-B as the preferred measure of business disciplinary competence. As Bush, Duncan, Sexton, and West
(2008) reported, the MFT-B has been found to be a very
good way of assessing student learning outcomes. It also
provides users with baseline data and the opportunity to
benchmark students’ performances against national norms,
and measure possible student learning outcome variations
across nine business subjects: accounting, economics, management, quantitative business analysis, finance, marketing,
legal and social environment, information systems, and international issues. Although some faculty have expressed
concerns about the content validity of the MFT-B (see
Bycio & Allen, 2007), Gerlich and Sollosy (2008) noted that
the MFT-B has become for many the de facto standard of
standardized assessment instruments for business programs.
In Martell’s (2007) survey, 46% of the respondents indicated
that their institutions were using the MFT-B.
Downloaded by [Universitas Maritim Raja Ali Haji] at 21:57 11 January 2016
LINKING COURSE-EMBEDDED ASSESSMENT MEASURES
From an empirical point of view, several
researchers—using similar approaches—have investigated the determinants of business disciplinary competence
as the main driver of AoL. For example, Allen and Bycio
(1997); Bycio and Allen (2007); Gerlich and Sollosy (2008);
Marshall (2007); Terry, Mills, and Sollosy (2008); and
Bush et al. (2008), among others, have attempted to address
some of Martell’s (2007) questions by conducting studies
analyzing assessment results. These studies have involved
one or more of the following predictor variables: overall
grade point average (GPA), business core GPA, SAT or ACT
scores, gender, academic business major, grades earned
in certain courses, student motivations, and international
students and transfer students, and have found consistent
evidence of their effects on the dependent variable for
business disciplinary competence, the ETS MFT-B.
More specifically, Bycio and Allen (2007) found a positive and significant effect of students’ GPA in business core
classes on MFT-B performance. In addition, the overall GPA
and general intellectual aptitude (as reflected by SAT Verbal and SAT Math performance) predicted MFT-B scores
equally well. In addition, the researchers noted a significant improvement in MFT-B performance as a function of
course credit incentive offerings (Bycio & Allen, 2007).
While controlling for gender, age, ACT scores, and GPA,
Black and Duhon (2003) found that management majors
were at a significant disadvantage relative to all other business
majors.
Terry et al. (2008) examined the determinants of student
performance on the MFT-B with a focus on student motivation using percentile scores on the MFT-B as the dependent variable and ACT and SAT scores, GPA, transfer status,
international student, gender, and two motivation variables
as independent variables. Similar to other studies, Terry et
al. found that academic ability, as measured by the ACT
and SAT, and GPA were statistically significant variables.
Other demographic variables were not statistically significant. However, the two motivation variables, a 10% application of the ETS score to the final course grade and a 20%
application of the ETS score to the final course grade, were
both positive and statistically significant.
Marshall (2007) conducted a study involving types of
business majors, gender, transfer students, and grade performance in selected courses, including the capstone course
in business. She discovered that overall, accounting and finance majors generally scored better than any of the other
majors, particularly management and marketing majors. In
comparing her results, which centered on GPAs and grades
in the business capstone course, and the studies that used
the MFT-B scores, she concluded that differences in performance existed among majors in business programs. In a
related issue, Allen and Bycio (1997), using the MFT-B as
an assessment measure, found that there was an overall significant difference in the MFT-B performance as a function
of business major.
103
No¨el, Michaels, and Levas (2003) collected questionnaire
data from graduating seniors at two regionally diverse universities and found differences in accounting and marketing
majors’ personality and self-monitoring behaviors, noting
that marketing types were creative, easygoing, and enthusiastic, whereas accountants were more likely to be more
reserved, practical, and prone to use concrete and focused
thinking. Such behaviors help explain why some business
students select a particular major (No¨el et al.). Of course,
it may be a good thing that those who work in accounting
are not too creative! Differences in pedagogical preferences
and personality and self-monitoring behaviors may help explain performance differences on the MFT-B among majors.
Ulrich (2005) provided similar results supporting this argumentation.
The implicit assumption and common factor among the
studies mentioned previously is that a successful assessment
process—one that meets or exceeds expectations—should
positively affect the scores in the standard measure of
business disciplinary competence. The positive effects that
assessment practices may have on ETS MFT-B performance are then captured, most commonly, through its interaction with variables such as business core GPA and
T −B
> 0). The remaining observed
overall GPA, (i.e., ∂MF
∂GP A
portion of the business competence variation is then explained by
variables
such as, but not limited to, SAT
control
T −B
>
0
—and
demographics such as gender
scores— ∂MF
∂SAT
and major. However, a common shortcoming of the previous
studies is that none of them used course-embedded assessment specific variables as determinants of overall business
competence. In particular, understanding the role that assessment processes and corresponding instruments used in this
process has received little attention in the literature. This is
to say that the role, direction, and magnitude effects that assessment instruments may have in advancing and achieving
the overall goal of business disciplinary competence remains
unanswered.
Thus, despite the advances made in understanding the
determinants of business disciplinary competence, as measured by performance in the ETS MFT-B, the role of courseembedded instruments as they relate to linking assessment efforts to achieve learning goals and objectives and their effects
on higher demonstration of business disciplinary competence
remains largely unexplained. Therefore, in this exploratory
study we aimed to advance the understanding of assurance
of learning and its relevance for demonstrating business disciplinary competence by expanding on previous research
and measuring the significance and explanatory power of
endogenous and program-specific course-embedded assessment measurements, along the lines of business-specific analytical knowledge and writing skills, and their effect on
student performance on the MFT-B (business disciplinary
competence). In the next section, we describe the program
setting and corresponding learning goals and objectives and
how they relate to the assurance of learning and assessment
104
G. A. BARBOZA AND J. PESEK
Goal 1.0 – Demonstrate Business Disciplinary Competence
Objective 1.1
Knowledge in the key business disciplines. Key business disciplines
shall include accounting, economics, finance, management, and marketing.
Goal 2.0 – Show an Awareness of the Ethical Dimensions of Business Issues
Objective 2.1
Objective 2.2
Knowledge and ability to examine ethical issues in business
An understanding of the social forces shaping the environment of business
Goal 3.0 – Communicate Effectively Orally and in Written Form
Objective 3.1
Objective 3.2
Demonstration of effective oral communication skills
Demonstration of effective written communication skills
Goal 4.0 – Demonstrate Analytical Thinking Skills
Objective 4.1
Objective 4.2
Objective 4.3
Interpretation of evidence
Identification and evaluation of points of view
Formulation of warranted, non-fallacious conclusions
Goal 5.0 – Understand Global Issues in the Functional Areas of Business
Downloaded by [Universitas Maritim Raja Ali Haji] at 21:57 11 January 2016
Objective 5.1
Understanding the forces affecting businesses that operate in the global
economy
Goal 6.0 – Demonstrate Effective Use of Technology and Data Analysis
Objective 6.1
Objective 6.2
Objective 6.3
FIGURE 1
Demonstration of communication and presentation technologies used in the
business environment
Understanding of and ability to use common methods of statistical inference
Understanding of data analysis and its use in business decision making
Bachelor of Science in business administration learning goals and objectives.
measures. We then present the model specification and discuss the data and empirical approach, followed by the analysis and discussion of results. We end with recommendations
and conclusions.
SETTING AND BACHELOR OF SCIENCE IN
BUSINESS ADMINISTRATION (BSBA)
ASSESSMENT BACKGROUND
The setting for this study is a small, mid-Atlantic rural state
university with approximately 7,300 students. The university’s primary mission is teaching and learning, as it offers
over 50 undergraduate majors and 11 master’s degree programs. Although the university admits students from a wide
range of backgrounds, it does not have an open access admissions policy. Over the last five years, university admissions
acceptance rates for first-time freshmen ranged from a high
of 78% for fall 2005 to a low of 68.5% for fall 2009. The
overall average SAT score (using only the verbal and mathematics scores) for regular admitted students over the last five
years (fall 2005 through fall 2009) has ranged from a low of
999 in fall 2006 to a high of 1006 in fall 2007.
In the business program, approximately 88% of the business majors were White or non-Hispanic and 80% were residents of the state. Although 63% of the university’s student
body was female, 54% of those majoring in business were
male. The College of Business Administration (COBA) has
been accredited since the late 1990s by AACSB International.
Undergraduate students in COBA are offered eight majors:
accounting, economics, finance, marketing, management, international business, industrial relations, and real estate.
To assess student learning, the COBA developed six learning goals with corresponding objectives and standard rubrics
to conduct assessment. The program performs continuous
assessment of all learning goals on a yearly basis at different
points in the curriculum. In addition, the program administers the ETS MFT-B as an external measure for assessment
of learning goal 1.0 (i.e., demonstrate business disciplinary
competence). In particular, in this study we focused only
in data corresponding to goals 1.0, 3.0 (objective 3.2), and
4.0 (see Figure 1), and its corresponding assessment of the
capstone course senior project on strategic management (we
explain in further detail subsequently).
MODEL
As indicated previously, the interest of this study was to
determine the effects and significance of course-embedded
assessment measures—in particular business-specific analytical skills and overall writing skills—and other demographic information on demonstrating business disciplinary
competence. To this purpose, we proposed to use a parsimonious model description—similar to that found in the
literature—that follows the general specification of
yi = xi′ β + εi .
(1)
where εi is the errors term and is assumed to be an independent and identically distributed random variable as usual;
xi′ is the matrix of explanatory variables; β is the vector of
coefficients to be estimated; and yi is the measure of business
disciplinary competence. The vector of independent variables
includes the conventional demographic elements of gender
and major. It also includes GPA and the SAT scores (total [SATT], verbal [SATV], and mathematics [SATM]). The
model in Equation 1 expands on the existent literature by
adding the endogenous academic performance variable of
LINKING COURSE-EMBEDDED ASSESSMENT MEASURES
business-specific analytical skills and overall writing skills
(see Appendix).
Thus, we proposed to estimate empirically the following
equation:
MF T − Bi = α0 + α1 SAT (T = V + M)i
+ α2 GP Ai + βi Maj ori
Downloaded by [Universitas Maritim Raja Ali Haji] at 21:57 11 January 2016
+ α3 Analyticali + α4 W ritingi + εi (2)
where MFT-Bi score is the dependent variable and is used
as an external (exogenous) performance measure to assess
learning goal 1.0 (demonstrate business disciplinary competence; see BSBA Goal 1.0 in Figure 1). We selected MFT-B1
as our dependent variable for several reasons. The first reason is because it is an exogenous variable to the program
and therefore eliminates any possible endogeneity problems
with the rest of the variables that were determined within
the program in our model specification. Second, the MFT-B
is a preferred measure of business disciplinary competence
because it is a nationally administered test and assumed to
be normally distributed. Last, we selected MFT-B as our
dependent variable to obtain the highest possible degree of
comparison with previous research in this field, which proves
useful when assessing the implications of our findings.
The explanatory variables of performance in the ETS
MFT-B exam are the following: (a) GPA, the cumulative
GPA at the time students took the MFT-B examination, thus
GPA is not influenced by MFT-B scores and vice versa; (b)
academic major, a dummy variable for the student’s declared
major when taking the ETS MFT-B1; (c) SATT (and its corresponding decomposition between SATV and SATM) is the
students’ aptitude score as a measure of student capabilities used for admission purposes2; (d) gender is a dummy
variable for the population composition between females (1)
and males (0); (e) analytical is overall score in the courseembedded assessment measure of the business-specific analytical knowledge competence of the final paper for the
capstone course in business along seven business-specific
dimensions3; and (f) writing is the course-embedded assessment measure of the written component along five dimensions4 of the same paper for the capstone course as they relate
to the learning goals 4.0 and 3.0, respectively (see Figure 1
and the Appendix).
Although previous research has acknowledged the role
that student motivation may play in performance on the MFTB (see Black & Duhon 2003), instructors in the capstone
course did not provide bonus points or a grade incentive
for the students. Rather, instructors encouraged students to
try their best since good performance on the exam could be
included on a resume or graduate school admissions application.
Based on the model outlined in Equation 2, we advanced
the following hypotheses, and proceeded to test them empirically:
105
Hypothesis 1 (H 1 ): α2 > 0 Higher GPA would correspond to
higher MFT-B scores, all other things being equal, and
demonstrating that a better comprehension of the academic curriculum would reflect a higher performance in
demonstrating business disciplinary competence (learning goal 1).
H 2 : α1 > 0 Higher scores on the SAT (verbal and mathematics) exam would have a positive (and possibly increasing) effect on the MFT-B performance as they relate
positively to test-taking ability and native intellectual
ability.
H 3 : γ1 ≈ γ2 = 0 Gender differences would not have an effect
on demonstrating business disciplinary competence as
measured by the MFT-B exam.
H 4 : α3 > 0 Higher scores in the business-specific analytical skill learning goal assessment results would have a
positive impact in demonstrating business disciplinary
competence, and therefore would be positively related
to MFT-B scores.
H 5 : α4 > 0 Higher scores in the writing learning goal assessment results would have a positive impact on demonstrating business disciplinary competence, and therefore
would be positively related to MFT-B scores.
DATA AND ESTIMATION PROCEDURE
Data for this study came from a sample of 173 graduating
seniors from an AACSB-accredited college of business. Actual data used for the empirical estimations are less than
the initial 173 because of missing observations for some
variables. The data cover the period from 2007 to 2009.
In particular, the MFT-B examination score—our dependent
variable—was administered to the graduating seniors in the
spring semester in the capstone course. Data for the independent variables were collected in the following way: GPAs
were collected after the capstone business course was completed. This is relevant because the capstone course has been
flagged as the main course in which several learning goals
are assessed, including business-specific analytical and overall writing learning goals along with the oral communication
goal.5 The SATT score is the official entering score on which
initial admission to the university was granted. We also used
SATV and SATM scores to determine any possible differential effects of each test component. Data for major came
from the MFT-B results report, as indicated by the declared
major in the examination, and thus a dual-major student was
forced to select only one major. Writing and analytical data
were collected from the assessment of the learning goals
in the capstone senior project along several dimensions, as
noted previously. These data cover 12 different categories
grouped in two sections in which each item was assessed
based on a 3-point Likert-type scale –ranging from 1 (fails to
meet expectations) to 3 (exceeds expectations). The writing
Downloaded by [Universitas Maritim Raja Ali Haji] at 21:57 11 January 2016
106
G. A. BARBOZA AND J. PESEK
component corresponds to the summation of items 1 through
5, and items 6 through 12 correspond to the business-specific
analytical skills component (see rubric in the Appendix for
more details).
To better understand the significance of our analytical
variable, it is relevant to note that analytical—an endogenous
variable to the program—was constructed from the assessment of the senior project in the capstone business course.
Analytical measured the degree of mastery that students
achieved on business-specific concepts, the use of business
analytical tools and their applications to discover business
problems and issues, and the consequent construction of feasible business strategies. Therefore, analytical measured the
development and accumulation of skills above and beyond
those that a student could acquire outside the business curriculum. To be more specific, the senior project is a multistep
process that makes extensive and intensive use of a strength,
weaknesses, opportunities and threats (SWOT) analysis and
correspondent Internal Factor Evaluation (IFE) and External
Factor Evaluation (EFE) matrices6; utilizes several businessspecific analytical tools such as the Boston Consulting Group
(BCG), Internal-External (IE), SPACE, GRAND, and Qualitative Strategic Planning Matrix (QSPM) matrices to identify
problems and issues; and develops alternative strategies and
consequently recommends feasible courses of action for a
Fortune 500 company.
Table 1 presents the descriptive statistics for selected variables, including corresponding correlations. It is interesting to note that the MFT-B and SAT scores are leptokurtic,
whereas the GPA scores are platykurtic. This is to say that the
MFT-B and SAT scores were highly concentrated around the
mean, with lower variation within the observations, whereas
GPA scores were more widely spread than normal and were
flatter around the mean, indicating a more wide variation
across students. In addition, the data indicate that students in
the sample had an average GPA of 3.099 on a 4-point scale
or (letter grade of B), with all students meeting the school’s
major requirements of a C average or better. In addition, the
analytical and writing assessment scores indicate that students met or exceeded the assessment target as established
in the assessment plan. However, average scores on these
categories indicate the existence of room for improvement in
terms of achieving higher scores on average. Although the
majority of the students (54%) in the school of business were
men, for this sample 52% were women.
FINDINGS
We performed several estimations of alternative model specifications of Equation 2 using a log-log transformation. The
log-log transformation allowed us to account for differences in units of measure and scales of several variables.
Thus, this transformation facilitates interpretation of the results. In addition, because our data were cross-sectional in
nature, we used ordinary least squares (OLS) with White
heteroscedasticity-consistent standard errors and covariance.
The estimated coefficients are interpreted as elasticities.
With these basic considerations in mind, Table 2 shows
results for all alternative models with Log (MFT-B) as the
dependent variable. We estimated a total of 9 models. Our
baseline models (1, 4, and 7) included GPA and SATT. We
then teased out the data by conducting estimations (models
2, 3, 5, 6, 8, and 9) with the decomposition of SAT between
verbal (SATV) and quantitative (SATM) to separate the specific effects of each SAT component on business disciplinary
competence. We then introduced our course-embedded variables and conducted joint and separate estimations with the
analytical and writing variables to determine their joint and
individual effects on business disciplinary competence and
determine whether multicollinearity between these two assessment course-embedded measures was present. In particular, we had a strong interest to determine if the observed
high correlation between analytical and writing (0.7284 in
Table 1) translated into possible model misspecification. All
estimations controlled for demographic effects as captured
by differences in major and gender. This is to say that our
course-embedded measures captured effects above and beyond those reported for the conventional determinants of ETS
MFT-B performance, as outlined in the literature.
Based on our estimations, there are several results worth
noting: first, the positive and significant effects that the GPA
TABLE 1
Descriptive Statistics and Correlations for Selected Variables
Variable
Analytical
ETS
GPA
SATM
SATT
SATV
Writing
M
SD
Median
Number of
observations
Analytical
ETS
GPA
SATM
SATT
SATV
Writing
14.330
152.460
3.100
514.830
993.510
484.370
11.230
4.160
13.505
0.478
80.157
135.986
70.291
2.354
14
150.5
3.036
510
1000
480
11
149
164
173
151
151
151
149
—
0.4737
0.5249
0.3098
0.3475
0.3356
0.7284
0.4737
—
0.5908
0.5929
0.7053
0.6887
0.4803
0.5249
0.5908
—
0.4247
0.5059
0.4938
0.6071
0.3098
0.5929
0.4247
—
0.9157
0.6645
0.3444
0.3475
0.7053
0.5059
0.9157
—
0.9017
0.4150
0.3356
0.6887
0.4938
0.6645
0.9017
—
0.4250
0.7284
0.4803
0.6071
0.3444
0.4150
0.4250
—
Note. ETS = Educational Testing Service; GPA = grade point average; SATM = SAT mathematics test; SATV = SAT verbal test.
Downloaded by [Universitas Maritim Raja Ali Haji] at 21:57 11 January 2016
TABLE 2
Log-Log Assessment Estimation Models With the ETS MFT-B
1
Variable
Constant
Log (GPA)
Log (SATT)
Log (SATV)
Log (SATM)
Management
Finance
Marketing
Economics
Log (Writing)
Log (Analytical)
M
F
R2
2
SE
2.959∗∗
0.1415∗∗
0.265∗∗
–0.028
0.004
–0.048∗∗
0.033
–0.009
0.046∗
0.005
21.75∗∗
0.628
0.348
0.046
0.053
0.018
0.014
0.013
0.027
0.041
0.022
0.010
3
SE
3.330∗∗
0.154∗∗
0.293
0.045
0.235∗∗
0.052
−0.034∗
0.002
–0.052∗∗
–0.029
–0.013
0.047∗
0.013
21.37∗∗
0.624
0.017
0.015
0.013
0.026
0.041
0.023
0.011
4
SE
3.609∗∗
0.176∗∗
0.248
0.048
0.178∗∗
–0.029
0.006
–0.049∗∗
–0.031
0.010
0.041
0.005
17.22∗∗
0.572
0.042
0.019
0.016
0.014
0.029
0.039
0.024
0.011
5
SE
2.964∗∗
0.139∗∗
0.263∗∗
0.353
0.043
0.054
–0.028
0.004
–0.048∗∗
–0.033
0.017
0.014
0.013
0.026
0.042∗
0.006
24.65∗∗
0.628
0.018
0.010
6
SE
3.332∗∗
0.150∗∗
0.296
0.043
0.232∗∗
0.052
–0.034∗
0.002
–0.053∗∗
–0.029
0.042∗
0.014
24.21∗∗
0.623
0.017
0.015
0.013
0.026
0.019
0.010
7
SE
3.609∗∗
0.179∗∗
0.245
0.046
0.179∗∗
–0.028
0.006
–0.049∗∗
–0.032
0.042
0.019
0.016
0.014
0.029
0.045∗
0.005
19.52∗∗
0.572
0.019
0.011
8
SE
2.984∗∗
0.150∗∗
0.262∗∗
0.359
0.048
0.054
–0.028
0.002
–0.052∗∗
–0.034
0.035
0.018
0.014
0.013
0.027
0.034
0.006
23.49∗∗
0.616
0.010
9
SE
3.355∗∗
0.164∗∗
0.307
0.048
0.231∗∗
0.054
–0.034∗
–0.001
–0.057∗∗
–0.029
0.033
0.014
23.02∗∗
0.611
0.017
0.014
0.013
0.026
0.034
0.011
SE
3.619∗∗
0.183∗∗
0.255
0.049
0.177∗∗
–0.029
0.004
–0.053∗∗
–0.032
0.049
0.043
0.019
0.016
0.014
0.030
0.031
0.006
18.79∗∗
0.562
0.01
Note. n observations = 126. ETS MFT-B = Educational Testing Service Major Field Test for Business; GPA = grade point average; SATT = total SAT test score; SATV = SAT verbal test score; SATM
= SAT mathematics test score.
∗ p = .05. ∗∗ p = .01.
107
Downloaded by [Universitas Maritim Raja Ali Haji] at 21:57 11 January 2016
108
G. A. BARBOZA AND J. PESEK
and SAT had on MFT-B performance confirm what previous
researchers have also found.7 In particular, our estimations
expand the present body of empirical evidence in the field by
demonstrating that the verbal section of the SAT examination
(SATV) had a larger impact on determining MFT-B performance than the corresponding quantitative section (SATM)
did. In all our estimations SATV had an estimated elasticity
of 0.26 versus a corresponding estimated elasticity of 0.17
for SATM. The coefficient for GPA ranged from 0.139 to
0.183 in the alternative models, and was the second most
important determinant of overall performance in the MFT-B.
Also, the R2 value was higher when we controlled for SATT.
In relation to differences across majors, we used the accounting variable as the reference group (omitted variable).
The coefficient for differences across majors was negative
and statistically significant for management majors when we
used SATV, but not when SATT or SATM were used. It appears that management majors performed significantly lower
than their accounting peers in the SATM, and these differences persisted across the curriculum. Marketing majors, on
the other hand, consistently performed at the lowest level
even after controlling for predetermined abilities on SAT
scores and GPA. This result is also a partial confirmation
of findings in previous research, as noted previously in the
literature review, but expands on the finding that marketing
majors’ lower performance accentuates during the business
program (i.e., marketing majors lag behind in relation to their
peers even after controlling for ex-ante differences in SAT
scores). There was no statistically significant difference between finance and economics majors with accounting majors
in any of the models.
As expected, there existed a high correlation between the
writing and analytical variables. In general, we observed that
a well-written paper tends to convey and receive high marks
for business-specific analytical thinking. However, this correlation does not imply causation. Rather, the positive sign
of the analytical coefficient should be interpreted as a true
and direct effect on increased business disciplinary competence, deriving solely from program-specific acquired analytical skills, and not from writing skills. This is a particularly relevant result because it provides direct evidence of
program-specific value-added components for business disciplinary competence achievement, and serves as an indication
of positive effects of AoL efforts. Finally, the results for the
gender effects indicate that there was no statistically significant difference between performance of women and men.
We can safely argue that the ETS MFT-B is a test that does
not favor one gender over the other.
To further explore the sources of variation on MFT-B performance, we conducted some sensitivity analysis. As noted
previously, although SAT and GPA show some degree of
correlation (and it is well documented in the literature that
high SAT scores are more likely to result in higher GPA
scores), they serve different functions in measuring business
disciplinary competence. This is because both variables are
statistically significant at the 1% level of confidence. More
relevantly, the business-specific analytical measure had explanatory power above and beyond what was depicted by
GPA performance. In this regard, if the analytical score were
to increase by 1% (0.03 points) the business disciplinary
competence score would increase by 0.04 points.
Further exploration of the relationship between SAT and
GPA on overall business competency reveals that the separation of SAT scores between the SATV and SATM variables
was relevant (see models 2, 3, 5, 6, 7, and 8). First, we discovered that SATV had a larger impact on MFT-B performance
in relation to SATM. This difference is robust to whether we
included analytical and writing concurrently or separately.
Second, the major variable for marketing was consistently
negative and significant, indicating a relatively lower performance in relation to the peers. For management majors, the
negative relationship was statistically significant only when
we used SATV as the control variable. Furthermore, our assessment measure for business-specific analytical skills was
statistically significant in all models but model 3.8 In addition, writing was not statistically significant in any model,
neither when included concurrently with analytical nor separately. We argue that writing skill effects are already captured
in the SATT (and in particular STAV) and GPA variables.
Our results yield the preliminary conclusions that SAT
scores (SATT and SATV) are the most relevant individual determinants of MFT-B performance. GPA and SATM
have an almost identical overall effect on overall business
competency performance. In principle, schools with higher
SAT standards should achieve higher MFT-B scores, all other
things being equal. In fact, these differences tend to be larger
the higher the SAT scores. However, this should not be taken
as a sufficient condition for higher performance in MFT-B
alone, as the initial effects of SAT diminish in importance as
students advance in their studies, and other variables such as
GPA and development of business-specific analytical skills
become more relevant. In this regard, GPA is the second most
important determinant of MFT-B scores, and all other things
being equal, the more students apply themselves in the learning process the greater the likelihood that they will score
higher in the exogenous measure of business disciplinary
competence, ETS MFT-B. Thus, high GPA performance may
offset possible initial disadvantages that schools with lower
SAT admission standards may have.
However, although we do not dispute the fact that SAT
scores are a relevant measure of general abilities, our empirical estimates indicate that business-specific analytical skills
are internally developed in the business program and serve
to enhance the MFT-B performance of the students, above
and beyond those capabilities capture by the SAT and GPA
scores. Thus, the robustness of the business-specific analytical assessment measure as a significant determinant of MFTB performance is an indication of two things. First, efforts
to develop business-specific analytical skills are a true measure of value added, as they relate to increasing business
Downloaded by [Universitas Maritim Raja Ali Haji] at 21:57 11 January 2016
LINKING COURSE-EMBEDDED ASSESSMENT MEASURES
disciplinary competence; second, although analytical and
writing are correlated, our analytical variable is clearly endogenous to the program and a reinforcement to AoL practices as mechanisms to demonstrate business competence.
That is to say, although there might be room for improved
business disciplinary competence through better writing,
these skills develop in large part ex-ante to students entering
the business curriculum. The evidence extends the argument
to indicate that this appears to be the case across all business majors. In terms of policy recommendations it is worth
noting that business programs have a larger control over the
analytical component of the program and its positive effects
on achieving higher levels of business disciplinary competence than other components. This is a clear indication of the
robust effect that the business-specific analytical assessment
variable has as a value-added measure for demonstrating
business disciplinary competence. This in fact is the primary
finding of this study. Although we acknowledge that there
may be an alternative explanation in this regard, namely
that the students may have acquired more general analytical skills prior to entering the business program—or from
a third source while in the program—this explanation does
not diminish the importance of our findings because our analytical variable is business-specific and endogenous to the
program curriculum. In addition, SAT scores capture some of
the writing and more general analytical abilities that are time
and program invariant. Finally, differences across majors indicate that some attention needs to be paid at the program
level to bring the performance of marketing and management
majors to the level of the reference group.
Put simply, students that apply themselves and successfully achieve higher GPAs and develop strong businessspecific analytical skills could potentially catch up with those
students that initially had a SAT advantage. It follows that
this improvement is highly correlated with the completeness, complexity, and overall thoroughness of the business
curriculum because achieving a higher GPA in a more rigorous program is more difficult than otherwise. In this regard,
placing an emphasis in learning activities that emphasize acquisition, development, and use of business-specific analytical skills play a determinant role in achieving higher levels
of business disciplinary competence, as our results elucidate
through the positive effects on MFT-B scores.
Incidentally, our business-specific analytical assessment
variable measured abilities that students had acquired during
their academic program in terms of conceptualization, identification, application, recommendation, and building of overall business strategies for effective decision making, which
are directly related to a comprehensive business disciplinary
competence. Assessment of this particular variable plays a
fundamental role in achieving the preset learning goals for
the program. The most relevant and immediate result is that
our endogenous (program and business-specific) measure of
analytical skills is also a good predictor of overall business
disciplinary competence. Thus, program value added builds
109
on pre-existent capabilities (SAT scores), and serves as a
measure of the program effectiveness to overcome any possible selectivity bias that could affect less stringent, yet rigorous, admission procedures—a result not found in the existent
literature.
Our results push the knowledge envelope on assessmentrelated issues on two major fronts. First, continuous assessment of learning goals at the program level proves
valuable in identifying, establishing, measuring, and guiding strategies conducive to increasing business competence
through course-embedded assessment. Second, although students may also develop and hone their analytical and critical
thinking skills in nonbusiness courses, in the present we study
made use of program-specific value-added measures in the
form of business-specific analytical skills development to assess and quantify their effect on business competence, while
controlling for conventional ability measures (GPA and SAT)
and other demographics.
RECOMMENDATIONS AND CONCLUSIONS
There are several relevant recommendations that spring from
our analysis. In particular, our results point out the possible
changes in curriculum and target levels for assessment in the
core areas of the business program to continue improving
business disciplinary competence achievement.
In terms of the program recommendations, it is interesting to notice the convergence of two effects. First, students
achieving high marks (meet or exceed expectations) in the
business-specific analytical component of their senior project
performed better in the MFT-B than otherwise. This is a result in support of a continuous assessment effort to foster
excellence in teaching and learning. Also, program efforts
directed to not only meet accreditation requirements and target levels, but aimed at boosting business-specific analytical
scores above the target, or that result in overall increased
targets, should also yield higher business disciplinary competence as measured by MFT-B scoring. On the same note,
adding more projects with a strong business-specific analytical content across the business curriculum should have
the same positive impact on achieving higher scores on the
external measure of business disciplinary competence. However, notice that achieving a higher MFT-B score is not a
learning goal in itself, but the result of a program learning goal and objective, which is achieving and successfully
completing a comprehensive business competence that derives in a significant amount of program value added. In this
regard, developing a simpler version of the business-specific
analytical skills rubric—presently used at the senior level
only—with the intention to assess business-specific knowledge upon admission to the business program could prove
useful to develop a true valued-added measure of business
analytical skills. This however, is beyond the scope of the
present study.
Downloaded by [Universitas Maritim Raja Ali Haji] at 21:57 11 January 2016
110
G. A. BARBOZA AND J. PESEK
The exploratory results of this study also point out to the
direction that assessment of the business curriculum provides
valuable information on achieving and meeting the learning
goals and corresponding objectives. Promoting practices that
raise the overall assessment results should positively affect
the overall learning process. More specifically, promoting
practices that result in increased GPA and higher businessspecific Analytical scores could prove beneficial and a significant determinant for improvement in business disciplinary
competence. These effects indicate that students excelling
in the classroom gain knowledge and develop their intellect
beyond the native abilities as reflected by the SAT scores.
The results also indicate that there are persistent differences across business majors, with accounting, finance, and
economics majors outperforming the rest. In this regard, two
questions remain unanswered and are possible outlets for
further research. First, are the observed differences across
majors correlated to the SAT scores? Or, more importantly,
are there program-specific differences in the business curriculum across majors that result in overall differentiated
learning outcomes that are major specific? Previous research
provides some insight to this question, but a closer examination of the curriculum across majors may prove beneficial. A
third less likely yet possible explanation relates to the MFT-B
construct and whether the examination design provides more
opportunities for accounting and finance majors to do well,
compared with other business majors, as noted by No¨el et al.
(2003). These questions are outside the scope of this article.
Finally, we summarize results from our study as the following: Assessment of the business curriculum proves to be a
valuable tool to assure a comprehensive and mission-oriented
learning process. Clear and direct program- and missionoriented learning goals and objectives that are comprehensive and use consistent rubrics with well-established targets
are valuable in measuring business disciplinary competence.
These positive effects of course-embedded assessments provide robust evidence that a continuous monitoring of the
academic progress results in improved performance and effective and efficient use of resources to enhance the overall
learning experience along the established mission statement.
NOTES
1. According to the ETS (2011) website, the ETS MFTB “contains 120 multiple-choice questions designed to
measure a student’s subject knowledge and the ability
to apply facts, concepts, theories and analytical methods. .. The questions represent a wide range of difficulty and cover depth and breadth in assessing students’
achievement levels.”
2. Previous research indicates that SAT scores
are relevant overall performance indicators for
lower levels courses—freshman- and sophomorelevel classes—whereas GPA becomes a more relevant performance determinants for higher level
3.
4.
5.
6.
7.
8.
courses—junior- and senior-level courses. Nevertheless, we incorporated SAT scores because the MFT-B
performance could potentially be exogenously determined, beyond actual learning taking place in the business curricula.
The seven dimensions were factual knowledge, application of strategic analytical tools, identification of
case problems and issues, application of case problems
and issues, generation of alternatives, recommendations, and business knowledge.
Writing dimensions were thesis or opening statement,
organization, spelling and word choice, grammar, and
sentence structure.
Oral communication scores were not used in this study.
The construction of the IFE and EFE matrices relies
heavily on the use and mastery of the Resource Based
View and the Porter’s Five Forces analysis, respectively.
Bycio and Allen (2007) noted that the positive effect of
SAT on MFT-B results could be an indication of general
test-taking ability. Furthermore, they noted that schools
with strong SAT profiles have stronger performance in
comparison with schools with less stringent admission
requirements.
The analytical variable was marginally significant at
8% in this model. The reduction is significance could
be a consequence of sample size. Notice that the coefficient magnitude and sign is robust across models.
REFERENCES
Allen, J. S., & Bycio, P. (1997). An evaluation of the Educational Testing
Service Major Field Achievement Test in Business. Journal of Accounting
Education, 15, 503–514.
Bycio, P., & Allen, J. S. (2007). Factors associated with performance on
the Educational Testing Service (ETS) Major Field Achievement Test in
Business (MFAT-B). Journal of Education for Business, 82, 196–201.
Bush, H. F., Duncan, F. H., Sexton, E. A., & West, C. T. (2008). Using the
Major Field Test-Business as an assessment tool and impetus for program
improvement: Fifteen years of experience at Virginia Military Institute.
Journal of College Teaching and Learning, 5, 75–88.
Educational Testing Service. (2011). Major Field Test in Business. Retrieved
from http://ets.org/mft/about/content/bachelor business
Gerlich, R. N., & Sollosy, M. (2008). Predicting assessment outcomes:
The effect of full-time and part-time faculty. Journal of Case Studies in
Accreditation and Assessment, 1, 1–9.
Marshall, L. L. (2007). Measuring assurance of learning at the degree program and academic major levels. Journal of Education for Business, 83,
101–109.
Martell, K. (2007). Assessing student learning: Are business schools making
the grade? Journal of Education for Business, 82, 189–195.
No¨el, N. M., Michaels, C., & Levas, M. G. (2003). The relationship of personality traits and self-monitoring behavior to choice of business major.
Journal of Education for Business, 78, 153–157.
Terry, N., Mills, L., & Sollosy, M. (2008). Student grade motivation as
a determinant of performance on the Business Major Field ETS Exam.
Journal of College Teaching & Learning, 5(7), 27–32.
Ulrich, T. A. (2005). The relationship of business major to pedagogical
strategies. Journal of Education for Business, 80, 269–274.
LINKING COURSE-EMBEDDED ASSESSMENT MEASURES
111
APPENDIX—EFFECTIVE WRITTEN COMMUNICATION RUBRIC
Draft—Scoring Rubric (Developed by Monique Forte, Stetson University)
Strategy Case Analysis to Assess Written Communication and Analytical Thinking Skills
Evaluative criteria
Written communication
1. Thesis/opening statement
2. Organization
Downloaded by [Universitas Maritim Raja Ali Haji] at 21:57 11 January 2016
3. Spelling & word choice
4. Grammar
5. Sentence structure
Analytical thinking
6. Factual knowledge
Fails to meet expectations (1)
Meets expectations (3)
Offers a weak or unfocused thesis
statement
Opens with a clear statement of case
problems/issues
Hooks reader with clever/insightful
opener to clearly identify case
issues
Uses few headings or paragraph
Provides organized analysis that
Provides clear organization scheme
breaks; shows weak logical flow
generally maintains focus
to guide reader through logic of
analysis
Has spellchecked, but may miss a typo Uses correct spelling throughout and
Uses many misspelled words and
or use an inappropriate word/term
demonstrates strong vocabulary
ISSN: 0883-2323 (Print) 1940-3356 (Online) Journal homepage: http://www.tandfonline.com/loi/vjeb20
Linking Course-Embedded Assessment Measures
and Performance on the Educational Testing
Service Major Field Test in Business
Gustavo A. Barboza & James Pesek
To cite this article: Gustavo A. Barboza & James Pesek (2012) Linking CourseEmbedded Assessment Measures and Performance on the Educational Testing Service
Major Field Test in Business, Journal of Education for Business, 87:2, 102-111, DOI:
10.1080/08832323.2011.576279
To link to this article: http://dx.doi.org/10.1080/08832323.2011.576279
Published online: 15 Dec 2011.
Submit your article to this journal
Article views: 90
View related articles
Citing articles: 7 View citing articles
Full Terms & Conditions of access and use can be found at
http://www.tandfonline.com/action/journalInformation?journalCode=vjeb20
Download by: [Universitas Maritim Raja Ali Haji]
Date: 11 January 2016, At: 21:57
JOURNAL OF EDUCATION FOR BUSINESS, 87: 102–111, 2012
C Taylor & Francis Group, LLC
Copyright
ISSN: 0883-2323 print / 1940-3356 online
DOI: 10.1080/08832323.2011.576279
Linking Course-Embedded Assessment Measures
and Performance on the Educational Testing Service
Major Field Test in Business
Gustavo A. Barboza and James Pesek
Downloaded by [Universitas Maritim Raja Ali Haji] at 21:57 11 January 2016
Clarion University of Pennsylvania, Clarion, Pennsylvania, USA
Assessment of the business curriculum and its learning goals and objectives has become a
major field of interest for business schools. The exploratory results of the authors’ model using
a sample of 173 students show robust support for the hypothesis that high marks in courseembedded assessment on business-specific analytical skills positively affect performance on
overall business disciplinary competence proxy by results in the Major Field Test in Business
(MFT-B) examination, while controlling for SAT score, GPA, major, and gender differences.
This particular result provides useful and relevant information to advance the assessment
process in schools of business as a valuable tool to enhance the overall learning experience.
The authors also found a marked difference across majors.
Keywords: assessment, business disciplinary competence, learning goals and objectives
Assessment of learning goals and objectives across the business curricula is increasingly receiving significant attention
as an important task conducive to successful Assurance of
Learning (AoL) and a driver for accreditation processes. In
this regard, assessment of learning goals plays a predominant
role in determining and evaluating students’ performance and
effectiveness in achieving overall business competence. Not
surprisingly, in recent years, a growing interest has been observed in the literature on learning outcomes assessment for
business students, in part, due to the emphasis that the Association to Advance Collegiate Schools of Business (AACSB)
International has placed on AoL since 2003. For example,
based on surveys conducted at several AACSB-accredited
schools in 2004 and 2006, Martell (2007) confirmed the dramatic progress those institutions made in developing and
supporting their AoL activities, including spending an average of $20,000 on AoL activities in 2006—a fivefold increase
over 2004. Using the 2006 survey results, Martell listed the
most popular direct methods of assessment as the following:
evaluating written and oral assignments with rubrics, courseembedded assessments, evaluating cases with a rubric, using the Educational Testing Service (ETS) Major Field Test
Correspondence should be addressed to Gustavo A. Barboza, Clarion
University of Pennsylvania, Department of Administrative Science, 319
Dana Still Hall, Clarion, PA 16214, USA. E-mail: [email protected]
in Business (MFT-B; Educational Testing Service, 2011),
and systematically evaluating teamwork. However, Martell
posed the question as to whether certain populations of students demonstrate different levels of achievement on learning goals, and added that uncovering those differences may
identify groups of students that may require remediation or
represent model practices or delivery systems that others may
emulate.
In this regard, a common practice across the assessment literature indicates a strong preference for using the
ETS MFT-B as the preferred measure of business disciplinary competence. As Bush, Duncan, Sexton, and West
(2008) reported, the MFT-B has been found to be a very
good way of assessing student learning outcomes. It also
provides users with baseline data and the opportunity to
benchmark students’ performances against national norms,
and measure possible student learning outcome variations
across nine business subjects: accounting, economics, management, quantitative business analysis, finance, marketing,
legal and social environment, information systems, and international issues. Although some faculty have expressed
concerns about the content validity of the MFT-B (see
Bycio & Allen, 2007), Gerlich and Sollosy (2008) noted that
the MFT-B has become for many the de facto standard of
standardized assessment instruments for business programs.
In Martell’s (2007) survey, 46% of the respondents indicated
that their institutions were using the MFT-B.
Downloaded by [Universitas Maritim Raja Ali Haji] at 21:57 11 January 2016
LINKING COURSE-EMBEDDED ASSESSMENT MEASURES
From an empirical point of view, several
researchers—using similar approaches—have investigated the determinants of business disciplinary competence
as the main driver of AoL. For example, Allen and Bycio
(1997); Bycio and Allen (2007); Gerlich and Sollosy (2008);
Marshall (2007); Terry, Mills, and Sollosy (2008); and
Bush et al. (2008), among others, have attempted to address
some of Martell’s (2007) questions by conducting studies
analyzing assessment results. These studies have involved
one or more of the following predictor variables: overall
grade point average (GPA), business core GPA, SAT or ACT
scores, gender, academic business major, grades earned
in certain courses, student motivations, and international
students and transfer students, and have found consistent
evidence of their effects on the dependent variable for
business disciplinary competence, the ETS MFT-B.
More specifically, Bycio and Allen (2007) found a positive and significant effect of students’ GPA in business core
classes on MFT-B performance. In addition, the overall GPA
and general intellectual aptitude (as reflected by SAT Verbal and SAT Math performance) predicted MFT-B scores
equally well. In addition, the researchers noted a significant improvement in MFT-B performance as a function of
course credit incentive offerings (Bycio & Allen, 2007).
While controlling for gender, age, ACT scores, and GPA,
Black and Duhon (2003) found that management majors
were at a significant disadvantage relative to all other business
majors.
Terry et al. (2008) examined the determinants of student
performance on the MFT-B with a focus on student motivation using percentile scores on the MFT-B as the dependent variable and ACT and SAT scores, GPA, transfer status,
international student, gender, and two motivation variables
as independent variables. Similar to other studies, Terry et
al. found that academic ability, as measured by the ACT
and SAT, and GPA were statistically significant variables.
Other demographic variables were not statistically significant. However, the two motivation variables, a 10% application of the ETS score to the final course grade and a 20%
application of the ETS score to the final course grade, were
both positive and statistically significant.
Marshall (2007) conducted a study involving types of
business majors, gender, transfer students, and grade performance in selected courses, including the capstone course
in business. She discovered that overall, accounting and finance majors generally scored better than any of the other
majors, particularly management and marketing majors. In
comparing her results, which centered on GPAs and grades
in the business capstone course, and the studies that used
the MFT-B scores, she concluded that differences in performance existed among majors in business programs. In a
related issue, Allen and Bycio (1997), using the MFT-B as
an assessment measure, found that there was an overall significant difference in the MFT-B performance as a function
of business major.
103
No¨el, Michaels, and Levas (2003) collected questionnaire
data from graduating seniors at two regionally diverse universities and found differences in accounting and marketing
majors’ personality and self-monitoring behaviors, noting
that marketing types were creative, easygoing, and enthusiastic, whereas accountants were more likely to be more
reserved, practical, and prone to use concrete and focused
thinking. Such behaviors help explain why some business
students select a particular major (No¨el et al.). Of course,
it may be a good thing that those who work in accounting
are not too creative! Differences in pedagogical preferences
and personality and self-monitoring behaviors may help explain performance differences on the MFT-B among majors.
Ulrich (2005) provided similar results supporting this argumentation.
The implicit assumption and common factor among the
studies mentioned previously is that a successful assessment
process—one that meets or exceeds expectations—should
positively affect the scores in the standard measure of
business disciplinary competence. The positive effects that
assessment practices may have on ETS MFT-B performance are then captured, most commonly, through its interaction with variables such as business core GPA and
T −B
> 0). The remaining observed
overall GPA, (i.e., ∂MF
∂GP A
portion of the business competence variation is then explained by
variables
such as, but not limited to, SAT
control
T −B
>
0
—and
demographics such as gender
scores— ∂MF
∂SAT
and major. However, a common shortcoming of the previous
studies is that none of them used course-embedded assessment specific variables as determinants of overall business
competence. In particular, understanding the role that assessment processes and corresponding instruments used in this
process has received little attention in the literature. This is
to say that the role, direction, and magnitude effects that assessment instruments may have in advancing and achieving
the overall goal of business disciplinary competence remains
unanswered.
Thus, despite the advances made in understanding the
determinants of business disciplinary competence, as measured by performance in the ETS MFT-B, the role of courseembedded instruments as they relate to linking assessment efforts to achieve learning goals and objectives and their effects
on higher demonstration of business disciplinary competence
remains largely unexplained. Therefore, in this exploratory
study we aimed to advance the understanding of assurance
of learning and its relevance for demonstrating business disciplinary competence by expanding on previous research
and measuring the significance and explanatory power of
endogenous and program-specific course-embedded assessment measurements, along the lines of business-specific analytical knowledge and writing skills, and their effect on
student performance on the MFT-B (business disciplinary
competence). In the next section, we describe the program
setting and corresponding learning goals and objectives and
how they relate to the assurance of learning and assessment
104
G. A. BARBOZA AND J. PESEK
Goal 1.0 – Demonstrate Business Disciplinary Competence
Objective 1.1
Knowledge in the key business disciplines. Key business disciplines
shall include accounting, economics, finance, management, and marketing.
Goal 2.0 – Show an Awareness of the Ethical Dimensions of Business Issues
Objective 2.1
Objective 2.2
Knowledge and ability to examine ethical issues in business
An understanding of the social forces shaping the environment of business
Goal 3.0 – Communicate Effectively Orally and in Written Form
Objective 3.1
Objective 3.2
Demonstration of effective oral communication skills
Demonstration of effective written communication skills
Goal 4.0 – Demonstrate Analytical Thinking Skills
Objective 4.1
Objective 4.2
Objective 4.3
Interpretation of evidence
Identification and evaluation of points of view
Formulation of warranted, non-fallacious conclusions
Goal 5.0 – Understand Global Issues in the Functional Areas of Business
Downloaded by [Universitas Maritim Raja Ali Haji] at 21:57 11 January 2016
Objective 5.1
Understanding the forces affecting businesses that operate in the global
economy
Goal 6.0 – Demonstrate Effective Use of Technology and Data Analysis
Objective 6.1
Objective 6.2
Objective 6.3
FIGURE 1
Demonstration of communication and presentation technologies used in the
business environment
Understanding of and ability to use common methods of statistical inference
Understanding of data analysis and its use in business decision making
Bachelor of Science in business administration learning goals and objectives.
measures. We then present the model specification and discuss the data and empirical approach, followed by the analysis and discussion of results. We end with recommendations
and conclusions.
SETTING AND BACHELOR OF SCIENCE IN
BUSINESS ADMINISTRATION (BSBA)
ASSESSMENT BACKGROUND
The setting for this study is a small, mid-Atlantic rural state
university with approximately 7,300 students. The university’s primary mission is teaching and learning, as it offers
over 50 undergraduate majors and 11 master’s degree programs. Although the university admits students from a wide
range of backgrounds, it does not have an open access admissions policy. Over the last five years, university admissions
acceptance rates for first-time freshmen ranged from a high
of 78% for fall 2005 to a low of 68.5% for fall 2009. The
overall average SAT score (using only the verbal and mathematics scores) for regular admitted students over the last five
years (fall 2005 through fall 2009) has ranged from a low of
999 in fall 2006 to a high of 1006 in fall 2007.
In the business program, approximately 88% of the business majors were White or non-Hispanic and 80% were residents of the state. Although 63% of the university’s student
body was female, 54% of those majoring in business were
male. The College of Business Administration (COBA) has
been accredited since the late 1990s by AACSB International.
Undergraduate students in COBA are offered eight majors:
accounting, economics, finance, marketing, management, international business, industrial relations, and real estate.
To assess student learning, the COBA developed six learning goals with corresponding objectives and standard rubrics
to conduct assessment. The program performs continuous
assessment of all learning goals on a yearly basis at different
points in the curriculum. In addition, the program administers the ETS MFT-B as an external measure for assessment
of learning goal 1.0 (i.e., demonstrate business disciplinary
competence). In particular, in this study we focused only
in data corresponding to goals 1.0, 3.0 (objective 3.2), and
4.0 (see Figure 1), and its corresponding assessment of the
capstone course senior project on strategic management (we
explain in further detail subsequently).
MODEL
As indicated previously, the interest of this study was to
determine the effects and significance of course-embedded
assessment measures—in particular business-specific analytical skills and overall writing skills—and other demographic information on demonstrating business disciplinary
competence. To this purpose, we proposed to use a parsimonious model description—similar to that found in the
literature—that follows the general specification of
yi = xi′ β + εi .
(1)
where εi is the errors term and is assumed to be an independent and identically distributed random variable as usual;
xi′ is the matrix of explanatory variables; β is the vector of
coefficients to be estimated; and yi is the measure of business
disciplinary competence. The vector of independent variables
includes the conventional demographic elements of gender
and major. It also includes GPA and the SAT scores (total [SATT], verbal [SATV], and mathematics [SATM]). The
model in Equation 1 expands on the existent literature by
adding the endogenous academic performance variable of
LINKING COURSE-EMBEDDED ASSESSMENT MEASURES
business-specific analytical skills and overall writing skills
(see Appendix).
Thus, we proposed to estimate empirically the following
equation:
MF T − Bi = α0 + α1 SAT (T = V + M)i
+ α2 GP Ai + βi Maj ori
Downloaded by [Universitas Maritim Raja Ali Haji] at 21:57 11 January 2016
+ α3 Analyticali + α4 W ritingi + εi (2)
where MFT-Bi score is the dependent variable and is used
as an external (exogenous) performance measure to assess
learning goal 1.0 (demonstrate business disciplinary competence; see BSBA Goal 1.0 in Figure 1). We selected MFT-B1
as our dependent variable for several reasons. The first reason is because it is an exogenous variable to the program
and therefore eliminates any possible endogeneity problems
with the rest of the variables that were determined within
the program in our model specification. Second, the MFT-B
is a preferred measure of business disciplinary competence
because it is a nationally administered test and assumed to
be normally distributed. Last, we selected MFT-B as our
dependent variable to obtain the highest possible degree of
comparison with previous research in this field, which proves
useful when assessing the implications of our findings.
The explanatory variables of performance in the ETS
MFT-B exam are the following: (a) GPA, the cumulative
GPA at the time students took the MFT-B examination, thus
GPA is not influenced by MFT-B scores and vice versa; (b)
academic major, a dummy variable for the student’s declared
major when taking the ETS MFT-B1; (c) SATT (and its corresponding decomposition between SATV and SATM) is the
students’ aptitude score as a measure of student capabilities used for admission purposes2; (d) gender is a dummy
variable for the population composition between females (1)
and males (0); (e) analytical is overall score in the courseembedded assessment measure of the business-specific analytical knowledge competence of the final paper for the
capstone course in business along seven business-specific
dimensions3; and (f) writing is the course-embedded assessment measure of the written component along five dimensions4 of the same paper for the capstone course as they relate
to the learning goals 4.0 and 3.0, respectively (see Figure 1
and the Appendix).
Although previous research has acknowledged the role
that student motivation may play in performance on the MFTB (see Black & Duhon 2003), instructors in the capstone
course did not provide bonus points or a grade incentive
for the students. Rather, instructors encouraged students to
try their best since good performance on the exam could be
included on a resume or graduate school admissions application.
Based on the model outlined in Equation 2, we advanced
the following hypotheses, and proceeded to test them empirically:
105
Hypothesis 1 (H 1 ): α2 > 0 Higher GPA would correspond to
higher MFT-B scores, all other things being equal, and
demonstrating that a better comprehension of the academic curriculum would reflect a higher performance in
demonstrating business disciplinary competence (learning goal 1).
H 2 : α1 > 0 Higher scores on the SAT (verbal and mathematics) exam would have a positive (and possibly increasing) effect on the MFT-B performance as they relate
positively to test-taking ability and native intellectual
ability.
H 3 : γ1 ≈ γ2 = 0 Gender differences would not have an effect
on demonstrating business disciplinary competence as
measured by the MFT-B exam.
H 4 : α3 > 0 Higher scores in the business-specific analytical skill learning goal assessment results would have a
positive impact in demonstrating business disciplinary
competence, and therefore would be positively related
to MFT-B scores.
H 5 : α4 > 0 Higher scores in the writing learning goal assessment results would have a positive impact on demonstrating business disciplinary competence, and therefore
would be positively related to MFT-B scores.
DATA AND ESTIMATION PROCEDURE
Data for this study came from a sample of 173 graduating
seniors from an AACSB-accredited college of business. Actual data used for the empirical estimations are less than
the initial 173 because of missing observations for some
variables. The data cover the period from 2007 to 2009.
In particular, the MFT-B examination score—our dependent
variable—was administered to the graduating seniors in the
spring semester in the capstone course. Data for the independent variables were collected in the following way: GPAs
were collected after the capstone business course was completed. This is relevant because the capstone course has been
flagged as the main course in which several learning goals
are assessed, including business-specific analytical and overall writing learning goals along with the oral communication
goal.5 The SATT score is the official entering score on which
initial admission to the university was granted. We also used
SATV and SATM scores to determine any possible differential effects of each test component. Data for major came
from the MFT-B results report, as indicated by the declared
major in the examination, and thus a dual-major student was
forced to select only one major. Writing and analytical data
were collected from the assessment of the learning goals
in the capstone senior project along several dimensions, as
noted previously. These data cover 12 different categories
grouped in two sections in which each item was assessed
based on a 3-point Likert-type scale –ranging from 1 (fails to
meet expectations) to 3 (exceeds expectations). The writing
Downloaded by [Universitas Maritim Raja Ali Haji] at 21:57 11 January 2016
106
G. A. BARBOZA AND J. PESEK
component corresponds to the summation of items 1 through
5, and items 6 through 12 correspond to the business-specific
analytical skills component (see rubric in the Appendix for
more details).
To better understand the significance of our analytical
variable, it is relevant to note that analytical—an endogenous
variable to the program—was constructed from the assessment of the senior project in the capstone business course.
Analytical measured the degree of mastery that students
achieved on business-specific concepts, the use of business
analytical tools and their applications to discover business
problems and issues, and the consequent construction of feasible business strategies. Therefore, analytical measured the
development and accumulation of skills above and beyond
those that a student could acquire outside the business curriculum. To be more specific, the senior project is a multistep
process that makes extensive and intensive use of a strength,
weaknesses, opportunities and threats (SWOT) analysis and
correspondent Internal Factor Evaluation (IFE) and External
Factor Evaluation (EFE) matrices6; utilizes several businessspecific analytical tools such as the Boston Consulting Group
(BCG), Internal-External (IE), SPACE, GRAND, and Qualitative Strategic Planning Matrix (QSPM) matrices to identify
problems and issues; and develops alternative strategies and
consequently recommends feasible courses of action for a
Fortune 500 company.
Table 1 presents the descriptive statistics for selected variables, including corresponding correlations. It is interesting to note that the MFT-B and SAT scores are leptokurtic,
whereas the GPA scores are platykurtic. This is to say that the
MFT-B and SAT scores were highly concentrated around the
mean, with lower variation within the observations, whereas
GPA scores were more widely spread than normal and were
flatter around the mean, indicating a more wide variation
across students. In addition, the data indicate that students in
the sample had an average GPA of 3.099 on a 4-point scale
or (letter grade of B), with all students meeting the school’s
major requirements of a C average or better. In addition, the
analytical and writing assessment scores indicate that students met or exceeded the assessment target as established
in the assessment plan. However, average scores on these
categories indicate the existence of room for improvement in
terms of achieving higher scores on average. Although the
majority of the students (54%) in the school of business were
men, for this sample 52% were women.
FINDINGS
We performed several estimations of alternative model specifications of Equation 2 using a log-log transformation. The
log-log transformation allowed us to account for differences in units of measure and scales of several variables.
Thus, this transformation facilitates interpretation of the results. In addition, because our data were cross-sectional in
nature, we used ordinary least squares (OLS) with White
heteroscedasticity-consistent standard errors and covariance.
The estimated coefficients are interpreted as elasticities.
With these basic considerations in mind, Table 2 shows
results for all alternative models with Log (MFT-B) as the
dependent variable. We estimated a total of 9 models. Our
baseline models (1, 4, and 7) included GPA and SATT. We
then teased out the data by conducting estimations (models
2, 3, 5, 6, 8, and 9) with the decomposition of SAT between
verbal (SATV) and quantitative (SATM) to separate the specific effects of each SAT component on business disciplinary
competence. We then introduced our course-embedded variables and conducted joint and separate estimations with the
analytical and writing variables to determine their joint and
individual effects on business disciplinary competence and
determine whether multicollinearity between these two assessment course-embedded measures was present. In particular, we had a strong interest to determine if the observed
high correlation between analytical and writing (0.7284 in
Table 1) translated into possible model misspecification. All
estimations controlled for demographic effects as captured
by differences in major and gender. This is to say that our
course-embedded measures captured effects above and beyond those reported for the conventional determinants of ETS
MFT-B performance, as outlined in the literature.
Based on our estimations, there are several results worth
noting: first, the positive and significant effects that the GPA
TABLE 1
Descriptive Statistics and Correlations for Selected Variables
Variable
Analytical
ETS
GPA
SATM
SATT
SATV
Writing
M
SD
Median
Number of
observations
Analytical
ETS
GPA
SATM
SATT
SATV
Writing
14.330
152.460
3.100
514.830
993.510
484.370
11.230
4.160
13.505
0.478
80.157
135.986
70.291
2.354
14
150.5
3.036
510
1000
480
11
149
164
173
151
151
151
149
—
0.4737
0.5249
0.3098
0.3475
0.3356
0.7284
0.4737
—
0.5908
0.5929
0.7053
0.6887
0.4803
0.5249
0.5908
—
0.4247
0.5059
0.4938
0.6071
0.3098
0.5929
0.4247
—
0.9157
0.6645
0.3444
0.3475
0.7053
0.5059
0.9157
—
0.9017
0.4150
0.3356
0.6887
0.4938
0.6645
0.9017
—
0.4250
0.7284
0.4803
0.6071
0.3444
0.4150
0.4250
—
Note. ETS = Educational Testing Service; GPA = grade point average; SATM = SAT mathematics test; SATV = SAT verbal test.
Downloaded by [Universitas Maritim Raja Ali Haji] at 21:57 11 January 2016
TABLE 2
Log-Log Assessment Estimation Models With the ETS MFT-B
1
Variable
Constant
Log (GPA)
Log (SATT)
Log (SATV)
Log (SATM)
Management
Finance
Marketing
Economics
Log (Writing)
Log (Analytical)
M
F
R2
2
SE
2.959∗∗
0.1415∗∗
0.265∗∗
–0.028
0.004
–0.048∗∗
0.033
–0.009
0.046∗
0.005
21.75∗∗
0.628
0.348
0.046
0.053
0.018
0.014
0.013
0.027
0.041
0.022
0.010
3
SE
3.330∗∗
0.154∗∗
0.293
0.045
0.235∗∗
0.052
−0.034∗
0.002
–0.052∗∗
–0.029
–0.013
0.047∗
0.013
21.37∗∗
0.624
0.017
0.015
0.013
0.026
0.041
0.023
0.011
4
SE
3.609∗∗
0.176∗∗
0.248
0.048
0.178∗∗
–0.029
0.006
–0.049∗∗
–0.031
0.010
0.041
0.005
17.22∗∗
0.572
0.042
0.019
0.016
0.014
0.029
0.039
0.024
0.011
5
SE
2.964∗∗
0.139∗∗
0.263∗∗
0.353
0.043
0.054
–0.028
0.004
–0.048∗∗
–0.033
0.017
0.014
0.013
0.026
0.042∗
0.006
24.65∗∗
0.628
0.018
0.010
6
SE
3.332∗∗
0.150∗∗
0.296
0.043
0.232∗∗
0.052
–0.034∗
0.002
–0.053∗∗
–0.029
0.042∗
0.014
24.21∗∗
0.623
0.017
0.015
0.013
0.026
0.019
0.010
7
SE
3.609∗∗
0.179∗∗
0.245
0.046
0.179∗∗
–0.028
0.006
–0.049∗∗
–0.032
0.042
0.019
0.016
0.014
0.029
0.045∗
0.005
19.52∗∗
0.572
0.019
0.011
8
SE
2.984∗∗
0.150∗∗
0.262∗∗
0.359
0.048
0.054
–0.028
0.002
–0.052∗∗
–0.034
0.035
0.018
0.014
0.013
0.027
0.034
0.006
23.49∗∗
0.616
0.010
9
SE
3.355∗∗
0.164∗∗
0.307
0.048
0.231∗∗
0.054
–0.034∗
–0.001
–0.057∗∗
–0.029
0.033
0.014
23.02∗∗
0.611
0.017
0.014
0.013
0.026
0.034
0.011
SE
3.619∗∗
0.183∗∗
0.255
0.049
0.177∗∗
–0.029
0.004
–0.053∗∗
–0.032
0.049
0.043
0.019
0.016
0.014
0.030
0.031
0.006
18.79∗∗
0.562
0.01
Note. n observations = 126. ETS MFT-B = Educational Testing Service Major Field Test for Business; GPA = grade point average; SATT = total SAT test score; SATV = SAT verbal test score; SATM
= SAT mathematics test score.
∗ p = .05. ∗∗ p = .01.
107
Downloaded by [Universitas Maritim Raja Ali Haji] at 21:57 11 January 2016
108
G. A. BARBOZA AND J. PESEK
and SAT had on MFT-B performance confirm what previous
researchers have also found.7 In particular, our estimations
expand the present body of empirical evidence in the field by
demonstrating that the verbal section of the SAT examination
(SATV) had a larger impact on determining MFT-B performance than the corresponding quantitative section (SATM)
did. In all our estimations SATV had an estimated elasticity
of 0.26 versus a corresponding estimated elasticity of 0.17
for SATM. The coefficient for GPA ranged from 0.139 to
0.183 in the alternative models, and was the second most
important determinant of overall performance in the MFT-B.
Also, the R2 value was higher when we controlled for SATT.
In relation to differences across majors, we used the accounting variable as the reference group (omitted variable).
The coefficient for differences across majors was negative
and statistically significant for management majors when we
used SATV, but not when SATT or SATM were used. It appears that management majors performed significantly lower
than their accounting peers in the SATM, and these differences persisted across the curriculum. Marketing majors, on
the other hand, consistently performed at the lowest level
even after controlling for predetermined abilities on SAT
scores and GPA. This result is also a partial confirmation
of findings in previous research, as noted previously in the
literature review, but expands on the finding that marketing
majors’ lower performance accentuates during the business
program (i.e., marketing majors lag behind in relation to their
peers even after controlling for ex-ante differences in SAT
scores). There was no statistically significant difference between finance and economics majors with accounting majors
in any of the models.
As expected, there existed a high correlation between the
writing and analytical variables. In general, we observed that
a well-written paper tends to convey and receive high marks
for business-specific analytical thinking. However, this correlation does not imply causation. Rather, the positive sign
of the analytical coefficient should be interpreted as a true
and direct effect on increased business disciplinary competence, deriving solely from program-specific acquired analytical skills, and not from writing skills. This is a particularly relevant result because it provides direct evidence of
program-specific value-added components for business disciplinary competence achievement, and serves as an indication
of positive effects of AoL efforts. Finally, the results for the
gender effects indicate that there was no statistically significant difference between performance of women and men.
We can safely argue that the ETS MFT-B is a test that does
not favor one gender over the other.
To further explore the sources of variation on MFT-B performance, we conducted some sensitivity analysis. As noted
previously, although SAT and GPA show some degree of
correlation (and it is well documented in the literature that
high SAT scores are more likely to result in higher GPA
scores), they serve different functions in measuring business
disciplinary competence. This is because both variables are
statistically significant at the 1% level of confidence. More
relevantly, the business-specific analytical measure had explanatory power above and beyond what was depicted by
GPA performance. In this regard, if the analytical score were
to increase by 1% (0.03 points) the business disciplinary
competence score would increase by 0.04 points.
Further exploration of the relationship between SAT and
GPA on overall business competency reveals that the separation of SAT scores between the SATV and SATM variables
was relevant (see models 2, 3, 5, 6, 7, and 8). First, we discovered that SATV had a larger impact on MFT-B performance
in relation to SATM. This difference is robust to whether we
included analytical and writing concurrently or separately.
Second, the major variable for marketing was consistently
negative and significant, indicating a relatively lower performance in relation to the peers. For management majors, the
negative relationship was statistically significant only when
we used SATV as the control variable. Furthermore, our assessment measure for business-specific analytical skills was
statistically significant in all models but model 3.8 In addition, writing was not statistically significant in any model,
neither when included concurrently with analytical nor separately. We argue that writing skill effects are already captured
in the SATT (and in particular STAV) and GPA variables.
Our results yield the preliminary conclusions that SAT
scores (SATT and SATV) are the most relevant individual determinants of MFT-B performance. GPA and SATM
have an almost identical overall effect on overall business
competency performance. In principle, schools with higher
SAT standards should achieve higher MFT-B scores, all other
things being equal. In fact, these differences tend to be larger
the higher the SAT scores. However, this should not be taken
as a sufficient condition for higher performance in MFT-B
alone, as the initial effects of SAT diminish in importance as
students advance in their studies, and other variables such as
GPA and development of business-specific analytical skills
become more relevant. In this regard, GPA is the second most
important determinant of MFT-B scores, and all other things
being equal, the more students apply themselves in the learning process the greater the likelihood that they will score
higher in the exogenous measure of business disciplinary
competence, ETS MFT-B. Thus, high GPA performance may
offset possible initial disadvantages that schools with lower
SAT admission standards may have.
However, although we do not dispute the fact that SAT
scores are a relevant measure of general abilities, our empirical estimates indicate that business-specific analytical skills
are internally developed in the business program and serve
to enhance the MFT-B performance of the students, above
and beyond those capabilities capture by the SAT and GPA
scores. Thus, the robustness of the business-specific analytical assessment measure as a significant determinant of MFTB performance is an indication of two things. First, efforts
to develop business-specific analytical skills are a true measure of value added, as they relate to increasing business
Downloaded by [Universitas Maritim Raja Ali Haji] at 21:57 11 January 2016
LINKING COURSE-EMBEDDED ASSESSMENT MEASURES
disciplinary competence; second, although analytical and
writing are correlated, our analytical variable is clearly endogenous to the program and a reinforcement to AoL practices as mechanisms to demonstrate business competence.
That is to say, although there might be room for improved
business disciplinary competence through better writing,
these skills develop in large part ex-ante to students entering
the business curriculum. The evidence extends the argument
to indicate that this appears to be the case across all business majors. In terms of policy recommendations it is worth
noting that business programs have a larger control over the
analytical component of the program and its positive effects
on achieving higher levels of business disciplinary competence than other components. This is a clear indication of the
robust effect that the business-specific analytical assessment
variable has as a value-added measure for demonstrating
business disciplinary competence. This in fact is the primary
finding of this study. Although we acknowledge that there
may be an alternative explanation in this regard, namely
that the students may have acquired more general analytical skills prior to entering the business program—or from
a third source while in the program—this explanation does
not diminish the importance of our findings because our analytical variable is business-specific and endogenous to the
program curriculum. In addition, SAT scores capture some of
the writing and more general analytical abilities that are time
and program invariant. Finally, differences across majors indicate that some attention needs to be paid at the program
level to bring the performance of marketing and management
majors to the level of the reference group.
Put simply, students that apply themselves and successfully achieve higher GPAs and develop strong businessspecific analytical skills could potentially catch up with those
students that initially had a SAT advantage. It follows that
this improvement is highly correlated with the completeness, complexity, and overall thoroughness of the business
curriculum because achieving a higher GPA in a more rigorous program is more difficult than otherwise. In this regard,
placing an emphasis in learning activities that emphasize acquisition, development, and use of business-specific analytical skills play a determinant role in achieving higher levels
of business disciplinary competence, as our results elucidate
through the positive effects on MFT-B scores.
Incidentally, our business-specific analytical assessment
variable measured abilities that students had acquired during
their academic program in terms of conceptualization, identification, application, recommendation, and building of overall business strategies for effective decision making, which
are directly related to a comprehensive business disciplinary
competence. Assessment of this particular variable plays a
fundamental role in achieving the preset learning goals for
the program. The most relevant and immediate result is that
our endogenous (program and business-specific) measure of
analytical skills is also a good predictor of overall business
disciplinary competence. Thus, program value added builds
109
on pre-existent capabilities (SAT scores), and serves as a
measure of the program effectiveness to overcome any possible selectivity bias that could affect less stringent, yet rigorous, admission procedures—a result not found in the existent
literature.
Our results push the knowledge envelope on assessmentrelated issues on two major fronts. First, continuous assessment of learning goals at the program level proves
valuable in identifying, establishing, measuring, and guiding strategies conducive to increasing business competence
through course-embedded assessment. Second, although students may also develop and hone their analytical and critical
thinking skills in nonbusiness courses, in the present we study
made use of program-specific value-added measures in the
form of business-specific analytical skills development to assess and quantify their effect on business competence, while
controlling for conventional ability measures (GPA and SAT)
and other demographics.
RECOMMENDATIONS AND CONCLUSIONS
There are several relevant recommendations that spring from
our analysis. In particular, our results point out the possible
changes in curriculum and target levels for assessment in the
core areas of the business program to continue improving
business disciplinary competence achievement.
In terms of the program recommendations, it is interesting to notice the convergence of two effects. First, students
achieving high marks (meet or exceed expectations) in the
business-specific analytical component of their senior project
performed better in the MFT-B than otherwise. This is a result in support of a continuous assessment effort to foster
excellence in teaching and learning. Also, program efforts
directed to not only meet accreditation requirements and target levels, but aimed at boosting business-specific analytical
scores above the target, or that result in overall increased
targets, should also yield higher business disciplinary competence as measured by MFT-B scoring. On the same note,
adding more projects with a strong business-specific analytical content across the business curriculum should have
the same positive impact on achieving higher scores on the
external measure of business disciplinary competence. However, notice that achieving a higher MFT-B score is not a
learning goal in itself, but the result of a program learning goal and objective, which is achieving and successfully
completing a comprehensive business competence that derives in a significant amount of program value added. In this
regard, developing a simpler version of the business-specific
analytical skills rubric—presently used at the senior level
only—with the intention to assess business-specific knowledge upon admission to the business program could prove
useful to develop a true valued-added measure of business
analytical skills. This however, is beyond the scope of the
present study.
Downloaded by [Universitas Maritim Raja Ali Haji] at 21:57 11 January 2016
110
G. A. BARBOZA AND J. PESEK
The exploratory results of this study also point out to the
direction that assessment of the business curriculum provides
valuable information on achieving and meeting the learning
goals and corresponding objectives. Promoting practices that
raise the overall assessment results should positively affect
the overall learning process. More specifically, promoting
practices that result in increased GPA and higher businessspecific Analytical scores could prove beneficial and a significant determinant for improvement in business disciplinary
competence. These effects indicate that students excelling
in the classroom gain knowledge and develop their intellect
beyond the native abilities as reflected by the SAT scores.
The results also indicate that there are persistent differences across business majors, with accounting, finance, and
economics majors outperforming the rest. In this regard, two
questions remain unanswered and are possible outlets for
further research. First, are the observed differences across
majors correlated to the SAT scores? Or, more importantly,
are there program-specific differences in the business curriculum across majors that result in overall differentiated
learning outcomes that are major specific? Previous research
provides some insight to this question, but a closer examination of the curriculum across majors may prove beneficial. A
third less likely yet possible explanation relates to the MFT-B
construct and whether the examination design provides more
opportunities for accounting and finance majors to do well,
compared with other business majors, as noted by No¨el et al.
(2003). These questions are outside the scope of this article.
Finally, we summarize results from our study as the following: Assessment of the business curriculum proves to be a
valuable tool to assure a comprehensive and mission-oriented
learning process. Clear and direct program- and missionoriented learning goals and objectives that are comprehensive and use consistent rubrics with well-established targets
are valuable in measuring business disciplinary competence.
These positive effects of course-embedded assessments provide robust evidence that a continuous monitoring of the
academic progress results in improved performance and effective and efficient use of resources to enhance the overall
learning experience along the established mission statement.
NOTES
1. According to the ETS (2011) website, the ETS MFTB “contains 120 multiple-choice questions designed to
measure a student’s subject knowledge and the ability
to apply facts, concepts, theories and analytical methods. .. The questions represent a wide range of difficulty and cover depth and breadth in assessing students’
achievement levels.”
2. Previous research indicates that SAT scores
are relevant overall performance indicators for
lower levels courses—freshman- and sophomorelevel classes—whereas GPA becomes a more relevant performance determinants for higher level
3.
4.
5.
6.
7.
8.
courses—junior- and senior-level courses. Nevertheless, we incorporated SAT scores because the MFT-B
performance could potentially be exogenously determined, beyond actual learning taking place in the business curricula.
The seven dimensions were factual knowledge, application of strategic analytical tools, identification of
case problems and issues, application of case problems
and issues, generation of alternatives, recommendations, and business knowledge.
Writing dimensions were thesis or opening statement,
organization, spelling and word choice, grammar, and
sentence structure.
Oral communication scores were not used in this study.
The construction of the IFE and EFE matrices relies
heavily on the use and mastery of the Resource Based
View and the Porter’s Five Forces analysis, respectively.
Bycio and Allen (2007) noted that the positive effect of
SAT on MFT-B results could be an indication of general
test-taking ability. Furthermore, they noted that schools
with strong SAT profiles have stronger performance in
comparison with schools with less stringent admission
requirements.
The analytical variable was marginally significant at
8% in this model. The reduction is significance could
be a consequence of sample size. Notice that the coefficient magnitude and sign is robust across models.
REFERENCES
Allen, J. S., & Bycio, P. (1997). An evaluation of the Educational Testing
Service Major Field Achievement Test in Business. Journal of Accounting
Education, 15, 503–514.
Bycio, P., & Allen, J. S. (2007). Factors associated with performance on
the Educational Testing Service (ETS) Major Field Achievement Test in
Business (MFAT-B). Journal of Education for Business, 82, 196–201.
Bush, H. F., Duncan, F. H., Sexton, E. A., & West, C. T. (2008). Using the
Major Field Test-Business as an assessment tool and impetus for program
improvement: Fifteen years of experience at Virginia Military Institute.
Journal of College Teaching and Learning, 5, 75–88.
Educational Testing Service. (2011). Major Field Test in Business. Retrieved
from http://ets.org/mft/about/content/bachelor business
Gerlich, R. N., & Sollosy, M. (2008). Predicting assessment outcomes:
The effect of full-time and part-time faculty. Journal of Case Studies in
Accreditation and Assessment, 1, 1–9.
Marshall, L. L. (2007). Measuring assurance of learning at the degree program and academic major levels. Journal of Education for Business, 83,
101–109.
Martell, K. (2007). Assessing student learning: Are business schools making
the grade? Journal of Education for Business, 82, 189–195.
No¨el, N. M., Michaels, C., & Levas, M. G. (2003). The relationship of personality traits and self-monitoring behavior to choice of business major.
Journal of Education for Business, 78, 153–157.
Terry, N., Mills, L., & Sollosy, M. (2008). Student grade motivation as
a determinant of performance on the Business Major Field ETS Exam.
Journal of College Teaching & Learning, 5(7), 27–32.
Ulrich, T. A. (2005). The relationship of business major to pedagogical
strategies. Journal of Education for Business, 80, 269–274.
LINKING COURSE-EMBEDDED ASSESSMENT MEASURES
111
APPENDIX—EFFECTIVE WRITTEN COMMUNICATION RUBRIC
Draft—Scoring Rubric (Developed by Monique Forte, Stetson University)
Strategy Case Analysis to Assess Written Communication and Analytical Thinking Skills
Evaluative criteria
Written communication
1. Thesis/opening statement
2. Organization
Downloaded by [Universitas Maritim Raja Ali Haji] at 21:57 11 January 2016
3. Spelling & word choice
4. Grammar
5. Sentence structure
Analytical thinking
6. Factual knowledge
Fails to meet expectations (1)
Meets expectations (3)
Offers a weak or unfocused thesis
statement
Opens with a clear statement of case
problems/issues
Hooks reader with clever/insightful
opener to clearly identify case
issues
Uses few headings or paragraph
Provides organized analysis that
Provides clear organization scheme
breaks; shows weak logical flow
generally maintains focus
to guide reader through logic of
analysis
Has spellchecked, but may miss a typo Uses correct spelling throughout and
Uses many misspelled words and
or use an inappropriate word/term
demonstrates strong vocabulary