Manajemen | Fakultas Ekonomi Universitas Maritim Raja Ali Haji joeb.84.1.40-46

Journal of Education for Business

ISSN: 0883-2323 (Print) 1940-3356 (Online) Journal homepage: http://www.tandfonline.com/loi/vjeb20

Deciphering Student Evaluations of Teaching: A
Factor Analysis Approach
Michael M. Barth
To cite this article: Michael M. Barth (2008) Deciphering Student Evaluations of Teaching:
A Factor Analysis Approach, Journal of Education for Business, 84:1, 40-46, DOI: 10.3200/
JOEB.84.1.40-46
To link to this article: http://dx.doi.org/10.3200/JOEB.84.1.40-46

Published online: 07 Aug 2010.

Submit your article to this journal

Article views: 72

View related articles

Citing articles: 12 View citing articles

Full Terms & Conditions of access and use can be found at
http://www.tandfonline.com/action/journalInformation?journalCode=vjeb20
Download by: [Universitas Maritim Raja Ali Haji]

Date: 11 January 2016, At: 22:43

Downloaded by [Universitas Maritim Raja Ali Haji] at 22:43 11 January 2016

Deciphering฀Student฀Evaluations฀of฀
Teaching:฀A฀Factor฀Analysis฀Approach
MICHAEL฀M.฀BARTH
THE฀CITADEL
CHARLESTON,฀SOUTH฀CAROLINA

ABSTRACT.฀The฀author฀examined฀the฀
student฀evaluation฀of฀teaching฀instrument฀
used฀in฀the฀College฀of฀Business฀Administration฀at฀Georgia฀Southern฀University฀(GSU)฀
to฀determine฀which฀traits฀have฀the฀greatest฀
impact฀on฀students’฀overall฀rating฀of฀all฀individual฀instructors.฀Using฀exploratory฀factor฀

analysis,฀the฀author฀found฀that฀the฀overall฀
instructor฀rating฀is฀primarily฀driven฀by฀the฀
quality฀of฀instruction.฀Although฀the฀results฀of฀
this฀research฀apply฀specifically฀to฀the฀survey฀
instrument฀used฀at฀GSU,฀these฀techniques฀
can฀be฀applied฀to฀evaluate฀the฀instructor฀rating฀instruments฀at฀other฀institutions.
Keyword:฀educational฀evaluation
Copyright฀©฀2008฀Heldref฀Publications

40฀

Journal฀of฀Education฀for฀Business

T

he฀purpose฀of฀this฀research฀was฀to฀
determine฀which฀aspects฀of฀the฀student฀evaluation฀of฀teaching฀(SET)฀instrument฀used฀at฀Georgia฀Southern฀University฀฀
(GSU)฀ have฀ the฀ greatest฀ impact฀ on฀ the฀
overall฀ instructor฀ rating.฀ For฀ part฀ of฀ its฀
instructor฀evaluation฀process,฀GSU฀uses฀a฀

student฀survey฀to฀evaluate฀all฀instructors฀
in฀ every฀ course฀ offered฀ each฀ semester.฀
The฀ questions฀ in฀ the฀ survey฀ instrument฀
ask฀ about฀ the฀ workload,฀ clarity฀ of฀ the฀
materials,฀the฀instructor’s฀delivery,฀prior฀
interest฀by฀the฀student,฀and฀other฀general฀
questions฀ to฀ elicit฀ more฀ detail฀ from฀ the฀
students฀ about฀ why฀ they฀ liked฀ (or฀ did฀
not฀like)฀the฀course฀and฀the฀instructor.฀To฀
preserve฀student฀anonymity,฀administrators฀ conduct฀ the฀ surveys฀ during฀ normal฀
class฀time฀while฀the฀instructor฀is฀out฀of฀
the฀room,฀and฀the฀results฀are฀shared฀with฀
the฀instructor฀only฀after฀the฀semester฀has฀
ended.฀ Students฀ answer฀ the฀ 20฀ survey฀
questions฀on฀a฀standard฀Likert-type฀scale฀
ranging฀ from฀ 1฀ (very฀ poor)฀ to฀ 5฀ (very฀
good).฀There฀ is฀ additional฀ space฀ on฀ the฀
back฀ of฀ the฀ survey฀ instrument฀ for฀ students’฀written฀comments.฀
The฀ questions฀ in฀ the฀ survey฀ form฀ of฀
GSU’s฀ SET฀ are฀ shown฀ in฀ Table฀ 1,฀ with฀

median฀ scores฀ for฀ each฀ question.฀ Many฀
of฀the฀responses฀to฀the฀20฀questions฀in฀the฀
survey฀ instrument฀ tend฀ to฀ be฀ highly฀ correlated฀with฀one฀another,฀making฀it฀harder฀
for฀ researchers฀ to฀ evaluate฀ differences฀
from฀instructor฀to฀instructor.฀Thus,฀administrators฀tend฀to฀focus฀on฀a฀single฀question฀

—for฀ example,฀ Question฀ 18,฀ “Overall,฀
how฀ would฀ you฀ rate฀ this฀ instructor?”฀
—rather฀than฀using฀all฀of฀the฀information฀
available฀in฀the฀survey฀responses.
The฀ SETs฀ are฀ an฀ important฀ element฀
of฀ the฀ annual฀ evaluation฀ of฀ individual฀
faculty฀members฀at฀GSU฀and฀other฀institutions.฀ The฀ SETs฀ also฀ feature฀ prominently฀ in฀ the฀ tenure฀ and฀ promotion฀
process.฀Because฀administrators฀weight฀
the฀ SET฀ heavily,฀ rightly฀ or฀ wrongly,฀ it฀
is฀important฀for฀faculty฀and฀administrators฀to฀have฀a฀clear฀understanding฀of฀the฀
determinants฀ of฀ the฀ ratings,฀ especially฀
with฀ regard฀ to฀ Question฀ 18฀ of฀ the฀ survey,฀ which฀ elicits฀ the฀ student’s฀ overall฀
rating฀of฀the฀instructor.฀
This฀research฀provides฀a฀more฀refined฀

assessment฀ of฀ the฀ GSU฀ SET฀ survey฀
results฀by฀analyzing฀the฀underlying฀factors฀ that฀ drive฀ the฀ overall฀ rating฀ of฀ the฀
instructor.฀ Providing฀ faculty฀ members฀
with฀ more฀ information฀ on฀ the฀ determinants฀ of฀ instructor฀ ratings฀ should฀ help฀
to฀eliminate฀some฀of฀the฀mistrust฀inherent฀in฀the฀current฀system.฀Although฀this฀
research฀focuses฀on฀the฀instrument฀that฀
administrators฀ use฀ at฀ GSU,฀ the฀ techniques฀ that฀ I฀ use฀ to฀ evaluate฀ the฀ GSU฀
survey฀ instrument฀ are฀ applicable฀ to฀
other฀institutions.
RELATED฀LITERATURE
Because฀of฀the฀relatively฀high฀weight฀
that฀student฀ratings฀of฀instruction฀possess฀฀

TABLE฀1.฀Student฀Evaluation฀of฀Teaching฀Questions฀and฀Median฀Scores฀

Downloaded by [Universitas Maritim Raja Ali Haji] at 22:43 11 January 2016

฀
Survey฀question฀
Q฀1฀

Q฀2฀
Q฀3฀
Q฀4฀
Q฀5฀
Q฀6฀฀
Q฀7฀
Q฀8฀
Q฀9฀
Q฀10฀
Q฀11฀
Q฀12฀
Q฀13฀
Q฀14฀
Q฀15฀
Q฀16฀
Q฀17฀
Q฀18฀
Q฀19฀
Q฀20฀
฀

Mdn
฀score
How฀much฀effort฀did฀you฀put฀into฀learning฀the฀material฀covered฀in฀this฀course?฀
How฀much฀did฀you฀learn฀in฀this฀course?฀
To฀what฀degree฀were฀you฀intellectually฀challenged฀in฀this฀course?฀
How฀often฀did฀you฀seek฀outside฀help฀with฀this฀course?฀
How฀difficult฀was฀this฀course?฀
How฀was฀the฀workload฀for฀this฀course?฀
Overall,฀how฀would฀you฀rate฀this฀course?฀
The฀degree฀to฀which฀important฀points฀were฀stressed฀in฀this฀course.฀
The฀instructor’s฀preparation฀for฀this฀course.฀
The฀instructor’s฀encouragement฀of฀class฀participation,฀discussion,฀or฀questions.฀
The฀organization฀of฀the฀course฀material.฀
The฀clarity฀of฀the฀presentation฀of฀the฀course฀material.฀
The฀degree฀to฀which฀tests฀and฀other฀graded฀activities฀reflected฀course฀content.฀
The฀instructor’s฀availability฀to฀students.฀
The฀instructor’s฀helpfulness฀to฀students.฀
The฀degree฀to฀which฀the฀class฀stayed฀focused฀on฀course฀objectives.฀
The฀instructor’s฀interest฀in฀the฀content฀of฀this฀course.฀

Overall,฀how฀would฀you฀rate฀this฀instructor?฀
Your฀level฀of฀interest฀in฀this฀subject฀matter฀before฀taking฀this฀course?฀
Your฀level฀of฀interest฀in฀this฀subject฀matter฀after฀taking฀this฀course?฀
What฀grade฀do฀you฀expect฀to฀receive฀in฀this฀class฀(A,฀B,฀C,฀D,฀or฀F)?

3.8
3.6
4.0
3.3
3.9
3.4
3.6
3.9
4.2
3.9
4.0
3.7
3.9
4.0
3.9

4.0
4.3
4.0
2.5฀
2.9

฀฀

Note.฀Q฀=฀question.฀Survey฀responses฀are฀based฀on฀a฀Likert-type฀scale฀ranging฀from฀1฀(very฀poor)฀to฀5฀(very฀good).฀

in฀ the฀ U.S.฀ collegiate฀ system,฀ the฀ academic฀research฀in฀this฀area฀is฀vast,฀accumulating฀ for฀ more฀ than฀ seven฀ decades.฀
There฀are฀a฀number฀of฀studies฀in฀print,฀
and฀ more฀ emerge฀ every฀ day.฀ The฀ present฀ literature฀ review฀ is฀ purposely฀ brief฀
and฀ concise,฀ but฀ I฀ direct฀ readers฀ to฀ the฀
summaries฀ of฀ the฀ existing฀ literature฀
that฀both฀Germain฀and฀Scandura฀(2005)฀
and฀ Crumbley฀ and฀ Fliedner฀ (2002)฀฀
have฀provided.
Griffin฀(2003)฀reported฀that฀the฀bulk฀
of฀ the฀ academic฀ research฀ showed฀ that฀

faculty฀ support฀ the฀ use฀ of฀ SET฀ surveys฀as฀a฀tool฀for฀improving฀teaching,฀
but฀that฀faculty฀are฀also฀uncomfortable฀
with฀ the฀ use฀ of฀ the฀ SET฀ surveys฀ as฀
part฀ of฀ their฀ annual฀ evaluation฀ process.฀Also,฀Yunker฀ and฀Yunker฀ (2003)฀
reported฀ that฀ the฀ majority฀ of฀ faculty฀
support฀the฀validity฀of฀SETs,฀although฀
they฀acknowledged฀that฀there฀is฀a฀great฀
deal฀ of฀ faculty฀ discomfort฀ with฀ the฀
SET฀ practices฀ at฀ many฀ universities.฀
Germain฀ and฀ Scandura฀ (2005)฀ provided฀additional฀discussion฀of฀(a)฀the฀pros฀
and฀ cons฀ of฀ student฀ evaluations฀ and฀
(b)฀faculty฀concerns฀about฀the฀validity฀
of฀ SETs.฀ Moore฀ (2006)฀ reported฀ that฀
SETs฀ are฀ in฀ general฀ effective฀ measures฀ of฀ performance.฀ At฀ least฀ some฀
฀

of฀ the฀ faculty฀ discomfort฀ with฀ SETs฀
must฀ stem฀ from฀ a฀ lack฀ of฀ trust฀ in฀ the฀
survey฀ instrument’s฀ ability฀ to฀ discern฀
quality฀teaching฀from฀popularity.฀Also,฀

because฀of฀the฀high฀degree฀of฀correlation฀among฀the฀answers฀to฀the฀individual฀ questions,฀ it฀ becomes฀ difficult฀ for฀
researchers฀ and฀ educators฀ to฀ decipher฀
the฀underlying฀meaning฀of฀the฀student฀
responses.
Some฀universities฀provide฀benchmark฀
scores,฀ whereas฀ others฀ do฀ not,฀ and฀ the฀
lack฀ of฀ a฀ benchmark฀ may฀ increase฀ the฀
level฀ of฀ discomfort.฀ Those฀ instructors฀
with฀higher฀ratings฀tend฀to฀favor฀the฀current฀student฀evaluation฀system,฀whereas฀
those฀with฀lower฀ratings฀tend฀to฀be฀dismissive฀ of฀ it฀ (Crumbley฀ &฀ Fliedner,฀
2002).฀Some฀faculty฀members฀feel฀that฀
higher฀ratings฀are฀associated฀with฀grade฀
inflation฀and฀lower฀academic฀standards฀
(Germain฀ &฀ Scandura,฀ 2005;฀ Griffin,฀
2004).฀Also,฀some฀faculty฀members฀have฀
criticized฀ the฀ student฀ ratings฀ because฀
they฀ assume฀ that฀ the฀ more฀ quantitative฀ business฀ courses,฀ such฀ as฀ statistics฀
and฀ finance,฀ generate฀ lower฀ evaluation฀
scores฀ that฀ reflect฀ the฀ topic฀ and฀ not฀
the฀ instructor฀ (Stapleton฀ &฀ Murkison,฀
2001).฀ If฀ these฀ complaints฀ are฀ valid,฀
then฀the฀use฀of฀the฀student฀surveys฀may฀

have฀an฀unintended฀negative฀impact฀on฀
merit฀pay฀and฀promotion฀for฀the฀faculty฀
members฀who฀teach฀in฀those฀areas.฀
One฀ line฀ of฀ research฀ has฀ focused฀ on฀
the฀relations฀between฀faculty฀demographics฀ and฀ SETs.฀ Researchers฀ have฀ found฀
links฀between฀nonteaching฀factors,฀such฀
as฀physical฀appearance,฀and฀SET฀scores.฀
Rinolo,฀ Johnson,฀ Sherman,฀ and฀ Misso฀
(2006)฀provided฀a฀historical฀summary฀of฀
the฀ research฀ along฀ this฀ axis฀ and฀ reported฀ that฀ studies฀ have฀ shown฀ that฀ physically฀attractive฀instructors฀(regardless฀of฀
whether฀they฀are฀male฀or฀female)฀receive฀
higher฀ ratings฀ than฀ their฀ less฀ attractive฀
colleagues.฀Felton,฀Mitchell,฀and฀Stinson฀
(2004)฀ reported฀ that฀ instructor฀ ratings฀
online฀ at฀ ratemyprofessors.com฀ highly฀
correlated฀with฀“hotness,”฀although฀Felton,฀Mitchell,฀and฀Stinton฀did฀not฀strictly฀
define฀the฀term฀(p.฀3).
Researchers฀ have฀ often฀ cited฀ lax฀
grading฀ standards฀ as฀ an฀ inflator฀ of฀ student฀ evaluations,฀ although฀ the฀ evidence฀
to฀ date฀ has฀ been฀ inconclusive฀ about฀
whether฀ lax฀ grading฀ standards฀ increase฀
evaluation฀ scores.฀ Two฀ recent฀ examples฀
include฀McPherson฀(2006),฀who฀reported฀
that฀grade฀inflation฀increases฀SETs,฀and฀
Moore฀(2006),฀who฀found฀that฀SETs฀were฀
not฀ easily฀ manipulated฀ through฀ grade฀฀
September/October฀2008฀

41

Downloaded by [Universitas Maritim Raja Ali Haji] at 22:43 11 January 2016

inflation.฀ The฀ perceived฀ link฀ between฀
higher฀ grades฀ and฀ higher฀ student฀ evaluations฀has฀raised฀concerns฀among฀many฀
faculty,฀ and฀ there฀ is฀ some฀ evidence฀ that฀
expected฀ grades฀ or฀ actual฀ grades฀ positively฀correlate฀with฀student฀evaluations.฀
However,฀ in฀ many฀ of฀ these฀ studies฀ the฀
researchers฀made฀no฀effort฀to฀determine฀
whether฀ the฀ students฀ had฀ earned฀ the฀
higher฀ grades.฀ Students฀ should฀ expect฀
to฀ earn฀ higher฀ grades฀ from฀ excellent฀
teachers฀ than฀ from฀ mediocre฀ teachers.฀
Higher฀grades฀are฀not฀the฀same฀as฀higher฀
unearned฀ grades,฀ although฀ much฀ of฀ the฀
discussion฀about฀grade฀inflation฀seems฀to฀
suggest฀that฀the฀two฀are฀synonymous.
Marsh฀ and฀ Roche฀ (2000)฀ found฀ a฀
stronger฀link฀between฀SETs฀and฀instructor฀performance฀than฀between฀SETs฀and฀
grades.฀ Prior฀ research฀ has฀ also฀ shown฀
a฀ correlation฀ between฀ faculty฀ evaluations฀of฀faculty฀(e.g.,฀peer฀reviews)฀and฀
student฀ evaluations฀ of฀ faculty฀ teaching,฀ although฀ there฀ are฀ some฀ lingering฀
questions฀ about฀ bias฀ in฀ those฀ results฀
(Hobson฀ &฀ Talbot,฀ 2001).฀Yunker฀ and฀
Yunker฀(2003)฀crafted฀a฀study฀to฀evaluate฀subsequent฀performance฀in฀intermediate฀ accounting฀ to฀ determine฀ whether฀
the฀ highly฀ rated฀ instructors฀ of฀ a฀ basic฀
accounting฀ course฀ had฀ better฀ performing฀ students฀ in฀ the฀ follow-on฀ courses.฀
Stapleton฀ and฀ Murkison฀ (2001)฀ also฀
tried฀ to฀ measure฀ performance฀ against฀
student฀ evaluations฀ by฀ using฀ a฀ single฀
SET฀measure฀of฀overall฀instructor฀quality฀ rather฀ than฀ the฀ full฀ set฀ of฀ survey฀
questions.฀Isely฀and฀Singh฀(2005)฀used฀
a฀ fixed฀ effects฀ model฀ to฀ control฀ for฀
instructor-฀ and฀ course-specific฀ differences฀in฀a฀sample฀of฀260฀economics฀and฀
finance฀ classes.฀ Isely฀ and฀ Singh฀ found฀
a฀positive฀relation฀between฀their฀dependent฀ variable,฀ the฀ average฀ values฀ for฀
25฀ different฀ instructor฀ evaluation฀ questions,฀and฀relative฀expected฀grades.฀One฀
of฀the฀limitations฀of฀their฀study฀is฀that,฀
although฀ they฀ have฀ a฀ detailed฀ survey฀
instrument฀ with฀ multiple฀ questions฀ on฀
various฀aspects฀of฀teaching฀quality,฀they฀
did฀not฀use฀the฀full฀value฀of฀the฀information฀ in฀ the฀ SETs฀ to฀ measure฀ effectiveness.฀Multipart฀SET฀survey฀instruments฀
measure฀ a฀ variety฀ of฀ instructor฀ and฀
course฀ traits,฀ thus฀ simplistic฀ measures฀
such฀as฀the฀overall฀instructor฀rating฀can฀
obfuscate฀ more฀ than฀ illuminate฀ in฀ this฀
type฀of฀empirical฀study.
42฀

Journal฀of฀Education฀for฀Business

Germain฀ and฀ Scandura฀ (2005)฀
reviewed฀a฀number฀of฀studies฀of฀bias฀in฀
student฀ ratings฀ and฀ suggested฀ that฀ more฀
careful฀ construction฀ of฀ the฀ student฀ rating฀of฀instruction฀survey฀instrument฀may฀
alleviate฀ some฀ of฀ the฀ unintended฀ biases฀
built฀ into฀ simplistic฀ student฀ evaluations.฀
Consequently,฀many฀universities฀(including฀ GSU)฀ have฀ sought฀ to฀ improve฀ the฀
evaluation฀ process฀ by฀ moving฀ to฀ more฀
complex฀ rating฀ instruments.฀ However,฀
asking฀ more฀ questions฀ does฀ not฀ necessarily฀make฀the฀instrument฀more฀useful.฀
If฀researchers฀and฀educators฀cannot฀fully฀
interpret฀the฀results,฀the฀additional฀questions฀add฀more฀confusion฀than฀clarity.
Toland฀and฀De฀Ayala฀(2005)฀discussed฀
the฀ various฀ research฀ efforts฀ that฀ have฀
produced฀ multidimensional฀ SET฀ instruments฀over฀the฀years.฀They฀cited฀Marsh’s฀
(1987)฀ 35-question฀ survey฀ instrument฀
that฀ addresses฀ nine฀ latent฀ factors฀ dealing฀ with฀ instructional฀ quality.฀ Those฀
nine฀factors฀are฀learning/value,฀instructor฀
enthusiasm,฀ organization/clarity,฀ group฀
interaction,฀individual฀rapport,฀breadth฀of฀
coverage,฀ examination/grading,฀ assignments/readings,฀ and฀ workload/difficulty.฀
Toland฀and฀De฀Ayala฀then฀proposed฀a฀27question฀ instrument฀ that฀ they฀ designed฀
around฀ three฀ latent฀ factors:฀ instructor฀
course฀ delivery,฀ instructor/student฀ interaction,฀ and฀ regulating฀ student฀ learning.฀
Toland฀and฀De฀Ayala฀used฀factor฀analysis฀
in฀the฀design฀and฀analysis฀of฀their฀instrument,฀ but฀ their฀ relatively฀ small฀ sample฀
size฀constricted฀their฀conclusions.
METHOD
Although฀ a฀ multidimensional฀ SET฀
survey฀instrument฀provides฀more฀information฀by฀which฀to฀evaluate฀teaching,฀it฀
also฀makes฀the฀evaluation฀more฀difficult฀
without฀steps฀to฀break฀down฀the฀results฀
for฀independent฀evaluation฀of฀the฀scores฀
on฀ the฀ various฀ dimensions฀ to฀ be฀ evaluated.฀Factor฀analysis฀is฀a฀statistical฀technique฀ that฀ can฀ take฀ highly฀ correlated฀
data,฀ such฀ as฀ those฀ of฀ the฀ responses฀ to฀
SET฀ survey฀ questions,฀ and฀ reconfigure฀
them฀ so฀ that฀ they฀ provide฀ an฀ objective฀
measure฀of฀the฀underlying฀traits฀that฀are฀
most฀ valued฀ by฀ students.฀When฀ faculty฀
members฀ better฀ understand฀ the฀ survey฀
responses,฀they฀should฀be฀able฀to฀develop฀ greater฀ acceptance฀ of฀ the฀ results฀ of฀
the฀ SET.฀ If฀ instructors฀ continue฀ to฀ dis-

miss฀ the฀ SET฀ as฀ simply฀ a฀ reflection฀ of฀
grade฀ inflation฀ or฀ a฀ popularity฀ contest,฀
then฀those฀instructors฀will฀never฀receive฀
the฀ feedback฀ from฀ the฀ SET,฀ including฀
information฀ about฀ the฀ traits฀ that฀ students฀truly฀value฀in฀a฀teacher.
In฀ the฀ present฀ research,฀ I฀ used฀ factor฀ analysis฀ to฀ decompose฀ the฀ SET฀
responses฀ from฀ the฀ survey฀ instrument฀
that฀ educators฀ used฀ at฀ GSU’s฀ College฀
of฀ Business฀Administration฀ to฀ provide฀
a฀better฀understanding฀of฀the฀traits฀that฀
students฀ rate฀ highest฀ in฀ their฀ overall฀
evaluation฀of฀an฀instructor.฀There฀were฀
two฀ steps฀ in฀ the฀ research฀ design:฀ (a)฀
develop฀a฀set฀of฀measures฀of฀the฀traits฀
that฀ are฀ latent฀ in฀ the฀ university’s฀ SET฀
instrument฀and฀(b)฀use฀multiple฀regression฀ analysis฀ to฀ evaluate฀ how฀ those฀
latent฀traits฀affect฀the฀value฀of฀Question฀
18,฀the฀overall฀instructor฀rating.
I฀ gathered฀ the฀ average฀ responses฀ by฀
class฀ for฀ each฀ of฀ the฀ 20฀ SET฀ questions฀
for฀four฀courses฀(see฀Table฀1):฀principles฀
of฀corporate฀finance,฀operations฀management,฀business฀statistics,฀and฀quantitative฀
methods.฀ One฀ of฀ the฀ reasons฀ for฀ using฀
these฀particular฀four฀courses฀was฀restriction฀on฀data฀availability,฀but฀another฀reason฀was฀comparability.฀These฀courses฀are฀
prerequisites฀ for฀ the฀ capstone฀ business฀
course,฀ and฀ all฀ are฀ somewhat฀ quantitative฀ in฀ design.฀Also,฀ unlike฀ some฀ of฀ the฀
other฀ courses฀ in฀ the฀ common฀ business฀
core,฀there฀are฀relatively฀few฀nonbusiness฀
majors฀taking฀these฀courses.฀
The฀present฀data฀represent฀33฀different฀
instructors฀ over฀ a฀ 3-year฀ period,฀ ranging฀
from฀adjunct฀instructors฀to฀tenured฀professors.฀For฀the฀most฀part,฀instructors฀taught฀
in฀ only฀ one฀ of฀ the฀ four฀ functional฀ areas฀
(finance,฀ operations฀ management,฀ business฀statistics,฀and฀quantitative฀methods),฀
although฀ a฀ handful฀ of฀ instructors฀ taught฀
in฀more฀than฀one฀area.฀In฀addition฀to฀the฀
class฀ average฀ scores฀ from฀ the฀ SET฀ surveys,฀I฀computed,฀(a)฀the฀expected฀grade฀
point฀average฀(GPA)฀for฀each฀section฀from฀
the฀ survey฀ instrument฀ and฀ (b)฀ the฀ actual฀
GPA฀ for฀ the฀ class฀ from฀ the฀ university’s฀
grade-history฀ database.฀ I฀ omitted฀ Question฀18฀because฀it฀was฀the฀dependent฀variable฀in฀stage฀two฀of฀the฀research,฀and฀there฀
were฀ 21฀ original฀ variables฀ to฀ include฀ in฀
the฀factor฀analysis.
To฀ be฀ included฀ in฀ the฀ analysis,฀ the฀
average฀class฀score฀for฀the฀questions฀in฀
the฀SET฀had฀to฀be฀based฀on฀at฀least฀15฀

Downloaded by [Universitas Maritim Raja Ali Haji] at 22:43 11 January 2016

student฀ responses.฀ The฀ lower฀ limit฀ on฀
the฀number฀of฀responses฀was฀a฀trade-off฀
between฀degrees฀of฀freedom฀and฀stability฀ of฀ the฀ average฀ response฀ values฀ for฀
each฀ of฀ the฀ 20฀ questions฀ because,฀ for฀
example,฀the฀average฀class฀response฀in฀a฀
class฀that฀had฀only฀3฀students฀would฀not฀
be฀ comparable฀ with฀ the฀ average฀ class฀
response฀ in฀ a฀ class฀ of฀ 30฀ students.฀ A฀
total฀of฀167฀usable฀observations฀(classes)฀ were฀ used฀ in฀ the฀ factor฀ analysis.฀
These฀ 167฀ classes฀ represented฀ the฀ collective฀evaluations฀of฀more฀than฀30฀different฀instructors,฀based฀on฀the฀surveys฀
from฀more฀than฀4,000฀students.฀Because฀
these฀ classes฀ are฀ part฀ of฀ the฀ common฀
business฀ core,฀ many—if฀ not฀ most—of฀
the฀ surveys฀ represent฀ the฀ same฀ set฀ of฀
students฀ who฀ were฀ evaluating฀ the฀ various฀instructors.
Factor฀analysis฀is฀a฀statistical฀method฀
that฀ researchers฀ can฀ use฀ to฀ reduce฀ the฀
dimensions฀ of฀ a฀ variable฀ set฀ of฀ highly฀
correlated฀ data฀ into฀ a฀ smaller฀ subset฀
of฀ factors฀ that฀ are฀ themselves฀ linear฀
composites฀ of฀ the฀ original฀ variables.฀
Simply,฀it฀is฀a฀data฀reduction฀technique.฀
The฀ factors฀ that฀ the฀ analysis฀ generates฀
are฀orthogonal฀to฀one฀another,฀but฀they฀
still฀ contain฀ most฀ of฀ the฀ information฀
from฀ the฀ original฀ variable฀ set.฀ Because฀
the฀ practical฀ use฀ of฀ factor฀ analysis฀ is฀
to฀ reduce฀ a฀ large฀ number฀ of฀ correlated฀
variables฀into฀a฀smaller฀subset฀of฀uncorrelated฀variables,฀the฀number฀of฀factors฀
that฀ the฀ factor฀ analysis฀ process฀ retains฀
is฀almost฀always฀fewer฀than฀the฀number฀
of฀ original฀ variables.฀ The฀ number฀ of฀
factors฀that฀are฀retained฀depends฀on฀the฀
dimensionality฀ of฀ the฀ original฀ data฀ and฀
the฀ ability฀ of฀ the฀ analyst฀ to฀ interpret฀
the฀ resulting฀ factors.฀ Ideally,฀ following฀
an฀ orthogonal฀ rotation฀ procedure,฀ each฀
of฀ the฀ resulting฀ factors฀ includes฀ data฀
from฀ several฀ of฀ the฀ original฀ variables,฀
and฀ each฀ of฀ the฀ original฀ variables฀ will฀
be฀included฀in฀only฀one฀of฀the฀resulting฀
factors.฀ In฀ practice,฀ a฀ variable฀ may฀ be฀
associated฀ with฀ more฀ than฀ one฀ factor,฀
which฀can฀make฀the฀factors฀more฀difficult฀to฀interpret.฀The฀factors฀are฀orthogonal฀ to฀ one฀ another฀ and฀ can฀ therefore฀
be฀ used฀ as฀ the฀ dependent฀ variables฀ in฀
a฀ regression฀ analysis฀ without฀ violating฀
the฀multicollinearity฀assumption.
I฀ conducted฀ this฀ analysis฀ by฀ using฀
SAS,฀a฀comprehensive฀statistical฀analysis฀system฀that฀includes฀specific฀subrou฀

tines฀for฀hundreds฀of฀statistical฀applications.฀ The฀ SAS฀ OnlineDoc฀ 9.13฀ (SAS฀
Institute,฀ 2004),฀ the฀ online฀ user฀ guide฀
for฀ the฀ SAS/STAT฀ software฀ product,฀
provides฀ (a)฀ an฀ overview฀ of฀ the฀ factor฀ analysis฀ concept฀ and฀ (b)฀ detailed฀
and฀ specific฀ instructions฀ on฀ alternative฀
methods฀ for฀ conducting฀ the฀ analysis.฀
The฀ subroutine฀ for฀ conducting฀ a฀ factor฀
analysis฀ is฀ known฀ as฀ PROC฀ FACTOR฀
and฀is฀described฀in฀SAS฀OnlineDoc฀9.13฀
(SAS฀Institute).
RESULTS
Factor฀Analysis฀Results
One฀of฀the฀decisions฀that฀an฀analyst฀
must฀make฀as฀part฀of฀the฀factor฀analysis฀ process฀ is฀ the฀ number฀ of฀ factors฀
to฀ retain.฀ Two฀ general฀ rules฀ are฀ commonly฀applied:฀the฀minimum-eigenvalue-of-one฀ rule฀ and฀ the฀ scree฀ diagram.฀
The฀eigenvalue฀rule฀requires฀that฀each฀
factor฀included฀explain฀at฀least฀as฀much฀
information฀ as฀ the฀ original฀ variables.฀
Therefore,฀only฀factors฀that฀generate฀an฀
eigenvalue฀of฀1.0฀or฀higher฀are฀retained.฀
The฀second฀method฀is฀an฀evaluation฀of฀
a฀scree฀diagram,฀which฀is฀a฀line฀plot฀of฀
the฀ eigenvalues.฀ Using฀ this฀ approach,฀
the฀ analyst฀ looks฀ for฀ a฀ point฀ at฀ which฀
the฀eigenvalues฀become฀level,฀meaning฀
that฀ they฀ are฀ explaining฀ less฀ and฀ less฀
of฀ the฀ total฀ variability.฀ A฀ third฀ criterion฀ that฀ researchers฀ use฀ in฀ practice฀ is฀
that฀ each฀ of฀ the฀ resulting฀ factors฀ must฀
be฀ interpretable฀ by฀ the฀ analyst.฀ In฀ the฀
present฀ analysis,฀ both฀ the฀ minimumeigenvalue-of-one฀ rule฀ and฀ the฀ scree฀
diagram฀ indicated฀ that฀ either฀ five฀ or฀
six฀factors฀would฀be฀appropriate.฀After฀
evaluating฀the฀results฀for฀both฀the฀fivefactor฀model฀and฀the฀six-factor฀model,฀
I฀ determined฀ the฀ five-factor฀ set฀ to฀ be฀
the฀most฀interpretable.
Table฀2฀shows฀each฀of฀the฀five฀factors฀
and฀its฀correlation฀with฀the฀original฀set฀
of฀ variables.฀ For฀ the฀ factor฀ analysis฀ to฀
be฀ interpretable,฀ each฀ of฀ the฀ original฀
variables฀ should฀ highly฀ correlate฀ with฀
only฀one฀or฀two฀of฀the฀resulting฀factors.฀
By฀ examining฀ which฀ of฀ the฀ original฀
variables฀load฀on฀which฀factor,฀researchers฀can฀identify฀the฀latent฀traits฀that฀each฀
factor฀represents.฀Theoretically,฀the฀factor฀ analysis฀ can฀ generate฀ as฀ many฀ factors฀as฀there฀are฀variables฀in฀the฀original฀

data฀ set,฀ but฀ the฀ goal฀ is฀ to฀ generate฀ a฀
reduced฀data฀set฀that฀eliminates฀the฀multicollinearity฀ problems฀ associated฀ with฀
the฀original฀data.฀
Factor฀ 1฀ had฀ a฀ high฀ level฀ of฀ positive฀
correlation฀ with฀ the฀ questions฀ concerning฀instructor฀preparation,฀clarity฀of฀presentation,฀ relevance฀ of฀ the฀ material฀ to฀
learning,฀ and฀ the฀ instructor’s฀ focus฀ on฀
course฀ objectives.฀ I฀ labeled฀ this฀ factor฀
as฀ quality฀ of฀ instruction,฀ and฀ it฀ generally฀ measures฀ (a)฀ the฀ degree฀ to฀ which฀
the฀ students฀ felt฀ that฀ the฀ instructor฀ was฀
prepared฀ and฀ (b)฀ their฀ perceptions฀ of฀
the฀overall฀quality฀of฀the฀presentation.฀If฀฀
the฀instructor฀is฀well฀prepared,฀and฀if฀the฀
course฀ content฀ and฀ objectives฀ are฀ clear฀
to฀the฀students,฀researchers฀would฀expect฀
them฀to฀feel฀that฀the฀course฀is฀organized,฀
well฀structured,฀and฀valuable.
Factor฀ 2,฀ which฀ I฀ labeled฀ as฀ course฀
rigor,฀was฀highly฀correlated฀with฀those฀
survey฀questions฀that฀measure฀the฀rigor฀
of฀ the฀ coursework.฀ The฀ effort฀ of฀ the฀
student,฀ intellectual฀ challenge,฀ course฀
difficulty,฀ and฀ workload฀ on฀ the฀ student฀
all฀loaded฀positively฀on฀this฀factor.฀
Factor฀3฀measured฀the฀student’s฀level฀
of฀ interest฀ in฀ the฀ subject฀ matter฀ before฀
and฀ after฀ taking฀ the฀ course฀ (Questions฀
19฀and฀20).฀Note฀that฀Factor฀1฀was฀also฀
highly฀ loaded฀ on฀ Question฀ 20,฀ which฀
asked฀ about฀ the฀ after-course฀ interest฀ in฀
the฀ subject.฀ Therefore,฀ Factor฀ 3฀ probably฀measured฀the฀overall฀interest฀of฀the฀
student฀and฀the฀student’s฀attitude฀toward฀
the฀ subject฀ matter,฀ whereas฀ Factor฀ 1฀
picked฀up฀the฀increase฀in฀student฀interest฀after฀taking฀the฀class.
Factor฀ 4,฀ which฀ I฀ labeled฀ as฀ grades,฀
loaded฀highly฀on฀both฀the฀average฀expected฀GPA฀of฀the฀students฀in฀the฀class฀and฀
their฀actual฀average฀GPA.฀It฀is฀interesting฀
that,฀ although฀ these฀ two฀ GPA฀ measures฀
highly฀ correlated฀ with฀ one฀ another,฀ the฀
expected฀GPAs฀that฀students฀reported฀in฀
the฀ survey฀ were฀ virtually฀ always฀ higher฀
than฀ their฀ actual฀ class฀ average฀ GPAs.฀
This฀ situation฀ also฀ could฀ suggest฀ that฀
when฀ researchers฀ or฀ educators฀ omit฀ the฀
weaker฀students฀(those฀failing฀the฀class,฀
who฀ have฀ already฀ given฀ up฀ and฀ are฀ not฀
attending)฀from฀the฀survey,฀the฀resulting฀
evaluations฀ bias฀ upward.฀ The฀ practice฀
in฀ the฀ College฀ of฀ Business฀ Administration฀ at฀ GSU฀ is฀ that฀ the฀ individual฀ class฀
instructor฀chooses฀the฀actual฀day฀within฀
a฀4-week฀period฀on฀which฀to฀administer฀
September/October฀2008฀

43

TABLE฀2.฀Rotated฀Factor฀Pattern฀and฀Factor฀Loadings
฀

Downloaded by [Universitas Maritim Raja Ali Haji] at 22:43 11 January 2016

฀
Original฀variable฀
Q฀1฀
Q฀2฀
Q฀3฀
Q฀4฀
Q฀5฀
Q฀6฀
Q฀7฀
Q฀8฀
Q฀9฀
Q฀10฀
Q฀11฀
Q฀12฀
Q฀13฀
Q฀14฀
Q฀15฀
Q฀16฀
Q฀17฀
Q฀19฀
Q฀20฀
Actual฀GPA฀
Expected฀GPA฀

Factor฀1฀

Factor฀2฀

Factor฀3฀

Factor฀4฀

Factor฀5

Quality฀of฀฀
Instruction฀

Course฀฀
Rigor฀

Level฀of฀฀
Interest฀

฀
Grades฀

Instructor฀
Helpfulness฀

.00฀
.78฀
.25฀
–.03฀
–.21฀
–.13฀
.86฀
.94฀
.94฀
.74฀
.96฀
.95฀
.88฀
.73฀
.78฀
.93฀
.76฀
.06฀
.56฀
.10฀
.31฀

.93฀
.36฀
.86฀
.76฀
.85฀
.66฀
.01฀
–.01฀
–.03฀
–.19฀
–.03฀
–.10฀
–.02฀
–.17฀
–.22฀
.02฀
–.11฀
–.08฀
–.08฀
–.22฀
–.33฀

.08฀
.39฀
.06฀
–.06฀
–.08฀
–.16฀
.37฀
.14฀
.06฀
.24฀
.03฀
.10฀
.08฀
.00฀
.06฀
.00฀
.17฀
.83฀
.80฀
–.01฀
.10฀

–.12฀
.10฀
–.26฀
–.12฀
–.38฀
.05฀
.21฀
.14฀
.03฀
.04฀
.05฀
.06฀
.32฀
.20฀
.18฀
.08฀
.02฀
–.01฀
.10฀
.67฀
.69฀

–.13
–.03
–.07
.06
–.02
–.06
.03
.06
–.01
.40
–.03
.01
.08
.53
.52
.01
.30
–.01
.11
.06
.02

Note.฀Values฀in฀bold฀are฀the฀factor฀loadings฀greater฀than฀.4054.฀Q฀=฀survey฀question;฀GPA฀=฀grade฀point฀average.

the฀ survey฀ to฀ their฀ classes.฀ Anecdotal฀
evidence฀ suggests฀ that฀ some฀ instructors฀
have฀selectively฀chosen฀the฀date฀that฀they฀
administer฀the฀survey฀so฀as฀to฀manipulate฀
the฀results.฀For฀example,฀some฀educators฀
have฀ said฀ that฀ a฀ good฀ day฀ on฀ which฀ to฀
administer฀ the฀ SET฀ is฀ a฀ Friday฀ after฀ an฀
exam฀because฀lower฀attendance฀is฀likely,฀
especially฀ by฀ the฀ weaker฀ students.฀ Both฀
(a)฀ whether฀ this฀ proposed฀ interpretation฀
is฀actually฀true฀and฀(b)฀what฀its฀effect฀on฀
instructor฀ratings฀would฀be฀remain฀areas฀
for฀future฀study.฀
Factor฀ 5฀ was฀ highly฀ correlated฀ with฀
Questions฀ 14฀ and฀ 15,฀ concerning฀ the฀
instructor’s฀availability฀and฀willingness฀
to฀provide฀outside฀help฀to฀the฀students.฀
These฀ questions฀ also฀ highly฀ correlated฀

with฀Factor฀1,฀which฀measured฀the฀overall฀ course฀ quality.฀ Therefore,฀ Factor฀ 5฀
was฀ slightly฀ more฀ ambiguous.฀ There฀
was฀ also฀ a฀ relatively฀ high฀ loading฀ with฀
Question฀10,฀which฀measured฀the฀degree฀
to฀which฀the฀instructor฀encourages฀class฀
participation฀ and฀ questions.฀ Factor฀ 5฀
seemed฀ to฀ measure฀ some฀ aspect฀ of฀ the฀
instructor’s฀personality,฀approachability,฀
or฀ openness฀ with฀ the฀ students,฀ which฀ I฀
labeled฀as฀instructor฀helpfulness.
Multiple฀Regression฀Analysis
I฀ then฀ used฀ the฀ five฀ factor฀ scores฀ for฀
each฀course฀as฀explanatory฀variables฀in฀a฀
multiple฀regression฀model฀to฀evaluate฀the฀
impact฀of฀these฀five฀latent฀characteristics฀

on฀Question฀18฀of฀the฀survey฀instrument,฀
the฀ overall฀ rating฀ of฀ the฀ instructor.฀ The฀
SAS฀ factor฀ analysis฀ procedure฀ outputs฀
scores฀ that฀ are฀ standardized฀ (i.e.,฀ M฀ =฀
0,฀ SD฀ =฀ 1)฀ so฀ that฀ the฀ five฀ factors฀ are฀
all฀ on฀ the฀ same฀ unit฀ scale.฀ The฀ magnitude฀of฀the฀coefficient฀estimates฀from฀the฀
regression฀analysis฀is฀therefore฀relevant,฀
and฀comparisons฀between฀the฀coefficient฀
values฀can฀be฀made.
The฀regression฀results,฀which฀I฀report฀
in฀Table฀3,฀show฀that฀each฀of฀these฀five฀
factors฀has฀a฀strong฀statistically฀significant฀relation฀with฀the฀overall฀instructor฀
rating.฀For฀the฀model,฀r2฀is฀equal฀to฀.95,฀
indicating฀that฀the฀five฀factors฀explained฀
nearly฀all฀of฀the฀variation฀in฀the฀overall฀
instructor฀rating฀(Question฀18).฀Because฀

TABLE฀3.฀Multiple฀Regression฀Results
฀
Variable฀

฀
Label฀

Parameter฀
estimate฀

฀
SE฀

฀

Intercept฀
Factor฀1฀
Factor฀2฀
Factor฀3฀
Factor฀4฀
Factor฀5฀

฀
Quality฀of฀Instruction฀
Course฀Rigor฀
Level฀of฀Interest฀
Grades฀
Instructor฀Helpfulness฀

3.965฀
0.489฀
฀฀–0.075฀
0.057฀
0.094฀
0.135฀

0.0094฀
0.0094฀
0.0097฀
0.0096฀
0.0107฀
0.0100฀

฀฀423.63฀
฀฀฀฀51.53฀
฀฀฀฀–7.72฀
฀฀฀฀฀฀5.96฀
฀฀฀฀฀฀8.76฀
฀฀฀฀13.03฀

Note.฀The฀dependent฀variable฀was฀responses฀to฀Question฀18฀(overall฀instructor฀rating).

44฀

Journal฀of฀Education฀for฀Business

t฀

฀
p฀