08832323.2014.919896
Journal of Education for Business
ISSN: 0883-2323 (Print) 1940-3356 (Online) Journal homepage: http://www.tandfonline.com/loi/vjeb20
Developing Students’ Thought Processes for
Choosing Appropriate Statistical Methods
James Murray & Elizabeth Knowles
To cite this article: James Murray & Elizabeth Knowles (2014) Developing Students’ Thought
Processes for Choosing Appropriate Statistical Methods, Journal of Education for Business,
89:8, 389-395, DOI: 10.1080/08832323.2014.919896
To link to this article: http://dx.doi.org/10.1080/08832323.2014.919896
Published online: 04 Nov 2014.
Submit your article to this journal
Article views: 69
View related articles
View Crossmark data
Full Terms & Conditions of access and use can be found at
http://www.tandfonline.com/action/journalInformation?journalCode=vjeb20
Download by: [Universitas Maritim Raja Ali Haji]
Date: 11 January 2016, At: 20:48
JOURNAL OF EDUCATION FOR BUSINESS, 89: 389–395, 2014
Copyright Ó Taylor & Francis Group, LLC
ISSN: 0883-2323 print / 1940-3356 online
DOI: 10.1080/08832323.2014.919896
Developing Students’ Thought Processes
for Choosing Appropriate Statistical Methods
James Murray and Elizabeth Knowles
Downloaded by [Universitas Maritim Raja Ali Haji] at 20:48 11 January 2016
University of Wisconsin-La Crosse, La Crosse, Wisconsin, USA
Students often struggle to select appropriate statistical tests when investigating research
questions. The authors present a lesson study designed to make students’ thought processes
visible while considering this choice. The authors taught their students a way to organize
knowledge about statistical tests and observed its impact in the classroom and in students’
written work. The results from this intervention were mixed, but the authors discuss where
they found evidence for improvement in students’ performance and thought processes. The
classroom observations revealed that students have difficulty identifying variables and
understanding the precise use of statistical language.
Keywords: introductory statistics, lesson study, research methods
While students are trained to implement a variety of statistical tests in introductory statistics, it is still common for
them to struggle in subsequent classes when they must
apply this body of knowledge to select appropriate statistical tests to investigate a research question. This application
can challenge students because it requires an advanced
organization of knowledge that considers not only the purpose of the test, but also the number of variables, scale of
measurement, and characteristics of the samples. For the
expert, this complicated cognitive process has been developed and practiced over years. A student who is a novice in
statistics needs to be carefully guided through the decisionmaking process.
The purpose of this article is to share a teaching
approach we developed to help students organize their
knowledge of statistical techniques in a way that is conducive to choosing appropriate statistical tests. We present a
decision tree that maps 11 parametric and nonparametric
tests with four reflection questions that a student could ask
himself or herself concerning the data and nature of the
research question. We conducted a classroom investigation
in Fall 2011 and Spring 2012 that revealed students’
thought processes for selecting four of these statistical tests,
before and after presenting the decision tree. The areas of
confusion that we identify inform instructors of the pitfalls
Correspondence should be addressed to James Murray, University of
Wisconsin-La Crosse, Department of Economics, 1725 State Street, La
Crosse, WI 54601, USA. E-mail: jmurray@uwlax.edu
to address when presenting this content. We summarize our
students’ performance in selecting appropriate statistical
tests both before and after our treatment, and suggest teaching improvement strategies.
REVIEW OF LITERATURE
Knowledge organization is broadly recognized as an important consideration to improve student learning (Ausubel,
1978). Ambrose, Bridges, DiPietro, Lovett, and Norman
(2010) suggested that when students begin developing
knowledge organizations without guidance, the connections
are often weak, superficial, and few in number, and these
superficial connections can impede student learning. Statistics instructors can be more effective when they help students see various dimensions of and connections between
statistical tests in a research settting.
There is relatively little literature on organizing knowledge and training students to choose statistical tests. Du
Prel, Rohrig, Hommel, and Blettner (2010) examined some
common statistical techniques in medical studies and present their own decision trees that are more limited than what
is presented here. Their knowledge organization is confined
to two criteria: scale of measurement and whether the
research design is paired or not paired.
While little else in the literature addresses the specific
idea of organizing knowledge about statistical tests, many
papers speak more generally to the importance of developing
Downloaded by [Universitas Maritim Raja Ali Haji] at 20:48 11 January 2016
390
J. MURRAY AND E. KNOWLES
and investigating the critical thinking process involved in
the selection of appropriate methods. Garfield and Chance
(2000) noted that there is a growing movement in statistics
courses’ learning outcomes to focus on critical thinking
skills. Mvududu (2005) made the case that instructors should
help students take what they already know and use it to create new knowledge. He further suggested that instructors
should study how students think about the material they
encounter and investigate students’ thought processes on a
deeper level than through everyday communication.
The goal of this study was to explore students’ thought
processes as they considered the choice of an appropriate
statistical test in a research methods class, and to examine
the impact of presenting students with a way to organize
their knowledge of statistical tests. We used a lesson study
approach, which is a teaching improvement activity in
which instructors jointly develop a lesson to make student
learning visible. With this approach, instructors look deeply
into the teaching and learning process: the goals of a teaching activity, the design of the classroom instruction, understanding the students’ reception and mental processing of
the class content, and making evidence-based improvements in teaching (Cerbin, 2011). While it is a standard
practice in Japanese elementary education, lesson study has
more recently been adapted to higher education in the
United States. Cerbin and Kopp (2006) had a useful discussion of its application in higher education.
METHOD
We recognized that when experts determine statistical
approaches to a problem, they use a mental mapping which
includes many dimensions, such as univariate versus multivariate techniques, parametric versus nonparametric tests,
scale of measurement, and whether the purpose of the
research is to investigate differences, comovement, or independence. We developed a decision tree that makes these
considerations and connections explicit and visually portrays the questions that students could ask as they attempt
to determine an appropriate statistical test. Our goal for this
decision tree was to provide a way for students to organize
their knowledge of all the statistical methods that they learn
in the class and that they are likely to encounter in their
undergraduate business curriculum. The decision tree was
organized around four considerations: (a) the number of
variables involved in the research question, (b) the scale of
measurement for these variables, (c) the intent of the
research question (i.e., whether the purpose is to test for differences or a comovement between two variables), and (d)
whether the samples are independent or paired. We tested
the impact of the knowledge organization on students ability to identify when to apply four of the most commonly
used statistical tests. The complete decision tree is given in
Figure 1 and is the key instructional intervention that is the
focus of this paper.
We taught separate sections of a business research methods course in Fall 2011 and Spring 2012. In Fall 2011, we
conducted the investigation in three sections of the course.
Elizabeth Knowles taught one of these sections and James
Murray taught two. In Spring 2012, we conducted the
investigation in four sections of the course. Elizabeth
Knowles taught one of these sections and James Murray
taught three. Based on our collaborative effort preparing
for the statistics unit of the course, we each discussed the
previous four considerations throughout the statistics portion of our courses. While we had regular discussions on
our teaching approaches and used some shared examples in
our lectures, we did not make an explicit attempt to make
the presentation of all the material identical. Therefore, the
FIGURE 1. Decision tree. While we presented this decision tree in class, we only used the statistical tests in bold in the in-class exercises.
Downloaded by [Universitas Maritim Raja Ali Haji] at 20:48 11 January 2016
DEVELOPING THOUGHT PROCESS FOR CHOOSING STATISTICAL METHODS
measures of our students’ performance that we report in the
next section may include instructor-specific effects.
In both instructors’ classes and in both semesters, we
jointly developed a lesson on how to organize statistical
knowledge. The lesson included a brief review of statistical tests and an in-class exercise designed to make
students’ thought processes visible. The exercise
included four scenarios corresponding to four common
elementary statistical tests: (A) the one-sample t-test for
a mean, (B) the independent samples t-test for a difference in means, (C) the paired samples t-test for a difference in means, and (D) the chi-square test of
independence. Each scenario presented a description of
a single research question and survey questions that
could be used to collect data to answer the research
question. The exercise prompted students to choose a
statistical test and describe what considerations led them
to this decision. In the Findings section, we refer to
each of the four scenarios in the exercise with these
labels (A–D). The exercise is given in Appendix A.
To make students’ thought processes visible, we put
them in groups of three or four to discuss the exercise aloud
while they completed the task. We sat down with students
to observe their discussions, and we took notes on their
thought processes. We noted whether students discussed
the four questions in the decision tree, whether they used
any irrelevant considerations to reach their conclusion, and
whether they were able to articulate their reasoning well.
We observed the discussions in six groups in Fall 2011 and
eight groups in Spring 2012. The classroom observation
guide that we used to facilitate this notetaking is provided
in Appendix B.
Subsequently, each student wrote and submitted individual answers for the in-class exercise, which may have varied
from their group members. We reviewed the responses and
recorded the following: whether the students chose the correct statistical test, whether the students’ rationale included
any of the four questions emphasized in the decision tree,
and whether the questions were correctly identified.
In the Fall 2011 lesson, the students completed two of
the four scenarios in their in-class exercise before we presented the decision tree. After presenting the decision tree,
students were asked to put away their notes and they completed the final two scenarios. The intention was to provide
students a mental organization for statistical procedures,
and compare the students’ performance and thought processes before and after this intervention. In each section, we
administered the four scenarios in the in-class exercise in a
different order so that every scenario appeared both before
the decision tree was presented and after. This allowed us to
compare the impact the decision tree had on students’ performance and thought processes for each of the scenarios.
While this timing for presenting the decision tree was
useful for the research design, in Spring 2012 we determined that it would be more pedagogically valuable to
391
present the decision tree at the beginning of the statistics
unit of the course and continue to refer to it throughout the
three week unit. This helped students organize new knowledge as they learned it, and it gave students practice using
the decision tree before conducting the classroom observation and in-class exercise. Again in Spring 2012 we randomized the order of the four scenarios in the in-class
exercise for each section of the course.
To measure students’ retention of the knowledge organization, we gave our students an unannounced quiz in the
week following our classroom observation. In the quiz, we
asked students to reconstruct the decision tree from memory. We evaluated our students’ work and noted whether it
included each of the four questions in the decision tree, and
whether or not students arrived at each of the four statistical
tests with any incorrect considerations.
FINDINGS
Table 1 shows the percentage of students that correctly
identified the appropriate statistical test in each of the four
scenarios of the in-class exercise. For Fall 2011, results are
reported for before and after the decision tree was introduced. In Spring 2012, because the decision tree was developed before the in-class exercise was administered, the
percentage correct reflects all attempts. The results from
Fall 2011 indicate some general improvement after the
introduction of the decision tree. We expected that performance would be better in Spring 2012, but the results were
mixed, depending on the statistical test.
The data suggests that the paired-samples t-test was the
most difficult for students. Interestingly, none of the 22 students who were given this question before the decision tree
intervention in the Fall 2011 semester answered this
question correctly. The percentage correct for the chisquare test of independence actually decreased after the
decision tree intervention. In Fall 2011, 86% of students
answered this question correctly before the intervention,
and only 58% afterward. In Spring 2012, only 39% of students correctly identified the chi-square test. The most common incorrect answer for this question was a Pearson
correlation coefficient, another test that examines a relationship between two variables, but used for interval/ratio
data rather than categorical data.
Table 2 identifies the overall percentage of students that
considered, either correctly or incorrectly, each of the following factors in their written responses: (a) number of variables, (b) scale of measurement, (c) the intent of the
statistical test (i.e., whether the method tests for differences
or a comovement between two variables), and (d) whether
samples are independent or paired. The table also reports
the percentage of students who considered each factor correctly. For the most part, students considered the number of
variables and the scale of measurement of the variables.
392
J. MURRAY AND E. KNOWLES
TABLE 1
Correct Answers by Statistical Test
One sample (A)
Semester
Independent samples (B)
Paired samples (C)
Chi square (D)
Percent correct Sample size Percent correct Sample size Percent correct Sample size Percent correct Sample size
Fall 2011: Prior to decision tree
Fall 2011: Following decision tree
Spring 2012
77%
82%
92%
35
22
84
54%
100%
56%
35
22
84
0%
67%
47%
22
36
84
86%
58%
39%
22
36
84
Downloaded by [Universitas Maritim Raja Ali Haji] at 20:48 11 January 2016
Note: Sample size is number of students who completed the exercise. As students discussed the problems in groups before submitting their individual
responses, this does not represent a number of independent observations.
Considering whether or not samples were independent or
paired and whether the scenario suggested examining differences or a relationship occurred less frequently. In scenario B (independent samples t-test), in Fall 2011 before
introducing the decision tree, less than half of the students
(49%) considered whether a test for a difference or a relationship was appropriate, and about one fourth (26%) considered whether the samples were independent or paired.
Following the decision tree, the percentage of students considering difference versus relationship increased to 82%
and the percentage of students considering independent versus paired samples increased to 64%. This is associated
with an increase in overall performance on the question
from 54% to 100%, as seen in Table 1. Regarding scenario
C (paired samples t-test), following the intervention, students were more likely to consider independent versus
paired samples but were less likely to consider whether a
test involved identifying differences or a relationship. The
results in Table 2 for Spring 2012 suggest that extended
use of the decision tree over the statistics unit did help students make the right considerations when choosing statistical tests. In Spring 2012, the majority of students
considered each factor. In most cases, the percentage of students correctly considering each factor in Spring 2012 are
nearly as high, or higher than, the percentages from Fall
2011 after the decision tree was introduced.
Table 3 summarizes the results of the quiz where students were asked to recreate the decision tree by memory a
week following the in-class exercise. It includes the percentage of students that correctly identified each of the four
statistical tests as well as the percentage of students that
considered each of the traits in the decision tree. In both
semesters, almost all students included the number of variables and the scale of measurement in their decision trees. In
Fall 2011, only half of the students included the concepts of
independent versus paired samples and whether a test was
for a difference or relationship. In Spring 2012, the students
had a much better recollection of the decision tree, with
TABLE 2
Factors Considered by Scenario and Statistical Test
One sample (A)
Number of
variables (%)
Considered correctly
(%)
Scale of measurement
(%)
Considered correctly
(%)
Difference–
comovement (%)
Considered correctly
(%)
Independent–paired
(%)
Considered correctly
(%)
Sample size
Independent samples (B)
Paired samples (C)
Chi square (D)
Fall
2011
before
Fall
2011
after
Spring
2012
Fall
2011
before
Fall
2011
after
Spring
2012
Fall
2011
before
Fall
2011
after
Spring
2012
Fall
2011
before
Fall
2011
after
Spring
2012
83
100
96
91
100
96
95
97
100
91
100
95
71
82
90
91
100
95
82
94
100
91
100
95
57
86
71
83
95
96
91
83
92
64
94
86
57
73
69
71
68
83
91
83
86
64
72
32
6
5
0
49
82
60
50
22
80
82
50
77
94
95
100
37
73
50
50
22
80
73
33
69
6
27
4
26
64
49
18
83
84
5
19
2
94
73
96
17
59
42
0
61
55
95
81
98
35
22
84
35
22
84
22
36
84
22
36
84
DEVELOPING THOUGHT PROCESS FOR CHOOSING STATISTICAL METHODS
393
TABLE 3
Student Recall of Decision Trees
Statistical test included
Downloaded by [Universitas Maritim Raja Ali Haji] at 20:48 11 January 2016
One sample
Independent samples
Paired samples
Chi square
Fall 2011 percent correct
Spring 2012 percent correct
Factors considered
Fall 2011
Spring 2012
72
67
51
65
89
76
73
83
Number of variables
Scale of measurement
Independent–paired
Difference–relationship
93
89
54
53
99
100
84
89
large majorities of students correctly mapping each of the
four tests in the decision tree, and large majorities including
all four traits in the decision tree.
For additional insight into students’ thought processes,
we also observed students’ conversations while they completed the exercise. We found that students often did not
pause to reflect on the purpose or intent of the research
question first, and sometimes never at all. Instead, many
students used some form of process of elimination to come
to a conclusion. Sometimes students eliminated a test
because it had already been used to answer a previous exercise, surmising that it would not be repeated. Other times a
process of elimination was used more appropriately. Students started with any test they could remember and tried to
find reasons to eliminate it. Even more substantially, some
students used a process of elimination by considering a factor present in the decision tree and then eliminated tests on
the basis of that single factor. For example, if a student
identified that a problem concerned a ratio variable, the chisquare test of independence was eliminated. These
approaches used at least some of the knowledge organization in the decision tree. There were times that the decision
tree caused confusion, at least initially. In some cases students asked the right set of questions, but were not able to
answer them correctly, and so were unable to reach the correct conclusion. Finally, we noticed that students rarely followed the order of the questions as they are presented in the
decision tree. The most striking thing we learned from the
student conversation was that the use of language was a
substantial hurdle. We were able to identify several consistent sources of confusion, which can grouped into three
broad categories: difficulties defining a variable, confusion
about the use of the word independent, and the colloquial
use of the word relationship.
A fundamental source of confusion arose from identifying what should be considered a variable, especially when a
research question involved comparing a measurement
across two or more groups. For example, when students
compared hours worked between students who were
employed versus not employed, there were two ways to
define the variables of interest. The first was a nominal–
ratio pair, employment status and hours worked; the second
was a ratio–ratio pair, hours worked for employed students
and hours worked for not employed students. Both pairings
are equivalent, but each construction is convenient in
different circumstances. When the research question
involves two independent samples, as in the previous example, the nominal–ratio pairing aligns with how SPSS (ver.
20) treats the data in its spreadsheet columns. On the other
hand, the ratio–ratio pairing is a convenient mental model
when using statistical notation, such as when constructing
the null hypothesis, H0 : m1 ¡ m2 . The ratio–ratio mental
model is also convenient for determining whether it is appropriate to use an independent or a paired samples test. When
analyzing paired samples, the ratio–ratio pairing is arguably
the most convenient model because both SPSS and Excel
(Microsoft, Seattle, WA) treat the data in this way in its
spreadsheet columns.
A second source of confusion concerned the use of the
term independence. By the end of the semester, students
were exposed to at least three different uses of the word.
Early in the semester when we discussed causal research,
we introduced the idea of independent versus dependent
variables and we returned to this idea later when discussing
correlation and regression. In this case, the word independent was used to distinguish explanatory variables from
outcome variables. The idea of independence arose again
when discussing the chi-square test of independence for
two categorical variables. In this case, the word described
the relationship between two variables, and the test itself
was used to determine the presence of dependence. Finally,
the term was used to describe whether one should choose
an independent-samples t-test or paired samples t-test. The
types of research questions behind these tests do not
involve ideas of comovement between the variables, and
the statistical tests do not determine a presence of dependence like the chi-square test of independence. To the novice, this use appeared to be completely different than the
previous uses of the word. Too often, students got distracted
by the word independent in independent samples t-test and
confused this idea with the other concepts in the course that
used the same word.
Students also struggled with the colloquial use of the
term relationship. In our classes, we described the Pearson
and Spearman correlation coefficients as measuring a relationship between two variables. We also used the term relationship to describe the purpose of the chi-square test of
independence, for example, when asking if there is a relationship between employment status and class rank (both
categorical variables). The term relationship also appeared
394
J. MURRAY AND E. KNOWLES
colloquially, yet appropriately, for situations in which a test
of differences is appropriate. For example, we might have
asked if there is a relationship between gender and the number of cigarettes smoked. Many students hung on to the
word relationship and used it as a basis for deciding that a
Pearson correlation test was appropriate.
Downloaded by [Universitas Maritim Raja Ali Haji] at 20:48 11 January 2016
CONCLUSION
Our lesson made students’ learning and thinking processes
visible, which revealed some unexpected sources of confusion among students. We discovered that students had problems with how to define variables, and that the interaction
between colloquial and statistical language created confusion for many students.
Faculty need to acknowledge to students the challenges
that statistical language presents. Explicit examples can illustrate the meanings of words such as relationship and independent. The challenge is to move students beyond a narrow
use of words and to consider broader meaning. We have
increased our classroom emphasis on articulating and restating what meaning we want to ascertain from a statistical test.
Another insight that we gained from this lesson study
was the impact of the teaching strategy we employed in
Spring 2012: developing the decision tree throughout the
unit on statistics. Though we failed to find evidence that
this led to an immediate overall improvement in student
performance on the in-class exercise, we found students
had a strong retention of the mental mapping. With continued practice this is likely to lead to an improvement in the
long-run ability to apply statistical methods. Students will
also likely have an enhanced ability to learn and retain new
statistical methods that they may come across in the future.
The knowledge organization gives them an ability to connect new ideas to an existing body of knowledge.
APPENDIX A: In-Class Exercises
A researcher is interested in exploring the relationship
between student unemployment and effort put forth toward
academics. She administers a survey to full-time University
of Wisconsin–La Crosse (UW-L) undergraduate students
which includes the following questions:
1. What is your employment status? [Full time, Part time, Not
employed]
2. On average, how many hours do you work per week?
(Numeric / open ended)
3. On average, how many hours to you study per week?
(Numeric / open ended)
4. What is your class standing? [Freshman, Sophomore,
Junior, Senior]
For each of the following scenarios, answer the following questions:
1. What statistical method / test would you use to answer this
question?
2. Explain your reasoning for the previous answer. What
characteristics of this research question and methodology
make the test you chose appropriate?
Exercise A
Suppose the national average for the number of hours fulltime college students work is 12 hours per week. The
researcher is interested in determining if UW-L students
work on average more hours than the national average?
Exercise B
The researcher is interested in determining whether there is
a difference in the average number of hours students study
per week between those who are employed (either full-time
or part-time) and those who are not employed.
Exercise C
REFERENCES
Ambrose, S. A., Bridges, M. W., DiPietro, M., Lovett, M. C., & Norman,
M. K. (2010). How learning works: Seven research-based principles for
smart teaching. San Francisco, CA: Jossey-Bass.
Ausubel, D. P. (1978). In defense of advance organizers: A reply to the
critics. Review of Educational Research, 48, 251–257.
Cerbin, B. (2011). Lesson study: Using classroom inquiry to improve
teaching and learning in higher education. Sterling, VA: Stylus.
Cerbin, W., & Kopp, B. (2006). Lesson study as a model for building pedagogical knowledge and improving teaching. International Journal of
Teaching and Learning in Higher Education, 18, 250–257.
Du Prel, J., Rohrig, B., Hommel, G., & Blettner, M. (2010). Choosing statistical tests: Part 12 of a series on evaluation of scientific publications.
Deutsches Aerzteblatt International, 107, 343–348.
Garfield, J., & Chance, B. (2000). Assessment in statistics education:
Issues and challenges. Mathematical Thinking and Learning, 2, 99–125.
Mvududu, N. (2005). Constructivism in the statistics classroom: From theory to practice. Teaching Statistics, 27, 49–54.
The researcher is interested in determining whether on
average students spend more hours studying than the number of hours students spend working.
Exercise D
The researcher is interested in determining whether there is a
relationship between class standing and employment status.
APPENDIX B: Classroom Observation Guide
Observe the group discussion about the statistical test
which is appropriate to answer the question. Record each
observation with a number to indicate the order that the students consider each element. They may circle around to an
element more than once, record this as it happens (i.e. any
DEVELOPING THOUGHT PROCESS FOR CHOOSING STATISTICAL METHODS
element may have more than one number beside it). If the
conclusion about the element is incorrect, record an X by
the number.
Exercise (circle): A B C D
Observation number (circle): 1 2 3 4
Downloaded by [Universitas Maritim Raja Ali Haji] at 20:48 11 January 2016
Element
Discuss the number of variables considered.
Discuss the scale of measurement.
Discuss whether variables are independent or not.
Reflect on the purpose or intent of statistical test
(determining the difference or relationship).
Other
Consideration
395
Other questions to consider:
1. Did the students take into account any irrelevant
considerations?
2. Did the students reach the correct conclusion without well
articulated reasons?
3. Did students reach the incorrect conclusion, yet used
mostly correct and well articulated reasons?
4. How many group members were actively engaged in the
conversation? (Include those who are actively listening to
understand the concepts, but not those just trying to write
the correct answer).
ISSN: 0883-2323 (Print) 1940-3356 (Online) Journal homepage: http://www.tandfonline.com/loi/vjeb20
Developing Students’ Thought Processes for
Choosing Appropriate Statistical Methods
James Murray & Elizabeth Knowles
To cite this article: James Murray & Elizabeth Knowles (2014) Developing Students’ Thought
Processes for Choosing Appropriate Statistical Methods, Journal of Education for Business,
89:8, 389-395, DOI: 10.1080/08832323.2014.919896
To link to this article: http://dx.doi.org/10.1080/08832323.2014.919896
Published online: 04 Nov 2014.
Submit your article to this journal
Article views: 69
View related articles
View Crossmark data
Full Terms & Conditions of access and use can be found at
http://www.tandfonline.com/action/journalInformation?journalCode=vjeb20
Download by: [Universitas Maritim Raja Ali Haji]
Date: 11 January 2016, At: 20:48
JOURNAL OF EDUCATION FOR BUSINESS, 89: 389–395, 2014
Copyright Ó Taylor & Francis Group, LLC
ISSN: 0883-2323 print / 1940-3356 online
DOI: 10.1080/08832323.2014.919896
Developing Students’ Thought Processes
for Choosing Appropriate Statistical Methods
James Murray and Elizabeth Knowles
Downloaded by [Universitas Maritim Raja Ali Haji] at 20:48 11 January 2016
University of Wisconsin-La Crosse, La Crosse, Wisconsin, USA
Students often struggle to select appropriate statistical tests when investigating research
questions. The authors present a lesson study designed to make students’ thought processes
visible while considering this choice. The authors taught their students a way to organize
knowledge about statistical tests and observed its impact in the classroom and in students’
written work. The results from this intervention were mixed, but the authors discuss where
they found evidence for improvement in students’ performance and thought processes. The
classroom observations revealed that students have difficulty identifying variables and
understanding the precise use of statistical language.
Keywords: introductory statistics, lesson study, research methods
While students are trained to implement a variety of statistical tests in introductory statistics, it is still common for
them to struggle in subsequent classes when they must
apply this body of knowledge to select appropriate statistical tests to investigate a research question. This application
can challenge students because it requires an advanced
organization of knowledge that considers not only the purpose of the test, but also the number of variables, scale of
measurement, and characteristics of the samples. For the
expert, this complicated cognitive process has been developed and practiced over years. A student who is a novice in
statistics needs to be carefully guided through the decisionmaking process.
The purpose of this article is to share a teaching
approach we developed to help students organize their
knowledge of statistical techniques in a way that is conducive to choosing appropriate statistical tests. We present a
decision tree that maps 11 parametric and nonparametric
tests with four reflection questions that a student could ask
himself or herself concerning the data and nature of the
research question. We conducted a classroom investigation
in Fall 2011 and Spring 2012 that revealed students’
thought processes for selecting four of these statistical tests,
before and after presenting the decision tree. The areas of
confusion that we identify inform instructors of the pitfalls
Correspondence should be addressed to James Murray, University of
Wisconsin-La Crosse, Department of Economics, 1725 State Street, La
Crosse, WI 54601, USA. E-mail: jmurray@uwlax.edu
to address when presenting this content. We summarize our
students’ performance in selecting appropriate statistical
tests both before and after our treatment, and suggest teaching improvement strategies.
REVIEW OF LITERATURE
Knowledge organization is broadly recognized as an important consideration to improve student learning (Ausubel,
1978). Ambrose, Bridges, DiPietro, Lovett, and Norman
(2010) suggested that when students begin developing
knowledge organizations without guidance, the connections
are often weak, superficial, and few in number, and these
superficial connections can impede student learning. Statistics instructors can be more effective when they help students see various dimensions of and connections between
statistical tests in a research settting.
There is relatively little literature on organizing knowledge and training students to choose statistical tests. Du
Prel, Rohrig, Hommel, and Blettner (2010) examined some
common statistical techniques in medical studies and present their own decision trees that are more limited than what
is presented here. Their knowledge organization is confined
to two criteria: scale of measurement and whether the
research design is paired or not paired.
While little else in the literature addresses the specific
idea of organizing knowledge about statistical tests, many
papers speak more generally to the importance of developing
Downloaded by [Universitas Maritim Raja Ali Haji] at 20:48 11 January 2016
390
J. MURRAY AND E. KNOWLES
and investigating the critical thinking process involved in
the selection of appropriate methods. Garfield and Chance
(2000) noted that there is a growing movement in statistics
courses’ learning outcomes to focus on critical thinking
skills. Mvududu (2005) made the case that instructors should
help students take what they already know and use it to create new knowledge. He further suggested that instructors
should study how students think about the material they
encounter and investigate students’ thought processes on a
deeper level than through everyday communication.
The goal of this study was to explore students’ thought
processes as they considered the choice of an appropriate
statistical test in a research methods class, and to examine
the impact of presenting students with a way to organize
their knowledge of statistical tests. We used a lesson study
approach, which is a teaching improvement activity in
which instructors jointly develop a lesson to make student
learning visible. With this approach, instructors look deeply
into the teaching and learning process: the goals of a teaching activity, the design of the classroom instruction, understanding the students’ reception and mental processing of
the class content, and making evidence-based improvements in teaching (Cerbin, 2011). While it is a standard
practice in Japanese elementary education, lesson study has
more recently been adapted to higher education in the
United States. Cerbin and Kopp (2006) had a useful discussion of its application in higher education.
METHOD
We recognized that when experts determine statistical
approaches to a problem, they use a mental mapping which
includes many dimensions, such as univariate versus multivariate techniques, parametric versus nonparametric tests,
scale of measurement, and whether the purpose of the
research is to investigate differences, comovement, or independence. We developed a decision tree that makes these
considerations and connections explicit and visually portrays the questions that students could ask as they attempt
to determine an appropriate statistical test. Our goal for this
decision tree was to provide a way for students to organize
their knowledge of all the statistical methods that they learn
in the class and that they are likely to encounter in their
undergraduate business curriculum. The decision tree was
organized around four considerations: (a) the number of
variables involved in the research question, (b) the scale of
measurement for these variables, (c) the intent of the
research question (i.e., whether the purpose is to test for differences or a comovement between two variables), and (d)
whether the samples are independent or paired. We tested
the impact of the knowledge organization on students ability to identify when to apply four of the most commonly
used statistical tests. The complete decision tree is given in
Figure 1 and is the key instructional intervention that is the
focus of this paper.
We taught separate sections of a business research methods course in Fall 2011 and Spring 2012. In Fall 2011, we
conducted the investigation in three sections of the course.
Elizabeth Knowles taught one of these sections and James
Murray taught two. In Spring 2012, we conducted the
investigation in four sections of the course. Elizabeth
Knowles taught one of these sections and James Murray
taught three. Based on our collaborative effort preparing
for the statistics unit of the course, we each discussed the
previous four considerations throughout the statistics portion of our courses. While we had regular discussions on
our teaching approaches and used some shared examples in
our lectures, we did not make an explicit attempt to make
the presentation of all the material identical. Therefore, the
FIGURE 1. Decision tree. While we presented this decision tree in class, we only used the statistical tests in bold in the in-class exercises.
Downloaded by [Universitas Maritim Raja Ali Haji] at 20:48 11 January 2016
DEVELOPING THOUGHT PROCESS FOR CHOOSING STATISTICAL METHODS
measures of our students’ performance that we report in the
next section may include instructor-specific effects.
In both instructors’ classes and in both semesters, we
jointly developed a lesson on how to organize statistical
knowledge. The lesson included a brief review of statistical tests and an in-class exercise designed to make
students’ thought processes visible. The exercise
included four scenarios corresponding to four common
elementary statistical tests: (A) the one-sample t-test for
a mean, (B) the independent samples t-test for a difference in means, (C) the paired samples t-test for a difference in means, and (D) the chi-square test of
independence. Each scenario presented a description of
a single research question and survey questions that
could be used to collect data to answer the research
question. The exercise prompted students to choose a
statistical test and describe what considerations led them
to this decision. In the Findings section, we refer to
each of the four scenarios in the exercise with these
labels (A–D). The exercise is given in Appendix A.
To make students’ thought processes visible, we put
them in groups of three or four to discuss the exercise aloud
while they completed the task. We sat down with students
to observe their discussions, and we took notes on their
thought processes. We noted whether students discussed
the four questions in the decision tree, whether they used
any irrelevant considerations to reach their conclusion, and
whether they were able to articulate their reasoning well.
We observed the discussions in six groups in Fall 2011 and
eight groups in Spring 2012. The classroom observation
guide that we used to facilitate this notetaking is provided
in Appendix B.
Subsequently, each student wrote and submitted individual answers for the in-class exercise, which may have varied
from their group members. We reviewed the responses and
recorded the following: whether the students chose the correct statistical test, whether the students’ rationale included
any of the four questions emphasized in the decision tree,
and whether the questions were correctly identified.
In the Fall 2011 lesson, the students completed two of
the four scenarios in their in-class exercise before we presented the decision tree. After presenting the decision tree,
students were asked to put away their notes and they completed the final two scenarios. The intention was to provide
students a mental organization for statistical procedures,
and compare the students’ performance and thought processes before and after this intervention. In each section, we
administered the four scenarios in the in-class exercise in a
different order so that every scenario appeared both before
the decision tree was presented and after. This allowed us to
compare the impact the decision tree had on students’ performance and thought processes for each of the scenarios.
While this timing for presenting the decision tree was
useful for the research design, in Spring 2012 we determined that it would be more pedagogically valuable to
391
present the decision tree at the beginning of the statistics
unit of the course and continue to refer to it throughout the
three week unit. This helped students organize new knowledge as they learned it, and it gave students practice using
the decision tree before conducting the classroom observation and in-class exercise. Again in Spring 2012 we randomized the order of the four scenarios in the in-class
exercise for each section of the course.
To measure students’ retention of the knowledge organization, we gave our students an unannounced quiz in the
week following our classroom observation. In the quiz, we
asked students to reconstruct the decision tree from memory. We evaluated our students’ work and noted whether it
included each of the four questions in the decision tree, and
whether or not students arrived at each of the four statistical
tests with any incorrect considerations.
FINDINGS
Table 1 shows the percentage of students that correctly
identified the appropriate statistical test in each of the four
scenarios of the in-class exercise. For Fall 2011, results are
reported for before and after the decision tree was introduced. In Spring 2012, because the decision tree was developed before the in-class exercise was administered, the
percentage correct reflects all attempts. The results from
Fall 2011 indicate some general improvement after the
introduction of the decision tree. We expected that performance would be better in Spring 2012, but the results were
mixed, depending on the statistical test.
The data suggests that the paired-samples t-test was the
most difficult for students. Interestingly, none of the 22 students who were given this question before the decision tree
intervention in the Fall 2011 semester answered this
question correctly. The percentage correct for the chisquare test of independence actually decreased after the
decision tree intervention. In Fall 2011, 86% of students
answered this question correctly before the intervention,
and only 58% afterward. In Spring 2012, only 39% of students correctly identified the chi-square test. The most common incorrect answer for this question was a Pearson
correlation coefficient, another test that examines a relationship between two variables, but used for interval/ratio
data rather than categorical data.
Table 2 identifies the overall percentage of students that
considered, either correctly or incorrectly, each of the following factors in their written responses: (a) number of variables, (b) scale of measurement, (c) the intent of the
statistical test (i.e., whether the method tests for differences
or a comovement between two variables), and (d) whether
samples are independent or paired. The table also reports
the percentage of students who considered each factor correctly. For the most part, students considered the number of
variables and the scale of measurement of the variables.
392
J. MURRAY AND E. KNOWLES
TABLE 1
Correct Answers by Statistical Test
One sample (A)
Semester
Independent samples (B)
Paired samples (C)
Chi square (D)
Percent correct Sample size Percent correct Sample size Percent correct Sample size Percent correct Sample size
Fall 2011: Prior to decision tree
Fall 2011: Following decision tree
Spring 2012
77%
82%
92%
35
22
84
54%
100%
56%
35
22
84
0%
67%
47%
22
36
84
86%
58%
39%
22
36
84
Downloaded by [Universitas Maritim Raja Ali Haji] at 20:48 11 January 2016
Note: Sample size is number of students who completed the exercise. As students discussed the problems in groups before submitting their individual
responses, this does not represent a number of independent observations.
Considering whether or not samples were independent or
paired and whether the scenario suggested examining differences or a relationship occurred less frequently. In scenario B (independent samples t-test), in Fall 2011 before
introducing the decision tree, less than half of the students
(49%) considered whether a test for a difference or a relationship was appropriate, and about one fourth (26%) considered whether the samples were independent or paired.
Following the decision tree, the percentage of students considering difference versus relationship increased to 82%
and the percentage of students considering independent versus paired samples increased to 64%. This is associated
with an increase in overall performance on the question
from 54% to 100%, as seen in Table 1. Regarding scenario
C (paired samples t-test), following the intervention, students were more likely to consider independent versus
paired samples but were less likely to consider whether a
test involved identifying differences or a relationship. The
results in Table 2 for Spring 2012 suggest that extended
use of the decision tree over the statistics unit did help students make the right considerations when choosing statistical tests. In Spring 2012, the majority of students
considered each factor. In most cases, the percentage of students correctly considering each factor in Spring 2012 are
nearly as high, or higher than, the percentages from Fall
2011 after the decision tree was introduced.
Table 3 summarizes the results of the quiz where students were asked to recreate the decision tree by memory a
week following the in-class exercise. It includes the percentage of students that correctly identified each of the four
statistical tests as well as the percentage of students that
considered each of the traits in the decision tree. In both
semesters, almost all students included the number of variables and the scale of measurement in their decision trees. In
Fall 2011, only half of the students included the concepts of
independent versus paired samples and whether a test was
for a difference or relationship. In Spring 2012, the students
had a much better recollection of the decision tree, with
TABLE 2
Factors Considered by Scenario and Statistical Test
One sample (A)
Number of
variables (%)
Considered correctly
(%)
Scale of measurement
(%)
Considered correctly
(%)
Difference–
comovement (%)
Considered correctly
(%)
Independent–paired
(%)
Considered correctly
(%)
Sample size
Independent samples (B)
Paired samples (C)
Chi square (D)
Fall
2011
before
Fall
2011
after
Spring
2012
Fall
2011
before
Fall
2011
after
Spring
2012
Fall
2011
before
Fall
2011
after
Spring
2012
Fall
2011
before
Fall
2011
after
Spring
2012
83
100
96
91
100
96
95
97
100
91
100
95
71
82
90
91
100
95
82
94
100
91
100
95
57
86
71
83
95
96
91
83
92
64
94
86
57
73
69
71
68
83
91
83
86
64
72
32
6
5
0
49
82
60
50
22
80
82
50
77
94
95
100
37
73
50
50
22
80
73
33
69
6
27
4
26
64
49
18
83
84
5
19
2
94
73
96
17
59
42
0
61
55
95
81
98
35
22
84
35
22
84
22
36
84
22
36
84
DEVELOPING THOUGHT PROCESS FOR CHOOSING STATISTICAL METHODS
393
TABLE 3
Student Recall of Decision Trees
Statistical test included
Downloaded by [Universitas Maritim Raja Ali Haji] at 20:48 11 January 2016
One sample
Independent samples
Paired samples
Chi square
Fall 2011 percent correct
Spring 2012 percent correct
Factors considered
Fall 2011
Spring 2012
72
67
51
65
89
76
73
83
Number of variables
Scale of measurement
Independent–paired
Difference–relationship
93
89
54
53
99
100
84
89
large majorities of students correctly mapping each of the
four tests in the decision tree, and large majorities including
all four traits in the decision tree.
For additional insight into students’ thought processes,
we also observed students’ conversations while they completed the exercise. We found that students often did not
pause to reflect on the purpose or intent of the research
question first, and sometimes never at all. Instead, many
students used some form of process of elimination to come
to a conclusion. Sometimes students eliminated a test
because it had already been used to answer a previous exercise, surmising that it would not be repeated. Other times a
process of elimination was used more appropriately. Students started with any test they could remember and tried to
find reasons to eliminate it. Even more substantially, some
students used a process of elimination by considering a factor present in the decision tree and then eliminated tests on
the basis of that single factor. For example, if a student
identified that a problem concerned a ratio variable, the chisquare test of independence was eliminated. These
approaches used at least some of the knowledge organization in the decision tree. There were times that the decision
tree caused confusion, at least initially. In some cases students asked the right set of questions, but were not able to
answer them correctly, and so were unable to reach the correct conclusion. Finally, we noticed that students rarely followed the order of the questions as they are presented in the
decision tree. The most striking thing we learned from the
student conversation was that the use of language was a
substantial hurdle. We were able to identify several consistent sources of confusion, which can grouped into three
broad categories: difficulties defining a variable, confusion
about the use of the word independent, and the colloquial
use of the word relationship.
A fundamental source of confusion arose from identifying what should be considered a variable, especially when a
research question involved comparing a measurement
across two or more groups. For example, when students
compared hours worked between students who were
employed versus not employed, there were two ways to
define the variables of interest. The first was a nominal–
ratio pair, employment status and hours worked; the second
was a ratio–ratio pair, hours worked for employed students
and hours worked for not employed students. Both pairings
are equivalent, but each construction is convenient in
different circumstances. When the research question
involves two independent samples, as in the previous example, the nominal–ratio pairing aligns with how SPSS (ver.
20) treats the data in its spreadsheet columns. On the other
hand, the ratio–ratio pairing is a convenient mental model
when using statistical notation, such as when constructing
the null hypothesis, H0 : m1 ¡ m2 . The ratio–ratio mental
model is also convenient for determining whether it is appropriate to use an independent or a paired samples test. When
analyzing paired samples, the ratio–ratio pairing is arguably
the most convenient model because both SPSS and Excel
(Microsoft, Seattle, WA) treat the data in this way in its
spreadsheet columns.
A second source of confusion concerned the use of the
term independence. By the end of the semester, students
were exposed to at least three different uses of the word.
Early in the semester when we discussed causal research,
we introduced the idea of independent versus dependent
variables and we returned to this idea later when discussing
correlation and regression. In this case, the word independent was used to distinguish explanatory variables from
outcome variables. The idea of independence arose again
when discussing the chi-square test of independence for
two categorical variables. In this case, the word described
the relationship between two variables, and the test itself
was used to determine the presence of dependence. Finally,
the term was used to describe whether one should choose
an independent-samples t-test or paired samples t-test. The
types of research questions behind these tests do not
involve ideas of comovement between the variables, and
the statistical tests do not determine a presence of dependence like the chi-square test of independence. To the novice, this use appeared to be completely different than the
previous uses of the word. Too often, students got distracted
by the word independent in independent samples t-test and
confused this idea with the other concepts in the course that
used the same word.
Students also struggled with the colloquial use of the
term relationship. In our classes, we described the Pearson
and Spearman correlation coefficients as measuring a relationship between two variables. We also used the term relationship to describe the purpose of the chi-square test of
independence, for example, when asking if there is a relationship between employment status and class rank (both
categorical variables). The term relationship also appeared
394
J. MURRAY AND E. KNOWLES
colloquially, yet appropriately, for situations in which a test
of differences is appropriate. For example, we might have
asked if there is a relationship between gender and the number of cigarettes smoked. Many students hung on to the
word relationship and used it as a basis for deciding that a
Pearson correlation test was appropriate.
Downloaded by [Universitas Maritim Raja Ali Haji] at 20:48 11 January 2016
CONCLUSION
Our lesson made students’ learning and thinking processes
visible, which revealed some unexpected sources of confusion among students. We discovered that students had problems with how to define variables, and that the interaction
between colloquial and statistical language created confusion for many students.
Faculty need to acknowledge to students the challenges
that statistical language presents. Explicit examples can illustrate the meanings of words such as relationship and independent. The challenge is to move students beyond a narrow
use of words and to consider broader meaning. We have
increased our classroom emphasis on articulating and restating what meaning we want to ascertain from a statistical test.
Another insight that we gained from this lesson study
was the impact of the teaching strategy we employed in
Spring 2012: developing the decision tree throughout the
unit on statistics. Though we failed to find evidence that
this led to an immediate overall improvement in student
performance on the in-class exercise, we found students
had a strong retention of the mental mapping. With continued practice this is likely to lead to an improvement in the
long-run ability to apply statistical methods. Students will
also likely have an enhanced ability to learn and retain new
statistical methods that they may come across in the future.
The knowledge organization gives them an ability to connect new ideas to an existing body of knowledge.
APPENDIX A: In-Class Exercises
A researcher is interested in exploring the relationship
between student unemployment and effort put forth toward
academics. She administers a survey to full-time University
of Wisconsin–La Crosse (UW-L) undergraduate students
which includes the following questions:
1. What is your employment status? [Full time, Part time, Not
employed]
2. On average, how many hours do you work per week?
(Numeric / open ended)
3. On average, how many hours to you study per week?
(Numeric / open ended)
4. What is your class standing? [Freshman, Sophomore,
Junior, Senior]
For each of the following scenarios, answer the following questions:
1. What statistical method / test would you use to answer this
question?
2. Explain your reasoning for the previous answer. What
characteristics of this research question and methodology
make the test you chose appropriate?
Exercise A
Suppose the national average for the number of hours fulltime college students work is 12 hours per week. The
researcher is interested in determining if UW-L students
work on average more hours than the national average?
Exercise B
The researcher is interested in determining whether there is
a difference in the average number of hours students study
per week between those who are employed (either full-time
or part-time) and those who are not employed.
Exercise C
REFERENCES
Ambrose, S. A., Bridges, M. W., DiPietro, M., Lovett, M. C., & Norman,
M. K. (2010). How learning works: Seven research-based principles for
smart teaching. San Francisco, CA: Jossey-Bass.
Ausubel, D. P. (1978). In defense of advance organizers: A reply to the
critics. Review of Educational Research, 48, 251–257.
Cerbin, B. (2011). Lesson study: Using classroom inquiry to improve
teaching and learning in higher education. Sterling, VA: Stylus.
Cerbin, W., & Kopp, B. (2006). Lesson study as a model for building pedagogical knowledge and improving teaching. International Journal of
Teaching and Learning in Higher Education, 18, 250–257.
Du Prel, J., Rohrig, B., Hommel, G., & Blettner, M. (2010). Choosing statistical tests: Part 12 of a series on evaluation of scientific publications.
Deutsches Aerzteblatt International, 107, 343–348.
Garfield, J., & Chance, B. (2000). Assessment in statistics education:
Issues and challenges. Mathematical Thinking and Learning, 2, 99–125.
Mvududu, N. (2005). Constructivism in the statistics classroom: From theory to practice. Teaching Statistics, 27, 49–54.
The researcher is interested in determining whether on
average students spend more hours studying than the number of hours students spend working.
Exercise D
The researcher is interested in determining whether there is a
relationship between class standing and employment status.
APPENDIX B: Classroom Observation Guide
Observe the group discussion about the statistical test
which is appropriate to answer the question. Record each
observation with a number to indicate the order that the students consider each element. They may circle around to an
element more than once, record this as it happens (i.e. any
DEVELOPING THOUGHT PROCESS FOR CHOOSING STATISTICAL METHODS
element may have more than one number beside it). If the
conclusion about the element is incorrect, record an X by
the number.
Exercise (circle): A B C D
Observation number (circle): 1 2 3 4
Downloaded by [Universitas Maritim Raja Ali Haji] at 20:48 11 January 2016
Element
Discuss the number of variables considered.
Discuss the scale of measurement.
Discuss whether variables are independent or not.
Reflect on the purpose or intent of statistical test
(determining the difference or relationship).
Other
Consideration
395
Other questions to consider:
1. Did the students take into account any irrelevant
considerations?
2. Did the students reach the correct conclusion without well
articulated reasons?
3. Did students reach the incorrect conclusion, yet used
mostly correct and well articulated reasons?
4. How many group members were actively engaged in the
conversation? (Include those who are actively listening to
understand the concepts, but not those just trying to write
the correct answer).