08832323.2013.856281
Journal of Education for Business
ISSN: 0883-2323 (Print) 1940-3356 (Online) Journal homepage: http://www.tandfonline.com/loi/vjeb20
Developing and Assessing E-Learning Techniques
for Teaching Forecasting
Yulia R. Gel, R. Jeanette O’Hara Hines, He Chen, Kimihiro Noguchi & Vivian
Schoner
To cite this article: Yulia R. Gel, R. Jeanette O’Hara Hines, He Chen, Kimihiro Noguchi & Vivian
Schoner (2014) Developing and Assessing E-Learning Techniques for Teaching Forecasting,
Journal of Education for Business, 89:5, 215-221, DOI: 10.1080/08832323.2013.856281
To link to this article: http://dx.doi.org/10.1080/08832323.2013.856281
Published online: 03 Jul 2014.
Submit your article to this journal
Article views: 98
View related articles
View Crossmark data
Full Terms & Conditions of access and use can be found at
http://www.tandfonline.com/action/journalInformation?journalCode=vjeb20
Download by: [Universitas Maritim Raja Ali Haji]
Date: 11 January 2016, At: 20:41
JOURNAL OF EDUCATION FOR BUSINESS, 89: 215–221, 2014
Copyright Ó Taylor & Francis Group, LLC
ISSN: 0883-2323 print / 1940-3356 online
DOI: 10.1080/08832323.2013.856281
Developing and Assessing E-Learning Techniques
for Teaching Forecasting
Yulia R. Gel
Downloaded by [Universitas Maritim Raja Ali Haji] at 20:41 11 January 2016
University of Waterloo, Waterloo, Ontario, Canada; and Saint Petersburg State University,
St. Petersburg, Russia
R. Jeanette O’Hara Hines
University of Waterloo, Waterloo, Ontario, Canada
He Chen
Spartan Fund Management Inc., Toronto, Ontario, Canada
Kimihiro Noguchi
Colorado State University, Fort Collins, Colorado, USA
Vivian Schoner
University of Waterloo, Waterloo, Ontario, Canada
In the modern business environment, managers are increasingly required to perform decision
making and evaluate related risks based on quantitative information in the face of
uncertainty, which in turn increases demand for business professionals with sound skills and
hands-on experience with statistical data analysis. Computer-based training technologies
allow the new cadre of business professionals to obtain such a hands-on experience in an
environment where mistakes can be made and outcomes can be measured. The authors
discuss their experiences in developing a new e-learning tool designed to apply
methodological forecasting concepts to real-life business and finance problems through an
interactive self-learning and self-assessing module of online case studies.
Keywords: business statistics, decision making and analysis, e-learning, forecasting,
interactive learning, technology in education
Recent developments in technology have advanced to
the point where nearly every quantitative area in business
relies on statistical analysis. As a result, there is an increasing student interest in statistical courses and a marketdriven demand for business professionals with solid skills
in applied data analysis who can perform reliable decision
making in the face of uncertainty. Hence, statistics, being at
the intersection of multiple disciplines, goes far beyond
Correspondence should be addressed to Yulia R. Gel, University of
Waterloo, Department of Statistics and Actuarial Science, 200 University
Avenue West, Waterloo, Ontario N2L 3G1, Canada. E-mail: ygl@math.
uwaterloo.ca
simply deriving mathematical formulas in the classroom.
Instead, statistics now requires well-developed practical
skills in data analysis that are best acquired through intensive work with real-world case studies, systems, and software. The challenge is to provide this training in an
environment where mistakes can be made and outcomes
can be measured (Arsham, 2013).
Nowadays, computer-assisted learning and various
related e-learning techniques are widely recognized as flexible, fast, and efficient ways to deliver new knowledge that
helps bring just-in-time critical professional improvements
and training while minimizing time and costs involved
(Burrill, 2009; Ghaoui & Janvier, 2004). Here, by e-learning
Downloaded by [Universitas Maritim Raja Ali Haji] at 20:41 11 January 2016
216
Y. R. GEL ET AL.
we mean all forms of computer technology assisted teaching
and learning, including both in- and out-of-class experience,
with a particular focus on web-based interactive applications
(Tavangarian, Leypold, N€
olting, & R€
oser, 2004). Nevertheless, although interactive e-learning modules are already utilized for developing competence in using introductory
statistics for business and finance applications, there still
exists a lack of interactive tools which offer a balanced
combination of rigorous modern statistical methodology, indepth practical studies, and self-paced educational options
for an academic environment (Nolan & Temple Lang, 2007;
2010; Pfeil, 2006; Pratt, Davies & Connor, 2011). In particular, while there have been a variety of different e-learningin-statistics initiatives worldwide (Alldredge & Som, 2002;
Davies & Barnett, 2005; Fields & Collins, 2005; Gonzalez
& Munoz, 2006; H€ardle, Klinke, & Ziegenhagen, 2006,
2007; Hilton & Christensen, 2002), the focus of such projects is mainly on introductory statistical courses rather than
on more advanced time series and forecasting methodology.
Moreover, while many textbooks either offer a complimentary software packages (see Bowerman, O’Connell, &
Koehler, 2004; Brockwell & Davis, 2002; Newton, 1988) or
provide examples in one of the statistical programming languages (see Cowpertwait, 2009; Cryer, 2008; Shumway &
Stoffer, 2006; Yaffe, 2000), these options are typically static
and do not allow for an interactive learning environment. A
few recent initiatives in this direction include the projects
by Madsen, Albeanu, Burtschy, and Popentiu-Vladicescu
(2006) and Albeanu, Madsen, Popentiu-Vladicescu, and
Dumitru (2010) who proposed the e-learning system eTJMES and JAVA-module for exploratory time series analysis. Aydinli, H€ardle, and R€
onz (2003) discussed integration
of the statistical software R and Microsoft Excel for teaching time series, and Georgios, Stoyanova, and Onkov
(2005) focused on e-learning of trend modeling in a web
environment. Recently, Shmueli (2012) emphasized the
importance of interactive e-learning techniques in providing
hands-on experience in forecasting and offered an interactive visualization package for exploring the nature of the
observed time series.
Even less is known about the effectiveness of e-learning
tools for enhancing educational outcomes of academic
courses in more advanced statistics. In this paper, our goal
is to bridge this gap between the modern e-learning techniques and academic environment of undergraduate courses
in statistics with applications in business, finance, and
insurance, and to assess the impact of interactive web-based
e-learning on teaching and learning progress. In particular,
our key innovation consists of developing, integration, and
evaluation of a new interactive e-learning concept for a
higher level undergraduate training in forecasting. The
main idea of the proposed e-learning approach is based on
a fusion of practical forecasting case studies illustrating
various statistical modeling procedures, statistical software
(in our case R) that is used for implementation of
computational methods, and interactive capabilities of the
Learning Management System (LMS; in our case ANGEL).
It is important to emphasize that our focus is not on developing a new software package for time series modeling and
prediction but rather on elaborating interactive e-learning
techniques for teaching forecasting. Note that the proposed
e-learning concept is not restricted to statistical software R
and can be easily transferred to any other computational
software (e.g., Matlab, SPSS, SAS) as well as it is not
restricted to the LMS ANGEL but is open to any other
LMS allowing for randomization of questions and answers
in online quizzes. Although in this paper we discuss our
experience on implementation and assessment of the proposed new web-based e-learning procedures for teaching a
higher level undergraduate course in forecasting at the University of Waterloo, Canada, based on our results, we
believe that similar interactive e-learning techniques might
be found useful for distance education courses and graduate
programs in computational finance, econometrics, statistics,
actuarial sciences and other quantitative disciplines, with a
strong focus on applied aspects of modeling and data
analysis.
The article is organized as follows. The next section
describes the developed e-learning tool. Then, we assess
effectiveness of the proposed e-learning techniques. Last,
we conclude with the discussion.
METHOD
Participants
In our project, we focus on the 400-level undergraduate
course Forecasting, offered by the Department of Statistics
and Actuarial Science at the University of Waterloo three
times per year (i.e., in each of the 12-week-long winter,
spring, and fall terms). The enrolled students are supposed
to have a background in applied linear regression. Most of
the enrolled students are either from actuarial science or
from the business administration and mathematics double
degree program. The Society of Actuaries (SOA) recognizes a final grade of at least 70 in the forecasting course as
qualifying a student for a SOA credit. Hence, the course is
getting increasingly popular over last few years. Indeed,
over the period of 2005–2011, the average number of registered students is about 135, while in 2012 the enrollment
surged to more than 200.
Instrument
Given the practically oriented majors of enrolled students,
the course is very application driven and computationally
intense, and focuses more on employment of statistical
methods to real-life situations, such as predicting sales,
stock returns, or supplies, rather than on derivation of
mathematical formulas and proofs. This, along with the
Downloaded by [Universitas Maritim Raja Ali Haji] at 20:41 11 January 2016
DEVELOPING AND ASSESSING E-LEARNING TECHNIQUES
increasing enrollment numbers, requires complementary
learning skills and revision of instructional approaches in
order to improve the fit of the lecture material into cuttingedge real life business applications. To enhance educational
outcomes, our idea was to incorporate into these two
courses elements of interactive learning, such as online
self-educational illustrative case studies with self-evaluation options, an online bank of test questions and a library
of real-world data sets. We then assessed the learning process in the new environment.
While the online repository of data sets serves a complementary role of an informational source for student projects,
homework, and exercises, the core element in the developed
module is a set of real-life case studies illustrating various
modeling and forecasting techniques in business, finance
and economics, which allow students to self-assess their
understanding of the methodological and computational
aspects using online interactive tests. In particular, we
developed three such case studies involving short-,
medium-, and long-term prediction of sales of Australian
fortified wine (data are from the repository of Hyndman [n.
d.]), the 25 years of household financial obligations and
daily stock returns of an electronics company in the United
States. Each case study is divided into a sequence of chapters such that the same data are analyzed and modeled using
different approaches showing a step-by-step application of
incrementally more sophisticated techniques and methods
that are discussed in class. Some examples of those techniques are exploratory diagnostic tests for time series data,
regression analysis with endo- and exogenous variables, the
Holt-Winters smoothing techniques, the Box-Jenkins methodology (Autoregressive Moving Average/Autoregressive
Integrated Moving Average [ARIMA]/Seasonal Autoregressive Integrated Moving Average), as well as stochastic volatility and conditional heteroscedastic models.
The goal is to teach students that there is no “perfect recipe” in business forecasting or modeling, and instead to
show that the same data sets can be approached from different angles, depending on the analyst’s task. Indeed, the
developed e-learning tool, including the interactive selfassessment options, is designed in such a way that the same
data set is consecutively step-by-step approached with different modeling and prediction methods. For example, students find that the Holt-Winters smoothing is preferred for
short-term forecasts of sales of Australian dessert wine but a
seasonal ARIMA models performs better for medium- and
long-term forecasting of the same product sales. All discussed modeling procedures are complemented by a
detailed description of their implementation in R. Each
chapter is then concluded with a set of online questions in a
multiple choice and multiple selection form, so students can
self-evaluate their understanding of the exhibited data patterns, the relevant statistical tools, and their software implementation. For instance, using exploratory analysis of
various time series plots of data sets similar to those
217
discussed in the case studies, students identify the main features of the data (e.g., stock trend channels, seasonality, outliers) and get interactive feedback. This part is especially
important because various exploratory analysis techniques
play a key role in quantitative finance and econometrics.
Some questions require students to replicate the presented
analysis on their own and carry out an extension of the presented study, for example, to assess prediction performance
of the same model with and without outliers or using various variance-stabilizing transformations.
It is important to emphasize that some parameter-based
questions with a more theoretical focus are designed in
such a way that the parameters are able to take on random
values within a predefined range. For instance, we can consider identification of a linear model order and parameters
by its theoretical autocorrelation and partial autocorrelation
functions (i.e., acf and pacf, respectively). While the true
underlying structure is a moving average model of order 1,
such as MA(1), the parameter of MA(1) is allowed to take
random values within the interval of identifiability (–1,1),
hence, producing myriads of MA(1) models with the
respective acf and pacf plots. The effect is that any such
question can generate a different scenario for each student,
and the probability that a student would have the same
question twice while practicing with the tool is relatively
low. The same parameter randomization approach is undertaken for other linear and nonlinear models that are discussed in the course. After completing the online tests,
students receive a grade and can review their tests along
with the correct answers.
Note that the key difference between our approach and
the old-fashioned approach of end-of-chapter homework
problems is that in our approach problem parameters are
randomized, allowing simulations of an unbounded number
of similar questions. Hence, in contrast to the old-fashioned
end-of-chapter approach, students can see the same type of
question not once but many times allowing repeated
answering until he/she is fully satisfied with the interactive
grade and the related understanding of the theory or methodology behind the posed question. Hence, our approach
gives a much richer variety of questions.
The bank of questions is implemented using an LMS
called ANGEL, while all the plots are prepared in the statistical programming language R. All case studies along with
the corresponding online self-evaluation tests are available
to the enrolled students via the LMS course website. The
chapters of the case studies appear online after the corresponding content is discussed in class, and students can
then work with the case studies an indefinite number of
times, accessing them from anywhere. Using the LMS, we
also keep track of who uses the tool, how many times a student accesses certain case studies, what online tests are performed, what grades are received, and the detailed statistics
on each question. Students were also encouraged to keep
track of their experience in informal logbooks that may be
218
Y. R. GEL ET AL.
employed to improve usability of a future version of the
e-learning tool.
It is important to emphasize that the developed interactive e-learning tool1 is not limited to any particular statistical software (e.g., R in our case) but rather should viewed
as a new e-learning methodological approach that can be
relatively easily adapted to other software packages and
LMS, as long as such LMS allow randomization of questions and answers. To our knowledge, most of modern LMS
(e.g., Blackboard, Desire2Learn) possess these technical
capabilities.
Downloaded by [Universitas Maritim Raja Ali Haji] at 20:41 11 January 2016
Data Collection and Design
Our goal is to investigate if adding work on case studies to
the curriculum of the Forecasting course improves the
students’ level of understanding of a variety of concepts,
with the level of understanding being measured by the final
exam grade. Our study is an observational one in which we
use the students in the one available class. The complete
case studies grade is based on answers to the online
questions.
Clearly, using a randomized design (i.e., randomly
assigning no case studies to half of the students, with
the other half having case studies) may appear as a better alternative to an observational approach. However,
such a randomization approach was viewed as unethical
and not allowed by the Office of Research Ethics. So
randomization is not feasible in our situation. In turn,
assigning case studies only to volunteers/interest groups
is also not applicable due to self-selection bias. Finally,
the limitations in funding duration and scheduling
issues, as well as the technical challenges due to the
change of learning management platforms in the University of Waterloo did not allow us to perform a sequential study when all enrolled students in one given year
were assigned case studies and students from the next
year were not. Hence, we proceed with an observational
design in the one available forecasting class.
In particular, in order to see if the inclusion of case studies benefited the students, we measure the impact of the
student’s (average) case studies grade2 on his or her performance in a final exam, given information on the overall
grade point average (GPA) over all previous terms, a major,
term of study, and gender. That is, we regress the students’
final exam grade on their case studies grade, including the
average GPA from previous terms major, term of study,
and gender as covariates in an attempt to control for the
incoming intelligence of the student.3
We found that only the overall GPA and case studies
grade are significant predictors, and hence all other factors
(i.e., major, term of study, and gender) are discarded from
the further study.
Note that while final examination grades are admittedly
imperfect, they are useful as measures for evaluation of the
effectiveness of a learning tool. In contrast to standardized
tests, they are not universally comparable among students
from different universities and colleges. However, they
remain a conventional quantitative measure of learning outcome assessment adopted internally for various disciplines
by numerous educational institutions worldwide (see discussion by Ewell, 2005; Johnson, Berg, & Heddens, 2005;
Ming, 2012; Sebastianelli & Tamimi, 2011, and references
therein) and they have been validated externally by the
Mathematical Association of America (see the guidelines
of the Subcommittee on Assessment of the Committee on
the Undergraduate Program in Mathematics of the Mathematical Association of America, 2005). Moreover, while in
our case we assess the overall student knowledge in the
course, reflected by the final course grade, as a weighted
combination of four take-home assignments involving data
analysis (15%), a take-home midterm (15%), group projects
(15%), and the in-class final exam (55%), the in-class final
examination grade represents the only available measure of
student individual work because the other three assessment
categories (i.e., take-home assignments, a take-home midterm, and group projects) may explicitly or implicitly
involve teamwork rather than individual activity. Finally,
the final examination represents a collection of randomized
computer-based multiple-choice and multiple-select questions covering the whole course curriculum and can be
viewed as a substantially extended version of the online
case study questions. However, in contrast to the interactive
online case studies, students are allowed to answer each
question just once. Hence, the employed final examination
can be considered to be a version of a test-retest assessment
(see Stayhorn, 2006, and references therein).
RESULTS
Table 1 presents the estimated coefficients along with the
respective standard errors, t statistics and p values from a
multiple regression of final exam grade on the case studies
grade and the GPA, while Figure 1 displays the scatter plot
of the final exam grade versus the case studies grade. There
were 155 students enrolled into the forecasting course of
which 128 attempted all three of the available case studies.
In addition, one student attempted only the first case study,
three did only the first two, and 23 students did not do any
of case studies.) Further examination of the case studies
TABLE 1
The Results From the Multiple Regression of a Final Exam Grade
on Case Studies Grade and Grade Point Average
Item
Case studies
Grade point average
Estimate
SE
t
p
.1359
.7218
.0159
.0792
2.296
9.116
.0234
ISSN: 0883-2323 (Print) 1940-3356 (Online) Journal homepage: http://www.tandfonline.com/loi/vjeb20
Developing and Assessing E-Learning Techniques
for Teaching Forecasting
Yulia R. Gel, R. Jeanette O’Hara Hines, He Chen, Kimihiro Noguchi & Vivian
Schoner
To cite this article: Yulia R. Gel, R. Jeanette O’Hara Hines, He Chen, Kimihiro Noguchi & Vivian
Schoner (2014) Developing and Assessing E-Learning Techniques for Teaching Forecasting,
Journal of Education for Business, 89:5, 215-221, DOI: 10.1080/08832323.2013.856281
To link to this article: http://dx.doi.org/10.1080/08832323.2013.856281
Published online: 03 Jul 2014.
Submit your article to this journal
Article views: 98
View related articles
View Crossmark data
Full Terms & Conditions of access and use can be found at
http://www.tandfonline.com/action/journalInformation?journalCode=vjeb20
Download by: [Universitas Maritim Raja Ali Haji]
Date: 11 January 2016, At: 20:41
JOURNAL OF EDUCATION FOR BUSINESS, 89: 215–221, 2014
Copyright Ó Taylor & Francis Group, LLC
ISSN: 0883-2323 print / 1940-3356 online
DOI: 10.1080/08832323.2013.856281
Developing and Assessing E-Learning Techniques
for Teaching Forecasting
Yulia R. Gel
Downloaded by [Universitas Maritim Raja Ali Haji] at 20:41 11 January 2016
University of Waterloo, Waterloo, Ontario, Canada; and Saint Petersburg State University,
St. Petersburg, Russia
R. Jeanette O’Hara Hines
University of Waterloo, Waterloo, Ontario, Canada
He Chen
Spartan Fund Management Inc., Toronto, Ontario, Canada
Kimihiro Noguchi
Colorado State University, Fort Collins, Colorado, USA
Vivian Schoner
University of Waterloo, Waterloo, Ontario, Canada
In the modern business environment, managers are increasingly required to perform decision
making and evaluate related risks based on quantitative information in the face of
uncertainty, which in turn increases demand for business professionals with sound skills and
hands-on experience with statistical data analysis. Computer-based training technologies
allow the new cadre of business professionals to obtain such a hands-on experience in an
environment where mistakes can be made and outcomes can be measured. The authors
discuss their experiences in developing a new e-learning tool designed to apply
methodological forecasting concepts to real-life business and finance problems through an
interactive self-learning and self-assessing module of online case studies.
Keywords: business statistics, decision making and analysis, e-learning, forecasting,
interactive learning, technology in education
Recent developments in technology have advanced to
the point where nearly every quantitative area in business
relies on statistical analysis. As a result, there is an increasing student interest in statistical courses and a marketdriven demand for business professionals with solid skills
in applied data analysis who can perform reliable decision
making in the face of uncertainty. Hence, statistics, being at
the intersection of multiple disciplines, goes far beyond
Correspondence should be addressed to Yulia R. Gel, University of
Waterloo, Department of Statistics and Actuarial Science, 200 University
Avenue West, Waterloo, Ontario N2L 3G1, Canada. E-mail: ygl@math.
uwaterloo.ca
simply deriving mathematical formulas in the classroom.
Instead, statistics now requires well-developed practical
skills in data analysis that are best acquired through intensive work with real-world case studies, systems, and software. The challenge is to provide this training in an
environment where mistakes can be made and outcomes
can be measured (Arsham, 2013).
Nowadays, computer-assisted learning and various
related e-learning techniques are widely recognized as flexible, fast, and efficient ways to deliver new knowledge that
helps bring just-in-time critical professional improvements
and training while minimizing time and costs involved
(Burrill, 2009; Ghaoui & Janvier, 2004). Here, by e-learning
Downloaded by [Universitas Maritim Raja Ali Haji] at 20:41 11 January 2016
216
Y. R. GEL ET AL.
we mean all forms of computer technology assisted teaching
and learning, including both in- and out-of-class experience,
with a particular focus on web-based interactive applications
(Tavangarian, Leypold, N€
olting, & R€
oser, 2004). Nevertheless, although interactive e-learning modules are already utilized for developing competence in using introductory
statistics for business and finance applications, there still
exists a lack of interactive tools which offer a balanced
combination of rigorous modern statistical methodology, indepth practical studies, and self-paced educational options
for an academic environment (Nolan & Temple Lang, 2007;
2010; Pfeil, 2006; Pratt, Davies & Connor, 2011). In particular, while there have been a variety of different e-learningin-statistics initiatives worldwide (Alldredge & Som, 2002;
Davies & Barnett, 2005; Fields & Collins, 2005; Gonzalez
& Munoz, 2006; H€ardle, Klinke, & Ziegenhagen, 2006,
2007; Hilton & Christensen, 2002), the focus of such projects is mainly on introductory statistical courses rather than
on more advanced time series and forecasting methodology.
Moreover, while many textbooks either offer a complimentary software packages (see Bowerman, O’Connell, &
Koehler, 2004; Brockwell & Davis, 2002; Newton, 1988) or
provide examples in one of the statistical programming languages (see Cowpertwait, 2009; Cryer, 2008; Shumway &
Stoffer, 2006; Yaffe, 2000), these options are typically static
and do not allow for an interactive learning environment. A
few recent initiatives in this direction include the projects
by Madsen, Albeanu, Burtschy, and Popentiu-Vladicescu
(2006) and Albeanu, Madsen, Popentiu-Vladicescu, and
Dumitru (2010) who proposed the e-learning system eTJMES and JAVA-module for exploratory time series analysis. Aydinli, H€ardle, and R€
onz (2003) discussed integration
of the statistical software R and Microsoft Excel for teaching time series, and Georgios, Stoyanova, and Onkov
(2005) focused on e-learning of trend modeling in a web
environment. Recently, Shmueli (2012) emphasized the
importance of interactive e-learning techniques in providing
hands-on experience in forecasting and offered an interactive visualization package for exploring the nature of the
observed time series.
Even less is known about the effectiveness of e-learning
tools for enhancing educational outcomes of academic
courses in more advanced statistics. In this paper, our goal
is to bridge this gap between the modern e-learning techniques and academic environment of undergraduate courses
in statistics with applications in business, finance, and
insurance, and to assess the impact of interactive web-based
e-learning on teaching and learning progress. In particular,
our key innovation consists of developing, integration, and
evaluation of a new interactive e-learning concept for a
higher level undergraduate training in forecasting. The
main idea of the proposed e-learning approach is based on
a fusion of practical forecasting case studies illustrating
various statistical modeling procedures, statistical software
(in our case R) that is used for implementation of
computational methods, and interactive capabilities of the
Learning Management System (LMS; in our case ANGEL).
It is important to emphasize that our focus is not on developing a new software package for time series modeling and
prediction but rather on elaborating interactive e-learning
techniques for teaching forecasting. Note that the proposed
e-learning concept is not restricted to statistical software R
and can be easily transferred to any other computational
software (e.g., Matlab, SPSS, SAS) as well as it is not
restricted to the LMS ANGEL but is open to any other
LMS allowing for randomization of questions and answers
in online quizzes. Although in this paper we discuss our
experience on implementation and assessment of the proposed new web-based e-learning procedures for teaching a
higher level undergraduate course in forecasting at the University of Waterloo, Canada, based on our results, we
believe that similar interactive e-learning techniques might
be found useful for distance education courses and graduate
programs in computational finance, econometrics, statistics,
actuarial sciences and other quantitative disciplines, with a
strong focus on applied aspects of modeling and data
analysis.
The article is organized as follows. The next section
describes the developed e-learning tool. Then, we assess
effectiveness of the proposed e-learning techniques. Last,
we conclude with the discussion.
METHOD
Participants
In our project, we focus on the 400-level undergraduate
course Forecasting, offered by the Department of Statistics
and Actuarial Science at the University of Waterloo three
times per year (i.e., in each of the 12-week-long winter,
spring, and fall terms). The enrolled students are supposed
to have a background in applied linear regression. Most of
the enrolled students are either from actuarial science or
from the business administration and mathematics double
degree program. The Society of Actuaries (SOA) recognizes a final grade of at least 70 in the forecasting course as
qualifying a student for a SOA credit. Hence, the course is
getting increasingly popular over last few years. Indeed,
over the period of 2005–2011, the average number of registered students is about 135, while in 2012 the enrollment
surged to more than 200.
Instrument
Given the practically oriented majors of enrolled students,
the course is very application driven and computationally
intense, and focuses more on employment of statistical
methods to real-life situations, such as predicting sales,
stock returns, or supplies, rather than on derivation of
mathematical formulas and proofs. This, along with the
Downloaded by [Universitas Maritim Raja Ali Haji] at 20:41 11 January 2016
DEVELOPING AND ASSESSING E-LEARNING TECHNIQUES
increasing enrollment numbers, requires complementary
learning skills and revision of instructional approaches in
order to improve the fit of the lecture material into cuttingedge real life business applications. To enhance educational
outcomes, our idea was to incorporate into these two
courses elements of interactive learning, such as online
self-educational illustrative case studies with self-evaluation options, an online bank of test questions and a library
of real-world data sets. We then assessed the learning process in the new environment.
While the online repository of data sets serves a complementary role of an informational source for student projects,
homework, and exercises, the core element in the developed
module is a set of real-life case studies illustrating various
modeling and forecasting techniques in business, finance
and economics, which allow students to self-assess their
understanding of the methodological and computational
aspects using online interactive tests. In particular, we
developed three such case studies involving short-,
medium-, and long-term prediction of sales of Australian
fortified wine (data are from the repository of Hyndman [n.
d.]), the 25 years of household financial obligations and
daily stock returns of an electronics company in the United
States. Each case study is divided into a sequence of chapters such that the same data are analyzed and modeled using
different approaches showing a step-by-step application of
incrementally more sophisticated techniques and methods
that are discussed in class. Some examples of those techniques are exploratory diagnostic tests for time series data,
regression analysis with endo- and exogenous variables, the
Holt-Winters smoothing techniques, the Box-Jenkins methodology (Autoregressive Moving Average/Autoregressive
Integrated Moving Average [ARIMA]/Seasonal Autoregressive Integrated Moving Average), as well as stochastic volatility and conditional heteroscedastic models.
The goal is to teach students that there is no “perfect recipe” in business forecasting or modeling, and instead to
show that the same data sets can be approached from different angles, depending on the analyst’s task. Indeed, the
developed e-learning tool, including the interactive selfassessment options, is designed in such a way that the same
data set is consecutively step-by-step approached with different modeling and prediction methods. For example, students find that the Holt-Winters smoothing is preferred for
short-term forecasts of sales of Australian dessert wine but a
seasonal ARIMA models performs better for medium- and
long-term forecasting of the same product sales. All discussed modeling procedures are complemented by a
detailed description of their implementation in R. Each
chapter is then concluded with a set of online questions in a
multiple choice and multiple selection form, so students can
self-evaluate their understanding of the exhibited data patterns, the relevant statistical tools, and their software implementation. For instance, using exploratory analysis of
various time series plots of data sets similar to those
217
discussed in the case studies, students identify the main features of the data (e.g., stock trend channels, seasonality, outliers) and get interactive feedback. This part is especially
important because various exploratory analysis techniques
play a key role in quantitative finance and econometrics.
Some questions require students to replicate the presented
analysis on their own and carry out an extension of the presented study, for example, to assess prediction performance
of the same model with and without outliers or using various variance-stabilizing transformations.
It is important to emphasize that some parameter-based
questions with a more theoretical focus are designed in
such a way that the parameters are able to take on random
values within a predefined range. For instance, we can consider identification of a linear model order and parameters
by its theoretical autocorrelation and partial autocorrelation
functions (i.e., acf and pacf, respectively). While the true
underlying structure is a moving average model of order 1,
such as MA(1), the parameter of MA(1) is allowed to take
random values within the interval of identifiability (–1,1),
hence, producing myriads of MA(1) models with the
respective acf and pacf plots. The effect is that any such
question can generate a different scenario for each student,
and the probability that a student would have the same
question twice while practicing with the tool is relatively
low. The same parameter randomization approach is undertaken for other linear and nonlinear models that are discussed in the course. After completing the online tests,
students receive a grade and can review their tests along
with the correct answers.
Note that the key difference between our approach and
the old-fashioned approach of end-of-chapter homework
problems is that in our approach problem parameters are
randomized, allowing simulations of an unbounded number
of similar questions. Hence, in contrast to the old-fashioned
end-of-chapter approach, students can see the same type of
question not once but many times allowing repeated
answering until he/she is fully satisfied with the interactive
grade and the related understanding of the theory or methodology behind the posed question. Hence, our approach
gives a much richer variety of questions.
The bank of questions is implemented using an LMS
called ANGEL, while all the plots are prepared in the statistical programming language R. All case studies along with
the corresponding online self-evaluation tests are available
to the enrolled students via the LMS course website. The
chapters of the case studies appear online after the corresponding content is discussed in class, and students can
then work with the case studies an indefinite number of
times, accessing them from anywhere. Using the LMS, we
also keep track of who uses the tool, how many times a student accesses certain case studies, what online tests are performed, what grades are received, and the detailed statistics
on each question. Students were also encouraged to keep
track of their experience in informal logbooks that may be
218
Y. R. GEL ET AL.
employed to improve usability of a future version of the
e-learning tool.
It is important to emphasize that the developed interactive e-learning tool1 is not limited to any particular statistical software (e.g., R in our case) but rather should viewed
as a new e-learning methodological approach that can be
relatively easily adapted to other software packages and
LMS, as long as such LMS allow randomization of questions and answers. To our knowledge, most of modern LMS
(e.g., Blackboard, Desire2Learn) possess these technical
capabilities.
Downloaded by [Universitas Maritim Raja Ali Haji] at 20:41 11 January 2016
Data Collection and Design
Our goal is to investigate if adding work on case studies to
the curriculum of the Forecasting course improves the
students’ level of understanding of a variety of concepts,
with the level of understanding being measured by the final
exam grade. Our study is an observational one in which we
use the students in the one available class. The complete
case studies grade is based on answers to the online
questions.
Clearly, using a randomized design (i.e., randomly
assigning no case studies to half of the students, with
the other half having case studies) may appear as a better alternative to an observational approach. However,
such a randomization approach was viewed as unethical
and not allowed by the Office of Research Ethics. So
randomization is not feasible in our situation. In turn,
assigning case studies only to volunteers/interest groups
is also not applicable due to self-selection bias. Finally,
the limitations in funding duration and scheduling
issues, as well as the technical challenges due to the
change of learning management platforms in the University of Waterloo did not allow us to perform a sequential study when all enrolled students in one given year
were assigned case studies and students from the next
year were not. Hence, we proceed with an observational
design in the one available forecasting class.
In particular, in order to see if the inclusion of case studies benefited the students, we measure the impact of the
student’s (average) case studies grade2 on his or her performance in a final exam, given information on the overall
grade point average (GPA) over all previous terms, a major,
term of study, and gender. That is, we regress the students’
final exam grade on their case studies grade, including the
average GPA from previous terms major, term of study,
and gender as covariates in an attempt to control for the
incoming intelligence of the student.3
We found that only the overall GPA and case studies
grade are significant predictors, and hence all other factors
(i.e., major, term of study, and gender) are discarded from
the further study.
Note that while final examination grades are admittedly
imperfect, they are useful as measures for evaluation of the
effectiveness of a learning tool. In contrast to standardized
tests, they are not universally comparable among students
from different universities and colleges. However, they
remain a conventional quantitative measure of learning outcome assessment adopted internally for various disciplines
by numerous educational institutions worldwide (see discussion by Ewell, 2005; Johnson, Berg, & Heddens, 2005;
Ming, 2012; Sebastianelli & Tamimi, 2011, and references
therein) and they have been validated externally by the
Mathematical Association of America (see the guidelines
of the Subcommittee on Assessment of the Committee on
the Undergraduate Program in Mathematics of the Mathematical Association of America, 2005). Moreover, while in
our case we assess the overall student knowledge in the
course, reflected by the final course grade, as a weighted
combination of four take-home assignments involving data
analysis (15%), a take-home midterm (15%), group projects
(15%), and the in-class final exam (55%), the in-class final
examination grade represents the only available measure of
student individual work because the other three assessment
categories (i.e., take-home assignments, a take-home midterm, and group projects) may explicitly or implicitly
involve teamwork rather than individual activity. Finally,
the final examination represents a collection of randomized
computer-based multiple-choice and multiple-select questions covering the whole course curriculum and can be
viewed as a substantially extended version of the online
case study questions. However, in contrast to the interactive
online case studies, students are allowed to answer each
question just once. Hence, the employed final examination
can be considered to be a version of a test-retest assessment
(see Stayhorn, 2006, and references therein).
RESULTS
Table 1 presents the estimated coefficients along with the
respective standard errors, t statistics and p values from a
multiple regression of final exam grade on the case studies
grade and the GPA, while Figure 1 displays the scatter plot
of the final exam grade versus the case studies grade. There
were 155 students enrolled into the forecasting course of
which 128 attempted all three of the available case studies.
In addition, one student attempted only the first case study,
three did only the first two, and 23 students did not do any
of case studies.) Further examination of the case studies
TABLE 1
The Results From the Multiple Regression of a Final Exam Grade
on Case Studies Grade and Grade Point Average
Item
Case studies
Grade point average
Estimate
SE
t
p
.1359
.7218
.0159
.0792
2.296
9.116
.0234