08832323.2011.592870
Journal of Education for Business
ISSN: 0883-2323 (Print) 1940-3356 (Online) Journal homepage: http://www.tandfonline.com/loi/vjeb20
The Use of Performance Control Charts in
Business Schools: A Tool for Assessing Learning
Outcomes
Michael J. Cervetti , Marla B. Royne & J. Michael Shaffer
To cite this article: Michael J. Cervetti , Marla B. Royne & J. Michael Shaffer (2012) The Use of
Performance Control Charts in Business Schools: A Tool for Assessing Learning Outcomes,
Journal of Education for Business, 87:4, 247-252, DOI: 10.1080/08832323.2011.592870
To link to this article: http://dx.doi.org/10.1080/08832323.2011.592870
Published online: 29 Mar 2012.
Submit your article to this journal
Article views: 91
View related articles
Full Terms & Conditions of access and use can be found at
http://www.tandfonline.com/action/journalInformation?journalCode=vjeb20
Download by: [Universitas Maritim Raja Ali Haji]
Date: 11 January 2016, At: 22:02
JOURNAL OF EDUCATION FOR BUSINESS, 87: 247–252, 2012
C Taylor & Francis Group, LLC
Copyright
ISSN: 0883-2323 print / 1940-3356 online
DOI: 10.1080/08832323.2011.592870
The Use of Performance Control Charts in Business
Schools: A Tool for Assessing Learning Outcomes
Michael J. Cervetti, Marla B. Royne, and J. Michael Shaffer
Downloaded by [Universitas Maritim Raja Ali Haji] at 22:02 11 January 2016
University of Memphis, Memphis, Tennessee, USA
The authors propose the use of performance control charts as a useful tool for tracking student
assessments that rely upon faculty administered exams and projects as well as assessing
teacher performance. This quantitative, formative evaluation process can be integrated into
the classroom during an actual semester and utilized as a means to improving performance in
future years.
Keywords: assurance of learning, performance control chart, quality control chart
Considerable research has assessed how and where faculty
can improve the outcomes of student learning. One particular issue that has been raised is the difference between
what students actually learn and the skill sets of graduating students upon entering the business workforce. Although
there are several potential explanations for this academic
performance–workplace performance gap, one may be related to communication and collaboration between students
and faculty, which may interfere with the learning process
in the classroom. The purpose of this study was to propose
a tool that can be integrated into the classroom during the
semester and utilized as a means to improving both student
and teacher performance
ASSURANCE OF STUDENT LEARNING
Benjamin (2009) utilized a rubric system to develop clear
directions and drive continuous improvement, as well as
to increase interaction and collaboration between students
and faculty in achieving overall learning goals. Benjamin
argued that instructors need to use “language that everyone
understands, turning expectations into detailed performance
contracts, monitoring performance regularly, and rewarding
and enhancing system deployment” (p. 40). The rubric system Benjamin used was generated from quality levels where
teachers were evaluated on continuous improvement on a
Correspondence should be addressed to Marla B. Royne, University
of Memphis, Fogelman College of Business & Economics, Department of
Marketing and Supply Chain Management, Memphis, TN 38152, USA.
E-mail: mstaffrd@memphis.edu
rating scale from level 1 to level 4. Each of these ratings
served as benchmarks for teachers’ steady progression to the
next level. As a result of the clear expectations, results were
successful. For example, one school’s participants showed a
46% achievement to level 3 in the initial year of deployment;
there was also a 93% success rate in achieving level 1 or
higher (Benjamin). Such results suggest that clear directions
contribute to a teacher’s ability to assess and make adjustments on a regular basis to merit continuous improvement.
One student involved in the research stated, “I enjoy that
we get to share our thoughts. We can tell if we have problems or if we’re struggling, but most of all it makes me
feel like I have a voice, and I control what happens with
the classroom” (Benjamin, 2009, p. 44). Clearly, these are
important factors for the collaboration of students and faculty. When students have no voice in learning outcomes, the
idea of continuous improvement on areas of concern lack the
mutual direction from all parties involved. When areas of a
curriculum are not fully understood, the customers—in this
case, the students—provide potential corrective actions that
ensure learning outcomes are exceeded.
Spruell, Hawkins, and Vicknair (2009) explored a richer
learning environment that included cocurricular programs.
They noted that although many universities incorporate some
type of cocurricular experiences, such expenses vary from
one institution to another based on the mission and goals of
the institution. Spruell et al. also pointed out that cocurricular
activities aim to extend the depth and richness of the student
learning environment. This concept and model directly relate
to a student centric environment that promotes collaboration
and adds value to students’ educational learning outcomes.
Weldy and Turnipseed (2010) examined the direct
and indirect learning measures based on the completion
Downloaded by [Universitas Maritim Raja Ali Haji] at 22:02 11 January 2016
248
M. J. CERVETTI ET AL.
of a real-world project for a required management course.
Upon completion of the study, students were given a
questionnaire as an indirect measure of learning based on
their own perceptions; these responses were then compared
with the direct measures from the teachers. This comparison
revealed that a gap did indeed exist between the direct
measures and the indirect measures. Specifically, the indirect
measures indicated a higher perception on two of the
four methods used, indicating that students’ perceptions
of what they had learned were higher than what teachers
perceived had actually been learned. Students also felt
that when seeking management information, conducting
research, and discovering trends and key issues, their
abilities and capabilities were there. Overall, this study
provided evidence that real-world-based projects can be
used as pedagogical tools in business curriculums to advance
the development and improve the learning outcomes of the
students.
Drost, Hanson, Molstad, Peake, and Newman (2008) integrated a portfolio approach into the course program to advance the outcomes of student learning. This allowed faculty
members to move away from student tests as the exclusive
measure of course assessment (Drost et al., 2008). The study
first developed five main learning goals that served as the
basis for creating the 12 specific learning objectives that
were used to evaluate the assessment of the program. For
the assessment to be evaluated, professors had to assign a
project that included at least four of the 12 learning objectives. These assignments allowed the students to express their
highest academic performance and understanding of the objectives chosen by the instructors. Grading for these assignments utilized a Likert-type scale for summative evaluation
of the learning objectives. However, this form of assessment
did not allow for any type of formative evaluation from the
students. Hence, Drost et al. created a self-evaluation essay
prepared from each individual student to address perceptions
in indirect measures. Unfortunately, administration noted a
portfolio storage issue, and these concerns led to the submission of electronic files by students; these were held for
a maximum of one year. Moreover, the administration concluded that this was “not one of the more administratively
‘efficient’ means for conducting outcome assessment” (Drost
et al., p. 111).
From an academic quality perspective, students felt a
portfolio assignment was much better than exams for content
application, and faculty believed this assignment should be
implemented across the board to improve the quality of the
programs. In fact, most students have expressed that the
assignments “were genuine learning experiences” (Drost et
al., 2008, p. 111). Unfortunately, the summative evaluations
of the portfolio systems revealed that some learning objectives were not addressed efficiently, but this did not mean
they were not included in the portfolio; rather, they were
not addressed to the desired measures. Results showed the
summative evaluations were less useful than the indirect and
formative measures of the essay evaluations provided by the
students, but these summative evaluations from the portfolio
helped to ensure that the set learning objectives were being
achieved.
Gardiner, Corbitt, and Adams (2010) proposed a practical,
how-to model for the implementation of the assurance of
learning process. The first step is to develop 6–8 learning
goals and outcomes for each program. Although these goals
should be owned by the faculty, they should directly support
the mission of the program itself (Gardiner et al.). After
the identification of these goals, specific learning objectives
should be developed to measure the assessment and progress
toward the specific learning goals.
The second step plans the assessment whereas the third
step manages the assessment execution. The evaluators must
receive the appropriate tools and rubrics for the assignment.
The fourth step analyzes the assessment data to achieve actionable results and generate reports that can provide specific
actions to improve measures (Gardiner et al., 2010). Gardiner
et al. noted that assessment systems that can track learning
outcomes for individual students and student cohorts are becoming more popular.
Identifying and agreeing on program improvements is the
next step. This is critical because lack of agreement generally results in lack of action, and the process of closing the
loop is not accomplished. Actions should then be evaluated
in terms of costs, significance, and time to reach an acceptable implementation method. Execution of these program
improvements follows, and this step should include the measurement of the intervention, the execution performance, and
reassessed post implementation. Ultimately, the end result of
this step should create a higher performance on the objectives set and help determine what changes are appropriate
for the future assessments. The final step requires recording
details of the previous steps. Overall, incorporating many of
these steps identified by Gardiner in the use of our proposed
performance control chart (PCC) will simplify the process
for assessment in teacher performance and the assurance of
learning by the students.
Stivers and Phillips (2009) found that faculty involvement
must include all members; attempts to speed up the process
by using a small group may not meet the purpose that was set.
Additionally, the involvement provides ownership and experience for the members to make improvements. Departments
and teams must acquire responsibility for the assessment, and
members must reach agreement on the goals and outcomes
because this framework is the backbone for the assessment
and potential improvements in the curriculum. For the process to work appropriately, funding should be administered
to adequately cover all needed assessment resources whether
tests, faculty-release time, training, data collection, or outside sources. Direct measures should be supported by the use
of indirect measures, and closing the loop assessment data
must be collected and used by the faculty to make changes
in materials, courses, and methods.
Downloaded by [Universitas Maritim Raja Ali Haji] at 22:02 11 January 2016
PERFORMANCE CONTROL CHARTS
Numerous studies have been designed and implemented
to further assess and document successful strategies for improving student learning. However, the model that we propose here not only meets the requirements listed previously,
but also helps define teacher performance. There are many
definitions of teacher performance and many ways to measure how well a teacher performs in the classroom. Two of the
most popular methods of measuring teacher performance in
higher education are through student evaluation of teachers
(SETs) and final class evaluations. Did the students like their
teacher? How many students passed the class and how many
students failed? Although such measures provide some feedback that can be incorporated into future semesters, they are
basically end-of-the-year, summative assessments that have
little, if any, value in terms of professional development during the school year. Finally, annual evaluations must be done
to achieve continuous improvements. Hence, we propose a
model that includes most of these issues.
249
described subsequently—are keys to the development of a
successful experience for students. It is important to note,
however, that the model proposed here does not seek to duplicate the exact QCC used in factories. Rather, we utilize
only the basic structure of the QCC and adapt it to a classroom environment.
Horsburgh (2010) argued that that a collaborative effort
between the teacher and the student produces the best teacher
with the best learning outcomes for the student. At the same
time, the basic QCC process provides a means for measuring
success and documenting professional development activities related to the particular subject being taught during the
school year—not at the end of the school year. And although
the use of a QCC is quantitative and tactical in nature, it is
also a philosophy of doing business, which is the approach
that we propose here. Hence, within the educational environment, it is essentially an entire school system working
toward a common goal of a quality curriculum and a chance
for continuous professional development. Because of this,
Melvin (1993) stated,
THE QUALITY CONTROL CHART
Formative evaluation, continuous improvement, and assurance of learning all examine the subject as it is being taught,
not after completion of a semester. Their purpose is to provide ongoing diagnosis and feedback so that teachers can
modify classroom activities during the year, if necessary.
The approach presented here allows the instructor that flexibility while building from one of the most basic of business
techniques, the quality control chart, developed by Walter A.
Shewhart while working for Bell Labs in the 1920’s and further refined by W. Edwards Deming. Shewhart believed that
“lack of information greatly hampered the efforts of control
and management processes in a production environment,”
(SkyMark, 2010). Shewhart’s ideas of combining creative
management with statistical analysis were noteworthy then
as now. Although quality control types of quantitative tools
are highly successful in the industrial and business world,
Schazmann, Regan, Ross, Dermot, and Pauli (2009) noted
that “quality control measures are less prevalent in teaching
laboratories (classrooms) than commercial settings possibly
owing to a lack of commercial incentives or teaching resources” (p. 1).
We propose that a modification of the quality control chart
(QCC) process will imbue the teacher and students with a
spirit of working together to reach a common goal: assurance of learning. We believe it provides “collaborative effort” between teachers and students—something presently
lacking in most higher education classrooms—and augment
the effort where it does exist. Further, we propose that a QCC
type of mechanism placed in the classroom will provide the
tool needed by teachers for continuous improvement. The
idea presented in this article draws its structure similar to
how Shewhart envisioned the QCC working in the factory,
that constant evaluation of teaching practices—via the PCC
The goal is to create an educational system that is capable of
constantly improving the quality of school-related processes
in order to influence the quality of educational outcomes in
the classroom, the school, the district, and the community.
Our efforts are founded on the premise that by focusing on
the improvement of customer-defined quality in both services
and products, a school system will experience both increased
public support and high staff and student satisfaction. (p. 74)
Hence, we propose a PCC—a modification of the
QCC—as a means of assessing learning outcomes for both
the instructor and the students. This allows us to focus on
the aspect of quality control that helps to improve both class
performance and the students’ perception of the classroom
environment.
PERFORMANCE CONTROL CHARTS
Generally, quality control charts are a simple graphic that
allows one to determine when a process gets out of control
(Savic, 2006). Figure 1 displays a typical quality control
chart; it utilizes a center line (CL), an upper control limit
(UCL), and a lower control limit (LCL). The UCL and LCL
are horizontal lines situated above and below the CL. The
CL represents the process average when it is in a state of
statistical control. Any points above the UCL or below the
LCL are considered out of control. That is, if the process is
out of control in the upper or lower range, then an assignable
cause must be documented and removed.
However, we propose an important difference in how the
control chart is implemented in a classroom. While a typical
quality control chart allows an engineer to detect deviation
from the process mean—the CL—and to make adjustments
250
M. J. CERVETTI ET AL.
Performance Control Chart
Control Level
---------------------------------------------------- UCL
{
----------------------------------------------------
•Could be used as resource for other teachers
•Document successful strategy for sharing purposes (teamwork)
•Document successful strategy in teacher porƞolio
Center Line
------------------------------------------------------------------------------------------------------- LCL
UCL
(Standard)
The upper control limit (UCL) provides a basis for
knowing what is expected. The UCL sets the
standard or goal for the course.
3
---------------------------------------------------------------CL (Norm)
The lower control limit (LCL) indicates the point at
which some sort of correcƟve acƟon or remediaƟon
is needed.
2
Time
LCL
1
{
Downloaded by [Universitas Maritim Raja Ali Haji] at 22:02 11 January 2016
FIGURE 1 A typical quality control chart, with upper control limit (UCL),
center line (CL), and lower control limit (LCL) (color figure available
online).
to the process only when needed, the control chart presented
here supports change and what might be considered out of
control processes. This is an important distinction from the
traditional QCC and Deming’s basic argument that changes
should be implemented only when a process is out of control.
Moreover, the control chart must be adapted to each individual instructor and classroom based on the previous semester’s
learning outcomes. For these reasons, we refer to the control
chart here as a PCC. One of the overall goals of the PCC
is to allow the instructor to analyze the information gleaned
from the highs and lows of the graph and to make appropriate
changes in the curriculum as needed. By using the PCC, an
instructor can reduce two possible errors: (a) taking action
when none is needed and (b) not taking action when it is
needed (Savic, 2006). Although we recognize this approach
is in contrast to Deming’s philosophy, we also argue that
it is critical here because teaching is a service-based product whereas typical control charts are used for goods-based
manufacturers. The now well-accepted differences between
services and goods generally require different approaches to
assessments of service quality (Parasuraman, Zeithaml, &
Berry, 1985).
The PCC in Figure 2 illustrates the established norm for a
given course based on the achievement level of the previous
semester in that course. A statistical confidence interval will
be the mechanism utilized to establish a PCC that will determine the expected performance level based on the previous
semester’s data (the CL of the confidence interval) and, at
the same time, provide a higher level standard of learning
for the present year (the UCL). It is important to note that
the UCL is only an artificial goal; it will likely take more
than one semester to reach the current UCL. Further, the
UCL will change from semester to semester, depending on
the previous semester’s mean and standard deviation. However, if a class is at the 100% level (perfect performance),
then the PCC could be used to simply monitor the future
performance of that class. Finally, a PCC will track the class
performance variance, or deviation from the norm, in either
a positive direction (above the norm at point 3 in Figure 2)
or in a negative direction (below the norm at points 1 and 2
FIGURE 2
The center line (CL) establishes the norm based on
last year’s performance.
•Find assignable cause(s)
•UƟlize check list of resources for remediaƟon
•Consult lead teacher or team teachers
•Consult curriculum specialist
•Teacher discerns problem area(s) independently, makes
adjustments
•Document assignable cause(s) and remediaƟon in
teacher porƞolio
Example of a performance control chart.
in Figure 2). As noted, a PCC gives the teacher the mechanism necessary to discern specifically where remediation is
needed.
Point 3 in Figure 2 indicates that this teacher could be
used as a resource for other teachers. Successful strategies
should be documented and placed in the teacher portfolio for
sharing purposes with other teachers (teamwork) and/or evaluation by administrators. Points 1 and 2 in Figure 2 indicate
where modification might be needed in the course. Specifically, an assignable cause should be diagnosed. A checklist of
resources should be developed and utilized. After the teacher
finds an assignable cause(s) and documents the remediation
utilized, it should then be placed in the teacher portfolio for
future reference or evaluation by administrators. As noted,
the PCC is not trying to emulate Deming’s model of total quality management with regard to process control. The
PCC only utilizes the structural concept of the QCC to highlight the highs and lows of the course content. In fact, the
PCC is a tool that is utilized to actually aid the instructor in
moving the CL, which, as noted, is the opposite of Deming’s
approach. Action is encouraged at points 1 and 2 in Figure
2. It is important to emphasize that a PCC is not a searchand-destroy tool, but rather one to search and remediate. As
noted, the PCC helps the teacher discern where to take action
when action is needed.
For example, a typical business statistics instructor could,
using Gardiner et al.’s (2010) model discussed previously,
develop a set of course-level learning objectives based on the
learning goals relevant to that course. These objectives might
be that students could demonstrate the required proficiency
in various quantitative tools such as regression, forecasting, simulation modeling, linear programming, and queuing.
These course objectives would be evaluated through courseembedded measures that relate to specific course assignments
(Association to Advance Collegiate Schools of Business International, 2007), which—in the case of the model we are
proposing—could also mean class tests. Figure 3 represents
Downloaded by [Universitas Maritim Raja Ali Haji] at 22:02 11 January 2016
PERFORMANCE CONTROL CHARTS
FIGURE 3
251
A business statistics class performance control chart (color figure available online).
an actual PCC. This PCC was developed by calculating the
mean for all of the tests that were given during the previous
semester. The UCL and LCL were calculated by adding and
subtracting one standard deviation to the mean. As research
is conducted and data are collected, these UCL and LCL can
and should be adjusted and adapted to meet the instructor’s
unique needs for the class. After the UCL (mean + standard deviation), CL (overall class mean), and LCL (mean –
standard deviation) is established and drawn, then means for
each individual test are plotted on the chart. This PCC shows
that the class, as a whole, performed well above the overall
mean on the regression component of the class objectives.
At the same time, the PCC also indicates that the simulation
component for the class was well below the overall class
mean.
This low point on the PCC, the point where the simulation assessment approaches the LCL, is also the point in the
model where the faculty member could utilize a checklist
of resources for remediation, document the remediation utilized, and place documentation in the teacher portfolio for
future reference or evaluation by administrators. It should
be remembered that students also see the PCC and share in
the remediation process; it is important for the psychosocial
climate of the classroom that students share in the responsibility of the improvement process. As Gardiner et al. (2010)
noted,
any model of quality in higher education needs to produce
evidence-based improvement in learning, clearly identify
needed quality improvements in either the processes or the
outcomes, and involve the faculty in the evaluation of learning and the implementations of the improvements. (p. 140)
The PCC model does just that while also involving students in the remediation process, something recommended
by several academicians (e.g., Stivers & Phillips, 2009)
Figure 4 illustrates the flow of the course as the teacher
utilizes the formative information from the PCC to make improvements in the course during the school year. Essentially,
the teacher assesses the performance of students via the PCC,
assigns a diagnosis, and subsequently implements a course
of action with the aid of a list of available resources and
strategies. The remediation could be as simple as spending
more time on the topic or as drastic as restructuring a part of
the course. In such cases, some strategies might have to be
implemented the following school year. The expectation is
that as the teacher works consistently on the weak points of
the course or curriculum, over time the norm or CL will increase. Each PCC should be analyzed in a longitudinal format
to discern if the same weak points occur through the years
for a particular teacher and to see if significant improvement
is made on the CL, or class average, of the PCC from year to
year.
252
M. J. CERVETTI ET AL.
Management of Student Performance
Teacher manages the curriculum and improves the curriculum in a conƟnuous cycle.
Established Curriculum
===================
Teacher Has Control and
Ownership
Manages
Performance
Quality
Assess
Performance
Improvement
ImplementaƟon
Improve
Performance
Quality
Select
Improvement
Area(s) Via
PCC
REFERENCES
Downloaded by [Universitas Maritim Raja Ali Haji] at 22:02 11 January 2016
----------------------------Find
Assignable
Causes
FIGURE 4
coming an increasingly critical concern in the educational
environment.
In short, the PCC is a formative evaluation process that can
be integrated into the classroom during an actual semester and
utilized for remediation when and where the instructor deems
necessary. As a result, the PCC has considerable potential
for helping teachers improve their classroom skills, which
should translate into enhanced learning outcomes that can
benefit the student and the ultimate employer.
UƟlize Available
Resources For
Improvement
Strategy
Flowchart of course remediation process.
The PCC should be displayed clearly on the wall of the
classroom where it is easily visible by students and teachers.
The teacher should encourage the students to take ownership
in the class progress and point out goals for each test. This
should encourage teamwork for the class and collaboration
with the teacher, and subsequently, a more favorable classroom environment. As pointed out by Dorman (2002), “without a consideration of the crucial role of the psychosocial
environment of classrooms, educational productivity cannot
be optimized” (p. 1).
Although the PCC may have some value for the Association to Advance Collegiate Schools of Business International accreditation process, its primary role is improvement
in the classroom. For example, it can be used to encourage self-evaluation of teacher performance by providing a
mechanism to chart progress, pinpoint problem areas in the
curriculum as they occur, adapt to unique school or classroom environment, provide immediate feedback as concerns
student progress toward reaching established goals, focus
on prevention of student failure at all stages of instruction,
provide a mechanism for continuous improvement of curriculum, provide a data base for assessing the results of any
remedial action that was taken, replace bureaucratic control
and management with self-regulation and self-motivation,
and promote teamwork among teachers of the same course
by sharing results in a meaningful way. Moreover, the PCC
provides a means of accountability, an issue that is be-
Association to Advance Collegiate Schools of Business International.
(2007). AACSB assurance of learning standards: An interpretation.
Tampa, FL: AACSB International Accreditation Coordinating Committee
and AACSB International Accreditation Quality Committee. Retrieved
from http://www/aacsb.edu/publications/papers/accreditation/assuranceof-learning.pdf
Benjamin, S. (2009). Keeping score: Use rubrics to advance continuous
improvement in schools. Quality Progress, 42, 40–44.
Dorman, J. (2002). Classroom environment research: Progress and possibilities. Queensland Journal of Educational Research, 18, 112–140.
Drost, D., Hanson, L., Molstad, C., Peake, L., & Newman, E. (2008) Outcome assessment of a management program using a portfolio approach:
Lessons learned. American Journal of Business Education, 1, 105–114.
Gardiner, L., Corbitt, G., & Adams, S. (2010). Program assessment: Getting to a practical how-to model. Journal of Education for Business, 85,
139–144.
Horsburgh, D. (2010, March 20). Measuring the performance of teachers
and students. Denver Post. Retrieved from http://blogs.denverpost.com/
eletters/ 2010/03/20/measuring-performance-of-teachers-and-students2-letters/8522/
Melvin, C. (1993). Application of control charts to an educational system.
Performance Improvement Quarterly, 6(3), 74–85.
Parasuraman, A., Zeithaml, V., & Berry, L (1985). A conceptual model
of service quality and its implications for future research. Journal of
Marketing, 49(4), 41–50.
Savic, M. (2006). p-Charts in the quality control of the grading process in
the high education. Panoeconomicus, 53, 335–347.
Schazmann, B., Regan, F., Ross, M., Dermot, D., & Pauli, B. (2009). Introducing quality control in the chemistry teaching laboratory using control
charts. Journal of Chemistry Education, 86, 1085–1086.
SkyMark. (2010). Walter Shewhart: The grandfather of total quality management. Retrieved from http://www.skymark.com/resources/leaders/
shewart.asp
Spruell, J., Hawkins, A., & Vicknair, D. (2009). Managing learning experiences in an AACSB environment: Beyond the classroom. American
Journal of Business Education, 2, 75–77.
Stivers, B., & Phillips, J. (2009). Assessment of student learning: A fast
track experience. Journal of Education for Business, 85, 258–262.
Weldy, T., & Turnipseed, D. (2010). Assessing and improving learning in
business schools: Direct and indirect measures of learning. Journal of
Education for Business, 85, 268–273.
ISSN: 0883-2323 (Print) 1940-3356 (Online) Journal homepage: http://www.tandfonline.com/loi/vjeb20
The Use of Performance Control Charts in
Business Schools: A Tool for Assessing Learning
Outcomes
Michael J. Cervetti , Marla B. Royne & J. Michael Shaffer
To cite this article: Michael J. Cervetti , Marla B. Royne & J. Michael Shaffer (2012) The Use of
Performance Control Charts in Business Schools: A Tool for Assessing Learning Outcomes,
Journal of Education for Business, 87:4, 247-252, DOI: 10.1080/08832323.2011.592870
To link to this article: http://dx.doi.org/10.1080/08832323.2011.592870
Published online: 29 Mar 2012.
Submit your article to this journal
Article views: 91
View related articles
Full Terms & Conditions of access and use can be found at
http://www.tandfonline.com/action/journalInformation?journalCode=vjeb20
Download by: [Universitas Maritim Raja Ali Haji]
Date: 11 January 2016, At: 22:02
JOURNAL OF EDUCATION FOR BUSINESS, 87: 247–252, 2012
C Taylor & Francis Group, LLC
Copyright
ISSN: 0883-2323 print / 1940-3356 online
DOI: 10.1080/08832323.2011.592870
The Use of Performance Control Charts in Business
Schools: A Tool for Assessing Learning Outcomes
Michael J. Cervetti, Marla B. Royne, and J. Michael Shaffer
Downloaded by [Universitas Maritim Raja Ali Haji] at 22:02 11 January 2016
University of Memphis, Memphis, Tennessee, USA
The authors propose the use of performance control charts as a useful tool for tracking student
assessments that rely upon faculty administered exams and projects as well as assessing
teacher performance. This quantitative, formative evaluation process can be integrated into
the classroom during an actual semester and utilized as a means to improving performance in
future years.
Keywords: assurance of learning, performance control chart, quality control chart
Considerable research has assessed how and where faculty
can improve the outcomes of student learning. One particular issue that has been raised is the difference between
what students actually learn and the skill sets of graduating students upon entering the business workforce. Although
there are several potential explanations for this academic
performance–workplace performance gap, one may be related to communication and collaboration between students
and faculty, which may interfere with the learning process
in the classroom. The purpose of this study was to propose
a tool that can be integrated into the classroom during the
semester and utilized as a means to improving both student
and teacher performance
ASSURANCE OF STUDENT LEARNING
Benjamin (2009) utilized a rubric system to develop clear
directions and drive continuous improvement, as well as
to increase interaction and collaboration between students
and faculty in achieving overall learning goals. Benjamin
argued that instructors need to use “language that everyone
understands, turning expectations into detailed performance
contracts, monitoring performance regularly, and rewarding
and enhancing system deployment” (p. 40). The rubric system Benjamin used was generated from quality levels where
teachers were evaluated on continuous improvement on a
Correspondence should be addressed to Marla B. Royne, University
of Memphis, Fogelman College of Business & Economics, Department of
Marketing and Supply Chain Management, Memphis, TN 38152, USA.
E-mail: mstaffrd@memphis.edu
rating scale from level 1 to level 4. Each of these ratings
served as benchmarks for teachers’ steady progression to the
next level. As a result of the clear expectations, results were
successful. For example, one school’s participants showed a
46% achievement to level 3 in the initial year of deployment;
there was also a 93% success rate in achieving level 1 or
higher (Benjamin). Such results suggest that clear directions
contribute to a teacher’s ability to assess and make adjustments on a regular basis to merit continuous improvement.
One student involved in the research stated, “I enjoy that
we get to share our thoughts. We can tell if we have problems or if we’re struggling, but most of all it makes me
feel like I have a voice, and I control what happens with
the classroom” (Benjamin, 2009, p. 44). Clearly, these are
important factors for the collaboration of students and faculty. When students have no voice in learning outcomes, the
idea of continuous improvement on areas of concern lack the
mutual direction from all parties involved. When areas of a
curriculum are not fully understood, the customers—in this
case, the students—provide potential corrective actions that
ensure learning outcomes are exceeded.
Spruell, Hawkins, and Vicknair (2009) explored a richer
learning environment that included cocurricular programs.
They noted that although many universities incorporate some
type of cocurricular experiences, such expenses vary from
one institution to another based on the mission and goals of
the institution. Spruell et al. also pointed out that cocurricular
activities aim to extend the depth and richness of the student
learning environment. This concept and model directly relate
to a student centric environment that promotes collaboration
and adds value to students’ educational learning outcomes.
Weldy and Turnipseed (2010) examined the direct
and indirect learning measures based on the completion
Downloaded by [Universitas Maritim Raja Ali Haji] at 22:02 11 January 2016
248
M. J. CERVETTI ET AL.
of a real-world project for a required management course.
Upon completion of the study, students were given a
questionnaire as an indirect measure of learning based on
their own perceptions; these responses were then compared
with the direct measures from the teachers. This comparison
revealed that a gap did indeed exist between the direct
measures and the indirect measures. Specifically, the indirect
measures indicated a higher perception on two of the
four methods used, indicating that students’ perceptions
of what they had learned were higher than what teachers
perceived had actually been learned. Students also felt
that when seeking management information, conducting
research, and discovering trends and key issues, their
abilities and capabilities were there. Overall, this study
provided evidence that real-world-based projects can be
used as pedagogical tools in business curriculums to advance
the development and improve the learning outcomes of the
students.
Drost, Hanson, Molstad, Peake, and Newman (2008) integrated a portfolio approach into the course program to advance the outcomes of student learning. This allowed faculty
members to move away from student tests as the exclusive
measure of course assessment (Drost et al., 2008). The study
first developed five main learning goals that served as the
basis for creating the 12 specific learning objectives that
were used to evaluate the assessment of the program. For
the assessment to be evaluated, professors had to assign a
project that included at least four of the 12 learning objectives. These assignments allowed the students to express their
highest academic performance and understanding of the objectives chosen by the instructors. Grading for these assignments utilized a Likert-type scale for summative evaluation
of the learning objectives. However, this form of assessment
did not allow for any type of formative evaluation from the
students. Hence, Drost et al. created a self-evaluation essay
prepared from each individual student to address perceptions
in indirect measures. Unfortunately, administration noted a
portfolio storage issue, and these concerns led to the submission of electronic files by students; these were held for
a maximum of one year. Moreover, the administration concluded that this was “not one of the more administratively
‘efficient’ means for conducting outcome assessment” (Drost
et al., p. 111).
From an academic quality perspective, students felt a
portfolio assignment was much better than exams for content
application, and faculty believed this assignment should be
implemented across the board to improve the quality of the
programs. In fact, most students have expressed that the
assignments “were genuine learning experiences” (Drost et
al., 2008, p. 111). Unfortunately, the summative evaluations
of the portfolio systems revealed that some learning objectives were not addressed efficiently, but this did not mean
they were not included in the portfolio; rather, they were
not addressed to the desired measures. Results showed the
summative evaluations were less useful than the indirect and
formative measures of the essay evaluations provided by the
students, but these summative evaluations from the portfolio
helped to ensure that the set learning objectives were being
achieved.
Gardiner, Corbitt, and Adams (2010) proposed a practical,
how-to model for the implementation of the assurance of
learning process. The first step is to develop 6–8 learning
goals and outcomes for each program. Although these goals
should be owned by the faculty, they should directly support
the mission of the program itself (Gardiner et al.). After
the identification of these goals, specific learning objectives
should be developed to measure the assessment and progress
toward the specific learning goals.
The second step plans the assessment whereas the third
step manages the assessment execution. The evaluators must
receive the appropriate tools and rubrics for the assignment.
The fourth step analyzes the assessment data to achieve actionable results and generate reports that can provide specific
actions to improve measures (Gardiner et al., 2010). Gardiner
et al. noted that assessment systems that can track learning
outcomes for individual students and student cohorts are becoming more popular.
Identifying and agreeing on program improvements is the
next step. This is critical because lack of agreement generally results in lack of action, and the process of closing the
loop is not accomplished. Actions should then be evaluated
in terms of costs, significance, and time to reach an acceptable implementation method. Execution of these program
improvements follows, and this step should include the measurement of the intervention, the execution performance, and
reassessed post implementation. Ultimately, the end result of
this step should create a higher performance on the objectives set and help determine what changes are appropriate
for the future assessments. The final step requires recording
details of the previous steps. Overall, incorporating many of
these steps identified by Gardiner in the use of our proposed
performance control chart (PCC) will simplify the process
for assessment in teacher performance and the assurance of
learning by the students.
Stivers and Phillips (2009) found that faculty involvement
must include all members; attempts to speed up the process
by using a small group may not meet the purpose that was set.
Additionally, the involvement provides ownership and experience for the members to make improvements. Departments
and teams must acquire responsibility for the assessment, and
members must reach agreement on the goals and outcomes
because this framework is the backbone for the assessment
and potential improvements in the curriculum. For the process to work appropriately, funding should be administered
to adequately cover all needed assessment resources whether
tests, faculty-release time, training, data collection, or outside sources. Direct measures should be supported by the use
of indirect measures, and closing the loop assessment data
must be collected and used by the faculty to make changes
in materials, courses, and methods.
Downloaded by [Universitas Maritim Raja Ali Haji] at 22:02 11 January 2016
PERFORMANCE CONTROL CHARTS
Numerous studies have been designed and implemented
to further assess and document successful strategies for improving student learning. However, the model that we propose here not only meets the requirements listed previously,
but also helps define teacher performance. There are many
definitions of teacher performance and many ways to measure how well a teacher performs in the classroom. Two of the
most popular methods of measuring teacher performance in
higher education are through student evaluation of teachers
(SETs) and final class evaluations. Did the students like their
teacher? How many students passed the class and how many
students failed? Although such measures provide some feedback that can be incorporated into future semesters, they are
basically end-of-the-year, summative assessments that have
little, if any, value in terms of professional development during the school year. Finally, annual evaluations must be done
to achieve continuous improvements. Hence, we propose a
model that includes most of these issues.
249
described subsequently—are keys to the development of a
successful experience for students. It is important to note,
however, that the model proposed here does not seek to duplicate the exact QCC used in factories. Rather, we utilize
only the basic structure of the QCC and adapt it to a classroom environment.
Horsburgh (2010) argued that that a collaborative effort
between the teacher and the student produces the best teacher
with the best learning outcomes for the student. At the same
time, the basic QCC process provides a means for measuring
success and documenting professional development activities related to the particular subject being taught during the
school year—not at the end of the school year. And although
the use of a QCC is quantitative and tactical in nature, it is
also a philosophy of doing business, which is the approach
that we propose here. Hence, within the educational environment, it is essentially an entire school system working
toward a common goal of a quality curriculum and a chance
for continuous professional development. Because of this,
Melvin (1993) stated,
THE QUALITY CONTROL CHART
Formative evaluation, continuous improvement, and assurance of learning all examine the subject as it is being taught,
not after completion of a semester. Their purpose is to provide ongoing diagnosis and feedback so that teachers can
modify classroom activities during the year, if necessary.
The approach presented here allows the instructor that flexibility while building from one of the most basic of business
techniques, the quality control chart, developed by Walter A.
Shewhart while working for Bell Labs in the 1920’s and further refined by W. Edwards Deming. Shewhart believed that
“lack of information greatly hampered the efforts of control
and management processes in a production environment,”
(SkyMark, 2010). Shewhart’s ideas of combining creative
management with statistical analysis were noteworthy then
as now. Although quality control types of quantitative tools
are highly successful in the industrial and business world,
Schazmann, Regan, Ross, Dermot, and Pauli (2009) noted
that “quality control measures are less prevalent in teaching
laboratories (classrooms) than commercial settings possibly
owing to a lack of commercial incentives or teaching resources” (p. 1).
We propose that a modification of the quality control chart
(QCC) process will imbue the teacher and students with a
spirit of working together to reach a common goal: assurance of learning. We believe it provides “collaborative effort” between teachers and students—something presently
lacking in most higher education classrooms—and augment
the effort where it does exist. Further, we propose that a QCC
type of mechanism placed in the classroom will provide the
tool needed by teachers for continuous improvement. The
idea presented in this article draws its structure similar to
how Shewhart envisioned the QCC working in the factory,
that constant evaluation of teaching practices—via the PCC
The goal is to create an educational system that is capable of
constantly improving the quality of school-related processes
in order to influence the quality of educational outcomes in
the classroom, the school, the district, and the community.
Our efforts are founded on the premise that by focusing on
the improvement of customer-defined quality in both services
and products, a school system will experience both increased
public support and high staff and student satisfaction. (p. 74)
Hence, we propose a PCC—a modification of the
QCC—as a means of assessing learning outcomes for both
the instructor and the students. This allows us to focus on
the aspect of quality control that helps to improve both class
performance and the students’ perception of the classroom
environment.
PERFORMANCE CONTROL CHARTS
Generally, quality control charts are a simple graphic that
allows one to determine when a process gets out of control
(Savic, 2006). Figure 1 displays a typical quality control
chart; it utilizes a center line (CL), an upper control limit
(UCL), and a lower control limit (LCL). The UCL and LCL
are horizontal lines situated above and below the CL. The
CL represents the process average when it is in a state of
statistical control. Any points above the UCL or below the
LCL are considered out of control. That is, if the process is
out of control in the upper or lower range, then an assignable
cause must be documented and removed.
However, we propose an important difference in how the
control chart is implemented in a classroom. While a typical
quality control chart allows an engineer to detect deviation
from the process mean—the CL—and to make adjustments
250
M. J. CERVETTI ET AL.
Performance Control Chart
Control Level
---------------------------------------------------- UCL
{
----------------------------------------------------
•Could be used as resource for other teachers
•Document successful strategy for sharing purposes (teamwork)
•Document successful strategy in teacher porƞolio
Center Line
------------------------------------------------------------------------------------------------------- LCL
UCL
(Standard)
The upper control limit (UCL) provides a basis for
knowing what is expected. The UCL sets the
standard or goal for the course.
3
---------------------------------------------------------------CL (Norm)
The lower control limit (LCL) indicates the point at
which some sort of correcƟve acƟon or remediaƟon
is needed.
2
Time
LCL
1
{
Downloaded by [Universitas Maritim Raja Ali Haji] at 22:02 11 January 2016
FIGURE 1 A typical quality control chart, with upper control limit (UCL),
center line (CL), and lower control limit (LCL) (color figure available
online).
to the process only when needed, the control chart presented
here supports change and what might be considered out of
control processes. This is an important distinction from the
traditional QCC and Deming’s basic argument that changes
should be implemented only when a process is out of control.
Moreover, the control chart must be adapted to each individual instructor and classroom based on the previous semester’s
learning outcomes. For these reasons, we refer to the control
chart here as a PCC. One of the overall goals of the PCC
is to allow the instructor to analyze the information gleaned
from the highs and lows of the graph and to make appropriate
changes in the curriculum as needed. By using the PCC, an
instructor can reduce two possible errors: (a) taking action
when none is needed and (b) not taking action when it is
needed (Savic, 2006). Although we recognize this approach
is in contrast to Deming’s philosophy, we also argue that
it is critical here because teaching is a service-based product whereas typical control charts are used for goods-based
manufacturers. The now well-accepted differences between
services and goods generally require different approaches to
assessments of service quality (Parasuraman, Zeithaml, &
Berry, 1985).
The PCC in Figure 2 illustrates the established norm for a
given course based on the achievement level of the previous
semester in that course. A statistical confidence interval will
be the mechanism utilized to establish a PCC that will determine the expected performance level based on the previous
semester’s data (the CL of the confidence interval) and, at
the same time, provide a higher level standard of learning
for the present year (the UCL). It is important to note that
the UCL is only an artificial goal; it will likely take more
than one semester to reach the current UCL. Further, the
UCL will change from semester to semester, depending on
the previous semester’s mean and standard deviation. However, if a class is at the 100% level (perfect performance),
then the PCC could be used to simply monitor the future
performance of that class. Finally, a PCC will track the class
performance variance, or deviation from the norm, in either
a positive direction (above the norm at point 3 in Figure 2)
or in a negative direction (below the norm at points 1 and 2
FIGURE 2
The center line (CL) establishes the norm based on
last year’s performance.
•Find assignable cause(s)
•UƟlize check list of resources for remediaƟon
•Consult lead teacher or team teachers
•Consult curriculum specialist
•Teacher discerns problem area(s) independently, makes
adjustments
•Document assignable cause(s) and remediaƟon in
teacher porƞolio
Example of a performance control chart.
in Figure 2). As noted, a PCC gives the teacher the mechanism necessary to discern specifically where remediation is
needed.
Point 3 in Figure 2 indicates that this teacher could be
used as a resource for other teachers. Successful strategies
should be documented and placed in the teacher portfolio for
sharing purposes with other teachers (teamwork) and/or evaluation by administrators. Points 1 and 2 in Figure 2 indicate
where modification might be needed in the course. Specifically, an assignable cause should be diagnosed. A checklist of
resources should be developed and utilized. After the teacher
finds an assignable cause(s) and documents the remediation
utilized, it should then be placed in the teacher portfolio for
future reference or evaluation by administrators. As noted,
the PCC is not trying to emulate Deming’s model of total quality management with regard to process control. The
PCC only utilizes the structural concept of the QCC to highlight the highs and lows of the course content. In fact, the
PCC is a tool that is utilized to actually aid the instructor in
moving the CL, which, as noted, is the opposite of Deming’s
approach. Action is encouraged at points 1 and 2 in Figure
2. It is important to emphasize that a PCC is not a searchand-destroy tool, but rather one to search and remediate. As
noted, the PCC helps the teacher discern where to take action
when action is needed.
For example, a typical business statistics instructor could,
using Gardiner et al.’s (2010) model discussed previously,
develop a set of course-level learning objectives based on the
learning goals relevant to that course. These objectives might
be that students could demonstrate the required proficiency
in various quantitative tools such as regression, forecasting, simulation modeling, linear programming, and queuing.
These course objectives would be evaluated through courseembedded measures that relate to specific course assignments
(Association to Advance Collegiate Schools of Business International, 2007), which—in the case of the model we are
proposing—could also mean class tests. Figure 3 represents
Downloaded by [Universitas Maritim Raja Ali Haji] at 22:02 11 January 2016
PERFORMANCE CONTROL CHARTS
FIGURE 3
251
A business statistics class performance control chart (color figure available online).
an actual PCC. This PCC was developed by calculating the
mean for all of the tests that were given during the previous
semester. The UCL and LCL were calculated by adding and
subtracting one standard deviation to the mean. As research
is conducted and data are collected, these UCL and LCL can
and should be adjusted and adapted to meet the instructor’s
unique needs for the class. After the UCL (mean + standard deviation), CL (overall class mean), and LCL (mean –
standard deviation) is established and drawn, then means for
each individual test are plotted on the chart. This PCC shows
that the class, as a whole, performed well above the overall
mean on the regression component of the class objectives.
At the same time, the PCC also indicates that the simulation
component for the class was well below the overall class
mean.
This low point on the PCC, the point where the simulation assessment approaches the LCL, is also the point in the
model where the faculty member could utilize a checklist
of resources for remediation, document the remediation utilized, and place documentation in the teacher portfolio for
future reference or evaluation by administrators. It should
be remembered that students also see the PCC and share in
the remediation process; it is important for the psychosocial
climate of the classroom that students share in the responsibility of the improvement process. As Gardiner et al. (2010)
noted,
any model of quality in higher education needs to produce
evidence-based improvement in learning, clearly identify
needed quality improvements in either the processes or the
outcomes, and involve the faculty in the evaluation of learning and the implementations of the improvements. (p. 140)
The PCC model does just that while also involving students in the remediation process, something recommended
by several academicians (e.g., Stivers & Phillips, 2009)
Figure 4 illustrates the flow of the course as the teacher
utilizes the formative information from the PCC to make improvements in the course during the school year. Essentially,
the teacher assesses the performance of students via the PCC,
assigns a diagnosis, and subsequently implements a course
of action with the aid of a list of available resources and
strategies. The remediation could be as simple as spending
more time on the topic or as drastic as restructuring a part of
the course. In such cases, some strategies might have to be
implemented the following school year. The expectation is
that as the teacher works consistently on the weak points of
the course or curriculum, over time the norm or CL will increase. Each PCC should be analyzed in a longitudinal format
to discern if the same weak points occur through the years
for a particular teacher and to see if significant improvement
is made on the CL, or class average, of the PCC from year to
year.
252
M. J. CERVETTI ET AL.
Management of Student Performance
Teacher manages the curriculum and improves the curriculum in a conƟnuous cycle.
Established Curriculum
===================
Teacher Has Control and
Ownership
Manages
Performance
Quality
Assess
Performance
Improvement
ImplementaƟon
Improve
Performance
Quality
Select
Improvement
Area(s) Via
PCC
REFERENCES
Downloaded by [Universitas Maritim Raja Ali Haji] at 22:02 11 January 2016
----------------------------Find
Assignable
Causes
FIGURE 4
coming an increasingly critical concern in the educational
environment.
In short, the PCC is a formative evaluation process that can
be integrated into the classroom during an actual semester and
utilized for remediation when and where the instructor deems
necessary. As a result, the PCC has considerable potential
for helping teachers improve their classroom skills, which
should translate into enhanced learning outcomes that can
benefit the student and the ultimate employer.
UƟlize Available
Resources For
Improvement
Strategy
Flowchart of course remediation process.
The PCC should be displayed clearly on the wall of the
classroom where it is easily visible by students and teachers.
The teacher should encourage the students to take ownership
in the class progress and point out goals for each test. This
should encourage teamwork for the class and collaboration
with the teacher, and subsequently, a more favorable classroom environment. As pointed out by Dorman (2002), “without a consideration of the crucial role of the psychosocial
environment of classrooms, educational productivity cannot
be optimized” (p. 1).
Although the PCC may have some value for the Association to Advance Collegiate Schools of Business International accreditation process, its primary role is improvement
in the classroom. For example, it can be used to encourage self-evaluation of teacher performance by providing a
mechanism to chart progress, pinpoint problem areas in the
curriculum as they occur, adapt to unique school or classroom environment, provide immediate feedback as concerns
student progress toward reaching established goals, focus
on prevention of student failure at all stages of instruction,
provide a mechanism for continuous improvement of curriculum, provide a data base for assessing the results of any
remedial action that was taken, replace bureaucratic control
and management with self-regulation and self-motivation,
and promote teamwork among teachers of the same course
by sharing results in a meaningful way. Moreover, the PCC
provides a means of accountability, an issue that is be-
Association to Advance Collegiate Schools of Business International.
(2007). AACSB assurance of learning standards: An interpretation.
Tampa, FL: AACSB International Accreditation Coordinating Committee
and AACSB International Accreditation Quality Committee. Retrieved
from http://www/aacsb.edu/publications/papers/accreditation/assuranceof-learning.pdf
Benjamin, S. (2009). Keeping score: Use rubrics to advance continuous
improvement in schools. Quality Progress, 42, 40–44.
Dorman, J. (2002). Classroom environment research: Progress and possibilities. Queensland Journal of Educational Research, 18, 112–140.
Drost, D., Hanson, L., Molstad, C., Peake, L., & Newman, E. (2008) Outcome assessment of a management program using a portfolio approach:
Lessons learned. American Journal of Business Education, 1, 105–114.
Gardiner, L., Corbitt, G., & Adams, S. (2010). Program assessment: Getting to a practical how-to model. Journal of Education for Business, 85,
139–144.
Horsburgh, D. (2010, March 20). Measuring the performance of teachers
and students. Denver Post. Retrieved from http://blogs.denverpost.com/
eletters/ 2010/03/20/measuring-performance-of-teachers-and-students2-letters/8522/
Melvin, C. (1993). Application of control charts to an educational system.
Performance Improvement Quarterly, 6(3), 74–85.
Parasuraman, A., Zeithaml, V., & Berry, L (1985). A conceptual model
of service quality and its implications for future research. Journal of
Marketing, 49(4), 41–50.
Savic, M. (2006). p-Charts in the quality control of the grading process in
the high education. Panoeconomicus, 53, 335–347.
Schazmann, B., Regan, F., Ross, M., Dermot, D., & Pauli, B. (2009). Introducing quality control in the chemistry teaching laboratory using control
charts. Journal of Chemistry Education, 86, 1085–1086.
SkyMark. (2010). Walter Shewhart: The grandfather of total quality management. Retrieved from http://www.skymark.com/resources/leaders/
shewart.asp
Spruell, J., Hawkins, A., & Vicknair, D. (2009). Managing learning experiences in an AACSB environment: Beyond the classroom. American
Journal of Business Education, 2, 75–77.
Stivers, B., & Phillips, J. (2009). Assessment of student learning: A fast
track experience. Journal of Education for Business, 85, 258–262.
Weldy, T., & Turnipseed, D. (2010). Assessing and improving learning in
business schools: Direct and indirect measures of learning. Journal of
Education for Business, 85, 268–273.