Pengelolaan Perikanan Pukat Cincin Berbasis Ekologiekonomi (Studi Kasus: Perikanan Di Perairan Selat Sunda).

BUILDING A MODEL TO PREDICT
SCHOOL ACCREDITATION RANK USING
BOOSTED CLASSIFICATION TREE

YESI NINDAHAYATI

GRADUATE SCHOOL
BOGOR AGRICULTURAL UNIVERSITY
BOGOR
2015

STATEMENT OF THE THESIS AND SOURCES OF
INFORMATION AND COPYRIGHT*
I hereby certify that thesis entitled Building a Model to Predict School
Accreditation Rank Using Boosted Classification Tree is an authentic work of
mine under the supervision of advisory board and has not been submitted in any
form to any college. Resources originating or quoted from published and
unpublished works of the other authors have been mentioned in the text and listed
in the bibliography at the end of this thesis.
I hereby assign copyright of my paper to the Bogor Agricultural University.
Bogor, August 2015

Yesi Nindahayati
G152130291

RINGKASAN
YESI NINDAHAYATI. Membangun Model untuk Memprediksi Peringkat
Akreditasi Sekolah Menggunakan Boosted Classification Tree. Dibimbing oleh
HARI WIJAYANTO dan BAGUS SARTONO.
Pemerintah Indonesia telah berkomitmen untuk meningkatkan mutu
pendidikan seperti yang tercantum pada undang-undang sistem pendidikan
nasional (UU No. 20/2003). Peringkat akreditasi sekolah yang dikeluarkan oleh
Badan Akreditasi Nasional untuk Sekolah/Madrasah (BAN S/M) merupakan
cerminan mutu pendidikan yang diselenggarakan oleh sekolah. Namun jumlah
sekolah yang terakreditasi belum sesuai dengan target sehingga pemerintah
mengalami kesulitan dalam perencanaan program dan anggaran. Oleh karena itu,
memprediksi peringkat akreditasi sekolah memiliki peranan yang penting sebagai
referensi untuk meningkatkan mutu pendidikan.
Tujuan dari penelitian ini adalah untuk memprediksi peringkat akreditasi
sekolah tingkat SMP di Provinsi Banten menggunakan boosted classification tree
dengan memanfaatkan database data pokok pendidikan (Dapodik). Model terbaik
akan digunakan untuk memprediksi peringkat akreditasi sekolah yang belum

terakreditasi. Metode yang digunakan pada penelitian ini adalah metode CART
sebagai metode klasifikasi tunggal dengan menggunakan kriteria Gini untuk
pohon klasifikasi nominal dan generalized Gini impurity untuk pohon klasifikasi
ordinal. Boosting merupakan salah satu metode ensambel yang sering digunakan
untuk meningkatkan akurasi dari prediksi. Pada penelitian ini, boosted
classification tree akan menggunakan metode CART sebagai metode dasarnya.
Hasil penelitian ini menunjukkan bahwa boosting telah berhasil
meningkatkan tingkat akurasi dari prediksi dan mengurangi masalah
ketidakstabilan prediksi. Persentase orang tua yang bekerja sebagai
petani/nelayan/peternak (X22), luas area yang dimiliki sekolah (X29), persentase
orang tua yang berpenghasilan  1 juta (X20), dan persentase orang tua yang
bekerja sebagai PNS/tentara/polisi/pegawai swasta (X23) diduga sebagai empat
prediktor terpenting dalam memprediksi peringkat akreditasi sekolah. Meskipun
boosted classification tree telah berhasil meningkatkan performa prediksi, akurasi
prediksi sangat tergantung pada kualitas Dapodik. Sehingga peningkatan mutu
Dapodik sangat diperlukan untuk memperoleh akurasi prediksi yang lebih baik.
Kata kunci: boosting, pohon klasifikasi, peringkat akreditasi sekolah

SUMMARY
YESI NINDAHAYATI. Building a Model to Predict School Accreditation Rank

Using Boosted Classification Tree. Supervised by HARI WIJAYANTO and
BAGUS SARTONO.
Indonesian government has committed to improve the education quality as
stated in act on national education system (Act No. 20/2003). School accreditation
rank which is issued by National Accreditation Board for School/Madrasah (BAN
S/M) is depiction of education quality provided by school. However the number
of accredited school has not met the target yet so that the government faces
difficulty in the planning of budget and actions. The prediction of school
classification based on accreditation rank to the schools that haven’t been
accredited, therefore, has important role as reference to improve quality of
education.
The objective of this research is to predict school accreditation rank of
junior secondary school in Banten Province using boosted classification tree
compared to single tree utilizing the education database (Dapodik). The best
model will be used to predict school accreditation rank of schools which haven’t
been accredited. CART method is used as single classifier with splitting method
of Gini criterion for nominal classification tree and generalized Gini impurity for
ordinal classification tree. Boosting is one of the widely used ensemble for
classification with a goal of improving the accuracy of classifier. Boosted
classification tree will use CART as a classifier.

It is showed that the accuracy of prediction is improved by use of boosting
method and boosting method also reduces instability issues. The percentage of
parents working as farmer/fisherman/cattleman (X22), lot size owned by school
(X29), percentage of parents have income  1 million (X20), and percentage of
parents working as government officer/army/police/ private employee (X23) are
estimated to be the four most important predictors. Though boosted classification
tree has succeeded to improve the predictive performance, the accuracy of
prediction depends on the quality of Dapodik. Thus, improvement of quality of
Dapodik is needed to get better accuracy of prediction.
Keywords: boosting, classification tree, school accreditation rank

© Copyright Bogor Agricultural University, 2015
Copyright is protected by law
It is prohibited to cite part or all of this thesis without referring or mentioning the
source. Citation is only permitted for educational purposes, research, scientific
paper, report writing, criticism, or reviewing an issue; and citations are not
detrimental to the interests of Bogor Agricultural University
It is prohibited to republish and reproduce part or all of this thesis without
permission of Bogor Agricultural University

BUILDING A MODEL TO PREDICT
SCHOOL ACCREDITATION RANK USING
BOOSTED CLASSIFICATION TREE

YESI NINDAHAYATI

Thesis
submitted to Graduate School of IPB in partial fulfillment
of the requirement for the degree of Master of Science
in Applied Statistics

GRADUATE SCHOOL
BOGOR AGRICULTURAL UNIVERSITY
BOGOR
2015

The Examiner : Dr Ir Indahwati, MSi

Thesis Title : Building a Model to Predict School Accreditation Rank Using
Boosted Classification Tree

Name
: Yesi Nindahayati
Student ID : G152130291

Approved by
Advisory Board

Dr Ir Hari Wijayanto, MSi
Advisor

Dr Bagus Sartono, MSi
Co-Advisor

Endorsed by

Program Coordinator of
Applied Statistics

Dean of Graduate School

Dr Ir Indahwati, MSi

Dr Ir Dahrul Syach, MScAgr

Date of Examination:
7th July 2015

Date of Graduation:

PREFACE
I would like to express my highest grateful to Allah SWT for His Blessing
so that this thesis was completed. The topic of this research is about utilizing
educational database (Dapodik) to classify school accreditation rank using
boosted classification tree. The title of this research is building a model to predict
school accreditation rank using boosted classification tree.
I would like to sincerely thank to my advisors Dr Ir Hari Wijayanto, MSi
and Dr Bagus Sartono, MSi for their guidance, advices and encouragement. I
would like to express my gratitude to the examiner Dr Ir Indahwati, MSi for
giving valuable advices and inputs to my thesis. I also wish to thank and give
appreciation to STT students for their friendliness and best years in study time. I

thank also to all STT lecturers for their sharing of valuable knowledge and
experience as well as STT staffs for their help and cooperation. Special thanks to
my husband, my lovely children, my parents and my parents-in-law who are
always supporting me. Thanks to the others that cannot be written here.
Hopefully, this thesis beneficial to all whom read it.

Bogor, August 2015
Yesi Nindahayati

TABLE OF CONTENTS
LIST OF TABLES

vi

LIST OF FIGURES

vi

LIST OF APPENDICES

vi

1 INTRODUCTION
Background
Research Objective

1
1
2

2 LITERATURE REVIEW
School Accreditation Rank
Classification Tree
Boosting
Performance Measures

2
2
3
4

7

3 RESEARCH METHODOLOGY
Data
Methods

7
7
8

4 RESULT AND DISCUSSION
Overview of Predictors
Comparison of Three Methods
Boosted Classification Tree

10
10
12
14

5 CONCLUSION AND RECOMMENDATION
Conclusion
Recommendation

17
17
18

REFERENCES

19

APPENDICES

21

AUTHOR BIOGRAPHY

25

LIST OF TABLES
1 Illustration of new data to be predicted by boosted classification tree ........ 6
2 Illustration of prediction calculation ............................................................ 6
3 Distribution of junior secondary school in Banten Province by school
accreditation rank ......................................................................................... 8
4 Average values of numeric predictors ........................................................ 11
5 Percentages of nominal predictors ............................................................. 12
6 Mean and standard deviation of misclassification rate, Kendall’s 
association, and squared loss by methods.a ................................................ 13
7 Distribution of accreditation rank of junior secondary schools,
including the prediction using boosted classification tree ......................... 14

LIST OF FIGURES
1
2
3
4

Illustration of boosted classification tree model ........................................... 6
Performance of boosted trees by number of trees/boosting ....................... 13
Relative importance of the predictors......................................................... 15
Partial dependence plot of school accreditation rank on four most
important variables. The red ticks at the base of the plots are quartiles
of the predictors. ......................................................................................... 15
5 Partial dependence plot of school accreditation rank on X30 and X18.
The red ticks at the base of the plots are quartiles of the predictors. ......... 16
6 Partial dependence plot of school accreditation rank on X24 and X17.
The red ticks at the base of the plots are quartiles of the predictors. ......... 16
7 Partial dependence plot of school accreditation rank on X25 and X5.
The red ticks at the base of the plots are quartiles of the predictors. ......... 17

LIST OF APPENDICES
1 List of predictors ........................................................................................ 21
2 Flow chart of the research .......................................................................... 22
3 Performance of boosted classification tree per number of trees/
boosting ...................................................................................................... 23
4 Friedman’s test statistic and corresponding asymptotic p-value ................ 24
5 Statistic and p-value of Wilcoxon signed rank test from the pairwise
comparison of methods .............................................................................. 24
6 Boxplot of misclassification rate, Kendall’s  association, and
squared loss by methods ............................................................................. 24

1

1 INTRODUCTION
Background
Education has a key role to make a better life. The Education for All (EFA)
is a global movement led by UNESCO, aiming to provide good basic education
for all children, youths and adults. Indonesian government has committed to
improve the education quality as stated in act on national education system (Act
No. 20/2003), which is reflected by three pillars of education: access, quality and
governance. As a part of quality assurance, government of Indonesia established
National Accreditation Board for School/Madrasah, namely BAN S/M, to
independently evaluate the school quality based on national education standards.
A school accreditation rank which is issued by BAN S/M is depiction of
education quality provided by the school.
Government established a nine year compulsory education program to meet
the mandate of the 1945 constitution Article 31, paragraph 1 which states that
every citizen has the right to education. Every citizen of Indonesia must go to
school for minimum nine years from first grade in primary school until ninth
grade in junior secondary school. Percentage of accredited primary school was
84.4%, while percentage of accredited junior secondary school was 70%
(Ministry of Education and Culture 2014). It is shown that percentage of
accredited junior secondary school is rather lower so that the government faces
difficulty in the planning of budget and actions. Study of school/madrasah
accreditation system by Ministry of National Education (2011) found some
constrains in the implementation of accreditation such as the amount of school to
be accredited was limited which depends on the national budget, many schools
are scattered in various region in Indonesia and difficult to reach. The prediction
of school classification based on accreditation rank to the schools which haven’t
been accredited, therefore, has important role as reference to improve quality of
education.
Ministry of National Education in 2010 has started to develop national
education database which is known as Data Pokok Pendidikan (Dapodik). In this
research, Dapodik will be utilized to predict school accreditation rank of schools
and limited to include junior secondary school in Banten Province only.
Classification Tree is one of the well-known class prediction methods.
Classification Tree is nonparametric computationally intensive method that has
greatly increased in popularity during the past decades. Classification Tree can be
applied to data sets having both a large number of cases and a large number of
variables, and it is extremely resistant to outliers (Sutton 2005).
In recent years the introduction of aggregation methods led to many new
techniques within the field of prediction and classification. Boosting is one of the
widely used ensemble for classification with a goal of improving the accuracy of
classifier. The principle is to use a basic discrimination method not only once but
for different versions of the data sets, boosting uses weights that depend on the
performance in the last sample (Tutz and Hechenbichler 2005). This research will
use Classification Tree as classifier method, and it will be demonstrated whether
boosting will improve the accuracy of prediction. Comparisons between the

2
methods are based on misclassification rates as well as criteria that take ordinality
into account like squared loss and Kendall’s  association measure.
Research Objective
The objective of this research is to predict school accreditation rank of
junior secondary school in Banten Province using boosted classification tree
compared to single tree utilizing the education database (Dapodik). The best
model will be used to predict school accreditation rank of schools which haven’t
been accredited.

2 LITERATURE REVIEW
School Accreditation Rank
Component of school accreditation assessment refers to the eight national
education standards (Act No. 20/2003) which consist of the standard of the
content, process, graduate outcomes, educational personnel, facilities and
equipment, management, funding and educational assessment. According to the
Minister of National Education Regulation No. 12 of 2009, the school/madrasah is
accredited if the school/madrasah meets all the following criteria (based on 100
points score)
1. Obtain the final score of accreditation for minimum of 56
2. It is not allowed more than two components of accreditation are less than
56
3. There is no accreditation component score less than 40
Accreditation rank is given to the school/madrasah which meet the above
criteria. Rank A is given if the school/madrasah obtain final score of accreditation
more than 85, rank B if the school/madrasah obtain final score of accreditation by
71 to 85, and rank C if the school obtain final score of accreditation of 56 to 70.
Based on BAN S/M (2014), school accreditation rank can be categorized into four
classes namely A (excellent), B (good), C (satisfactory) and Unaccredited (failed).
New England Association of Schools and Colleges (2006) examined the
impact of accreditation on the quality of education at accredited schools. This
research demonstrated that accreditation had a profound and enduring impact on
quality of education. Adisantoso et al. (1997) examined the relation between pure
national examination score namely Nilai Ebtanas Murni (NEM) and school
accreditation rank in private school in Jawa Barat Province. It resulted that both
logistic regression model and discriminant function which used NEM as predictor
can’t be used as alternative of school accreditation status. While Mongi (2014)
used school accreditation rank and national examination score to cluster and map
cities/regencies in West Java Province. William (2008) found that principal
characteristic such as principal highest degree was one of the significant
predictors of school accreditation status in Virginia.

3
Classification Tree
There are so many methods available for tree-structured classification and
regression. Some well-known methods are CART (Classification And Regression
Trees), CHAID (Chi-squared Automatic Interaction Detector), QUEST (Quick
Unbiased Efficient Statistical Tree), Quinlan’s C4.5 and C5 (which are descendant
of ID3). This research will use CART method. Consider a data set
{( , , , … , � ); = , … , } that contains the values of a response Y with K
classes and characterized by a
-dimensional vector of predictors
=
( , , … , � ) . When deriving a Classification Tree, all observations start
together in the root node, t. Then, for predictors 1, 2,…, , the best binary split is
determined, where splits resulting in increasingly more homogeneous nodes with
respect to class are desired (Archer and Mas 2009).
For node t, the optimal split divides the observations to the left and right
descendent/child nodes, and � , respectively, and the proportion of cases in
each of the � classes within these nodes are called the node proportions, that is,
| +
| + ⋯+
| for = , , … , � such that
�| = . For
nominal response classification, the within-node impurity measure most
commonly used is the Gini criterion (Breiman et al. 1984), defined as
|

=∑∑
≠

|

The node impurity is largest when all classes are equally mixed together and
smallest (zero) when the node contains only one class.
One of the impurity function that can be used for ordinal response
prediction is the generalized Gini impurity (Breiman et al. 1984), defined as
=∑∑
≠

|

|

|

which factors in
| is the cost of misclassifying an observation belonging to
class as class . In classifying an ordinal response, it may assumed that for each
combination of true class and predicted class, there is a known loss or cost
|
giving the negative utility of the consequences of predicting when the true class
is (Archer and Mas 2009). Suppose that a set of increasing scores { < <
⋯ < } is assigned to the ordered categories of the response Y. The quadratic
| =
misclassification cost is expressed as
−
.
The ordinal impurity is well-suited for partitioning the ordinal response
categories when monotonic associations are presents. In the case that predictors
are not monotonically associated with the ordinal response, the Gini impurity
(nominal classification tree) have performed well because it would have been
capable in detecting associations between covariates that are not monotonic with
the ordinal class (Archer and Mas 2009). This research includes some predictors
that are not monotonically associated with the ordinal response. Thus, the nominal
Gini impurity function is used.
For selecting the right-sized tree, first step is to grow a very large tree,
splitting subsets in the current partition of even if a split does not lead to an
appreciable decrease in impurity. Then a sequence of smaller trees can be created
by pruning the large tree, where in the pruning process, splits that were made are
canceled and a tree having a fewer number of nodes is produced. The accuracies
of the members of this sequence of subtrees are then compared using good

4
estimates of their misclassification rates (either based on a test sample or obtained
by cross-validation), and the best performing tree in the sequence is chosen as the
classifier (Sutton 2005).
The established method is cost-complexity pruning, first introduced by
Breiman et al. (1984). The cost complexity measure � � is defined as
� + �|�̃|
� � =
Here, |�̃| is a measure of tree complexity, the number of terminal nodes in T. R(T)
is related to misclassification cost and  is the complexity parameter. � � is
formed by adding to the misclassification cost of the tree a cost penalty for
complexity. The pruning process produces a finite sequence of subtrees
� , � , … , � with progressively fewer terminal nodes. If the resubstitution
estimate R(Th) is used as criterion, the largest tree T1 would be selected (Breiman
et al. 1984). The V-fold cross-validation is performed to get an “honest” estimate
̂ �ℎ of the misclassification cost, the optimum-sized tree can be reached by
selecting the tree Th0 such that
�
�ℎ = min � �ℎ
ℎ

Boosting

Boosting is a method of combining classifiers, which are iteratively created
from weighted versions of the learning sample, with the weights adaptively
adjusted at each step to give increased weight to the cases which were
misclassified on the previous step (Sutton 2005). The final predictions are
obtained by weighting the results of the iteratively produced predictors. The
motivation for boosting was a procedure that combines the output of many weak
classifiers to produce a powerful committee (Hastie et al. 2009). One of the best
known boosting algorithm is AdaBoost (Freund and Schapire 1996), but it can be
only applied to binary classification problem. AdaBoost.M1 is the extension of
AdaBoost for multiclass classification problem (Mukherjee and Schapire 2013).
} where takes
,…, � , ,…, � ,
Given a training set TS= { � ,
values in 1, 2, …, K. Suppose b is number of iteration/boosting or number of trees,
and let � �� denote a classifier with input of predictors � on the b-th
iteration/boosting. This research will use CART method as a classifier. The
weight � is assigned to each observation � and is initially set to ⁄ . This
value will be updated after each step. A basic classifier � �� is built on this new
training set ( � ) and is applied to every training sample. The error rate of this
classifier is represented by � and is calculated as
�

=∑
=

�

�

�

�� ≠

where I(.) is the indicator function which outputs 1 if the inner expression is true
and 0 if otherwise.
From the error of the classifier in the b-th iteration, the �� is calculated
− �
. The new
and used for weight updating. Breiman (1998) used �� = ⁄ ln

weight for the (b+1)-th iteration will be (Alfaro et al. 2013)
�+

=

�

exp �� �

�

� ≠

�

�

5
And then the calculated weights are normalized so that the sum of weights is one.
Consequently, the weights of the wrongly classified observations are increased,
and the weights of the rightly classified are decreased, forcing the classifier built
in the next iteration to focus on the difficult cases. Alpha (�� ) can be interpreted
as a learning rate calculated as a function of the error made in each step. Moreover,
this �� is also used in the final decision rule giving more importance to the
individual classifiers that made a lower error. This process is repeated every step
for b=1, …, B. Finally, the ensemble classifier calculates, for each class, the
weighted sum of its votes. Therefore, the class with the highest vote is assigned.
AdaBoost.M1 algorithm can be described briefly as follows (Alfaro et al.
2013)
= ⁄ , = , ,…,
1. Start with �
2. Repeat for = , , … ,
on � .
a. Fit the classifier � � = { , , … , �} using weights �
�
is
classifier
with
input
of
predictors
�
on
the
b-th
boosting
�
and � is training set on b-th boosting.
− �
and �� = ⁄ ln
b.Compute: � = ∑ = � � � � ≠
c. Update the weight �+
normalize them
3. Output of the final classifier

=

�

exp �� �

�

� = arg max ∑�= �� �
∈�

≠

�

�

and

=

The original Freund and Schapire algorithm exits from the construction loop when
� becomes equal to or greater than 0.5, and if
� equals to zero makes the
subsequent step undefined (Breiman 1996).
Single tree methods are highly interpretable. The entire model can be
completely represented by a simple two-dimensional graphic (binary tree) that is
easily visualized. In the case of boosted classification tree, variable importance is
used as a measure of relevance for each predictor variable. For K-class
, = , , … , � are induced, each
classification, � separate models
consisting of a sum of trees (Hastie et al. 2009). The squared importance measure
is defined as
� =

∑� �

�=

�

Here � is the relevance of � in separating the class k observations from the other
class. B is number of boosting/trees, while � � � is the squared relative
importance of variable � in separating the class k observations from the other
class in the b-th tree model and defined as
� �

�

= ∑ ̂� . {
�=

where J is the number of internal nodes (non-leaves) and ̂ � is the improvement in
training misclassification error from making the t-th split. The overall relevance of
� is obtained by averaging over all the classes
� =

�

∑�
=

Recently, several studies on boosted classification tree have used AdaBoost
algorithms. Alfaro et al. (2008) found that AdaBoost with classification tree

6
decreased the generalization error by about 30% with respect to error produced
with neural networks (NNs) on bankruptcy prediction. Novakovic and Veljovic
(2014) demonstrated AdaBoost algorithm have been found to be accurate and
computationally feasible across five medical datasets using five different decision
tree algorithms. Liu et al. (2013) showed that AdaBoost helped a lot to improve
the accuracy to classify retail outlets into different classes based on the sales
ability.
Illustration of calculating prediction score using boosted classification tree
will be described in this section. For example boosted classification tree is built
using Dapodik with School Accreditation Rank (Y) as response variable and 45
predictors as shown in Appendix 1. For simple illustration, maximum depth of
tree is defined as 2 and number of boosting is 3. The boosted classification tree
model for this case can be seen at Figure 1.

Figure 1 Illustration of boosted classification tree model
Suppose that we have new data to be predicted as shown in Table 1. The
ensemble classifier calculates, for each class, the weighted sum of its votes.
Therefore, the class with the highest vote is assigned. For the first case, Tree1 and
Tree3 predict rank B, while Tree2 predicts to have rank C. In this case, weighted
sum for rank C is weight of Tree2 (i.e. 0.121), weighted sum for rank B is
summation of weight of Tree1 and Tree3 (i.e. 0.298 + 0.170 = 0.468), and
weighted sum for rank A is 0 since none of the Tree which predicts to be rank A.
For this first case, the final prediction is rank B since it has the highest vote
(weighted sum). The brief calculation of prediction can be seen in Table 2.
Table 1 Illustration of new data to be predicted by boosted classification tree
Case
1
2
3

X12
3
3
9

X13
76
91
237

X21
87
88
05

X22
0
0
22

X23
58
59
0

X32
3
3
9

X41
0
1
0

Table 2 Illustration of prediction calculation
Case
1
2
3

Prediction of
Tree1 Tree2 Tree3
B
C
B
C
C
B
A
B
A

Vote (weighted sum) of
Rank C Rank B Rank A
0.121
0.468
0
0.419
0.170
0
0
0.121
0.468

Final
prediction
B
C
A

7
Performance Measures
The evaluation of the methods is based on several measures of accuracy. As
criterion for the accuracy of prediction, Misclassification Rate (MCR), whereas
every misclassification is considered equally costly, is calculated as
∑�

≠ ̂

=

Squared Loss is used to take into account ordinality that a larger distance
is a more severe error than a wrong classification into a neighbor class. Suppose
that a set of increasing scores { < < ⋯ < } is assigned to the ordered
categories of the response Y. Squared Loss is expressed as
∑
=

��

−

�̂�

In order to avoid the influence of the number chosen to represent the classes
on the performance assessment, it has been argued that one should only look at the
order relation between “true” and “predicted” class numbers (Cardoso and Sousa
2011). Kendall’s coefficient  has been advocated as a better measure for ordinal
variables because it is independent of the values used to represent classes. The 
coefficient can be computed as
�=

−
−

/

where c refers to concordant pairs and d for discordant pairs. The
classification result of a pair of samples is called concordant if the relative order
of their class values is the same in the classification compared to the true values.
If the relative order is reverse to the true values, the pair is called discordant. The
 coefficient attains its highest value, 1, when both sequences agree completely,
and -1 when two sequences totally disagree.

3 RESEARCH METHODOLOGY
Data
This research will utilize education database (Dapodik) per June 2014.
Dapodik consists of four entities: school, educational personnel (teacher, principal,
and administration officer), student, and facilities. The response variable is school
accreditation rank and 45 variables are generated as predictors to classify school
accreditation rank. The list of those predictors is shown in Appendix 1. The
predictors depicts as close as possible to the component of eight national
education standard. Aside from the component of eight national education
standard, this research also uses student’s background such as parent’s income
and occupation as predictors.
From total 1283 junior secondary schools in the Province of Banten, only
818 schools have completed the database correctly. Distribution of junior
secondary school based on accreditation rank can be seen in Table 3, the
distribution is excluding the schools haven’t completed the database correctly.

8
From 818 schools which have completed the database correctly, 667 schools are
accredited and we will use this data to build the model. The best model will be
used to predict school accreditation rank of 151 schools which haven’t been
accredited. As seen in Table 3, the data consists of three classes of school
accreditation rank namely A, B and C. In terms of quadratic misclassification cost,
squared loss, and Kendall’s , scores are assigned to the ordered categories of
school accreditation rank: rank C scores 1, rank B is 2, and rank A is 3.
This research uses several packages of R software. Package of rpart is
used to build classification tree, rpartScore is used for ordinal classification
tree and boosting is performed using adabag. R package sampling is used to
perform stratified sampling.
Table 3

Distribution of junior secondary school in Banten Province by school
accreditation rank
School accreditation rank
A
B
C
Unaccredited
Haven’t been accredited
Total

Junior Secondary School
n
%
214
26.2%
336
41.1%
117
14.3%
0
0.0%
151
18.5%
818
100.0%

Methods
In this research, the data is divided into training set and testing set using a
stratified random scheme. The training set contains 70% of the data and the
remaining data is going to testing set. A training set is used to grow and prune
classification trees as well as build boosted classification tree. These trees will be
used to predict the scores of the testing set. 100 different random splits into
learning and testing sets are used and give the mean over these splits. The best
model will be chosen based on three evaluation measures (i.e. misclassification
rate, squared loss, and Kendall’s  association). The brief steps are shown in flow
chart of the research in Appendix 2. The detail steps in this research are as follow
1. Data is divided into training set and testing set using stratified random scheme
a. The school accreditation rank is treated as strata (i.e. rank A, B, C).
b. The training set will be drawn from each strata whereas contains 70% of
the data
c. The remaining data will be going to testing set.
2. Training set will be used to grow and prune classification tree and ordinal
classification tree as well as build boosted classification tree
a) Algorithm to grow and prune classification tree
i.
Use training dataset to grow classification tree using Gini Criterion
splitting method (classification tree) with accreditation rank as
response variable and 45 predictors from Dapodik (see Appendix 1
for the list of predictors)

9
ii.

3.
4.
5.
6.

Prune back the tree to avoid over fitting the data with selecting a tree
size that minimizes the cross-validated error (10 fold cross validation)
b) Algorithm to grow and prune ordinal classification tree
i.
Use training dataset to grow classification tree using generalized Gini
impurity splitting method with quadratic misclassification cost
ii. Prune back the tree to avoid over fitting the data with selecting a tree
size that minimizes the cross-validated error (10 fold cross validation)
c) Algorithm to build boosted classification tree
i.
Use training dataset to build boosted classification tree
ii. Set number of boosting b=10,20,…,400
Trees from step 2 will be used to predict the class of the testing set
Prediction and real scores of the testing set will be used to calculate
misclassification rate, squared loss and Kendall’s  association.
Generate 100 different random splits into learning and testing sets in step 1
and apply to step 2 – 4.
The global hypothesis of equality of the three classification methods is tested
using Friedman’s nonparametric rank test in a randomized complete block
design treating each of the 100 training & testing sets as a block. This test is
applied to all evaluation measures (i.e. misclassification rate, squared loss and
Kendall’s  association).
Suppose that { } × where the entry
is the rank of
within block i.
Friedman’s test statistic is given by (Hollander and Wolfe 1973)
∑ = .̅ − ̅
=
∑= ∑ =
− ̅
−

when n or m is large, p-value is given by
− ≥
7. Since each classification tree method is applied to the same resamples, to
identify which classifiers contribute to the observed significant difference
between three classifiers, pairwise comparison are performed by applying the
Wilcoxon signed rank test. This test is applied to all evaluation measures (i.e.
misclassification rate, squared loss and Kendall’s  association).
Let
denote the rank of pairs from smallest absolute difference to largest
absolute difference | , − , | for i = 1, 2,…., Nr. Nr is the reduced sample
size, excluding pairs with | , − , | = 0. Wilcoxon’s test statistic is given by
(Hollander and Wolfe 1973)
+

��

� = ∑[I
=

(

,

−

,

)>

.

]

� + is the sum of the positive signed ranks. As Nr increases, the sampling
distribution of � + converges to a normal distribution. A z-score can be
calculated as

10

=

�+ −

�� �� +

� �+

4

, � �+ = √

�� �� +

4

�� +

8. The best method will be selected based on results on step 6 and 7
9. Use 667 data to build a model using the best method in step 8
10. The school accreditation rank of 151 schools will be predicted using model in
step 9

4 RESULT AND DISCUSSION
Overview of Predictors
Overview of predictors are presented in this section to illustrate the four
entities of database (i.e school, student, teacher, and facilities). Table 4 shows the
average values of numeric predictors, while percentages of nominal predictors are
shown in Table 5. Based on Table 4, it can be seen that schools with higher
accreditation rank tend to have more teachers (X1) as well as more high quality
teachers (X2, X4, X5). Schools with higher accreditation rank also tend to have
certified principal (X6). Percentage of schools have head of laboratory of science
(X9) is still low (i.e. less than 5%), however it shows that percentage of schools
have head of laboratory of science is larger in the higher accreditation rank group.
Percentage of well-educated librarian (X8) is also directionally larger in the higher
accreditation rank group.
In terms of students (Table 4), average number of students (X12) is
increasing with an increase of school accreditation rank. Thus, schools of higher
accreditation rank have more students. Average number of class groups (X11)
shows the positive pattern, the higher accreditation rank group tends to have larger
number of class groups. Ratio of students per teacher (X17) tends to larger in
higher accreditation rank. This positive pattern is also found on ratio of students
per class group (X18). The percentage of drop out students is very small (X15),
however it can be seen that the percentage of drop out students decreases with an
increase of school accreditation rank. The student backgrounds are also used as
predictors (Table 5), particularly on parents’ income and occupation (X20-X25).
Percentage of parents’ income indicatively shows that the higher accreditation
rank group has more parents with high income. In line with income, occupation
also depicts the same pattern. Percentage of parents working as
government/army/police/private employee (X23) tends to be larger in higher
accreditation rank group. In contrast, the percentage of parents working as
farmer/fisherman/cattlemen (X22) is higher in lower accreditation rank group. If
parents’ income and occupation are seen as student’s socioeconomic status (SES),
this means that higher accreditation rank group has more high-SES students.
It is as expected that higher accreditation rank group has more school
facilities since availability of school facilities is one of the component in assessing
school accreditation rank. Table 5 shows that higher percentages of availability of
teacher room, principal room, library, laboratory of science, and administration
room (X32 – X36) are found in higher accreditation rank group. It’s also found in
number of classroom (X31), number of hand washing facilities (X37), and number
of student toilet (X38), the average increases with an increase of accreditation

11
rank (Table 4). Based on school location (Table 5), percentage of school in remote
area (X40) is higher in lower accreditation rank group. While percentage of school
in border area (X41) is larger in higher accreditation rank group. On average,
schools with higher accreditation rank are older (X30). In the school accreditation
rank A group, interestingly percentage of private school (62.1%) is higher than
public school (37.9%).
From data exploration, we can briefly say that higher accreditation rank group
tends to have more teachers, more high quality teachers, has head of laboratory of
science, larger number of class groups, lower percentage of drop out students,
higher ratio of student per teacher, and better availability of school facilities. In
terms of student’s background, higher school accreditation rank group seems to
have more high-SES students. It could be related to school-based management
which allows parents to participate actively to improve school’s quality.
Table 4 Average values of numeric predictors
Variables
X1
X2
X3
X4
X5
X11
X12
X13
X14
X15
X16
X17
X18
X19
X20
X21
X22
X23
X24
X25
X29
X30
X31
X37
X38
X39

Total
19.79
14.94
69.91
7.25
30.53
10.82
369.39
0.01
1.85
0.87
0.00
19.39
31.05
1.03
41.41
12.67
16.62
18.16
19.15
19.00
4974.34
15.60
10.63
1.24
2.03
1.30

School Accreditation Rank
Haven't been
Rank C Rank B Rank A
accredited
13.74
20.24
27.79
9.63
16.18
21.28
64.91
77.01
72.15
3.47
7.27
13.36
22.38
33.86
43.48
6.66
11.46
16.37
211.55 394.77 581.88
0.00
0.01
0.03
3.09
2.35
0.35
1.63
0.80
0.07
0.00
0.00
0.00
16.65
20.50
22.48
30.20
32.02
33.32
1.03
1.04
1.01
62.27
46.51
16.98
3.24
7.52
25.81
33.74
16.14
2.12
6.68
14.62
32.17
16.30
18.93
21.86
21.56
23.95
11.07
3468.14 5675.03 5871.77
10.85
17.36
22.67
6.61
11.07
16.44
0.81
1.02
1.93
1.77
1.96
2.50
0.47
1.48
1.04

12.13
7.32
54.83
1.49
11.05
4.75
134.09
0.01
1.91
1.57
0.01
14.65
26.35
1.04
48.49
12.83
24.99
15.05
18.03
17.25
3310.42
5.32
4.56
1.06
1.70
1.94

12
Table 5 Percentages of nominal predictors
School Accreditation Rank
Variables
X6
X7
X8
X9
X10
X26 (Public School)
X27 (Foundation)
X28
X32
X33
X34
X35
X36
X40
X41
X42
X43
X44
X45

Total
40.10%
5.26%
2.69%
3.67%
3.18%
46.21%
52.32%
81.42%
90.83%
83.50%
78.97%
52.57%
78.36%
20.42%
19.93%
4.65%
2.08%
0.61%
2.57%

Rank C Rank B Rank A Haven't been
accredited
36.75% 41.37% 42.06% 37.09%
5.98% 5.36% 7.01%
1.99%
2.56% 2.68% 3.27%
1.99%
2.56% 3.27% 6.54%
1.32%
2.56% 2.38% 6.07%
1.32%
52.14% 51.49% 37.85% 41.72%
46.15% 47.32% 59.35% 58.28%
81.20% 83.93% 81.78% 75.50%
86.32% 93.75% 94.86% 82.12%
71.79% 91.37% 95.79% 57.62%
66.67% 86.31% 94.86% 49.67%
29.91% 57.74% 77.57% 23.18%
61.54% 86.61% 96.26% 47.68%
44.44% 13.99% 4.67% 38.41%
13.68% 17.86% 29.44% 15.89%
5.98% 2.98% 4.21%
7.95%
2.56% 1.19% 1.40%
4.64%
0.00% 0.60% 0.93%
0.66%
1.71% 1.49% 4.21%
3.31%

Comparison of Three Methods
Figure 2 illustrates the results of boosted classification tree per number of
trees using three evaluation measures (i.e. misclassification rate, Kendall’s 
association, and squared loss). The evaluation score is coming from prediction of
202 schools in testing dataset, while the model is built from training datasets
(n=465). The scores in Figure 2 are the average of 100 different random splits of
training and testing sets. The results suggest that the optimum number of
trees/boosting is 380 since it has the lowest squared loss (66.08) and the highest
Kendall’s  association (0.601). It also has low misclassification rate (0.310). The
changes of misclassification rate are getting slower after 100 trees. The detail
figures of performance of boosted trees can be seen in Appendix 3. This optimum
boosted classification tree model will be compared to single classifiers (i.e.
classification tree and ordinal classification tree).

13

Figure 2 Performance of boosted trees by number of trees/boosting
The mean scores and standard deviations of each evaluation measure are
reported in Table 6 under three classification tree methods. All evaluation
measures lead to same conclusion that boosting method has significantly
improved the predictive performance. Mean scores of boosted classification tree
are the lowest on squared loss and misclassification rate. In terms of Kendall’s 
association, mean score of boosted classification tree is also the highest compared
to classification tree and ordinal classification tree. The evidences are
strengthened by results of Friedman’s test and Wilcoxon signed test (Appendix 45). Boosted classification tree significantly out-performed over the other two
methods (=0.05). Larger standard deviation of evaluation scores may indicate
instability issues on single classifiers. In this case, boosting method can improve
the predictive performance and reduce the instability issues.
Table 6

Mean and standard deviation of misclassification rate, Kendall’s 
association, and squared loss by methods.a

Methods
Classification Tree
Ordinal Classification
Tree
Boosted Classification
Tree (B=380)
a

Performance of Prediction
Misclassification Rate Kendall's  Squared Loss
0.372
0.496
83.18
(0.035)
(0.053)
(9.26)
0.363
0.517
78.61
(0.032)
(0.048)
(7.81)
0.310
0.601
66.08
(0.025)
(0.038)
(5.99)

figures in the bracket are standard deviation values

In terms of single-tree classifiers, ordinal classification tree is able to
perform better than classification tree. Ordinal classification tree has lower
misclassification rate and squared loss than classification tree. Ordinal
classification tree also has reached higher Kendall’s  association than
classification tree. Higher predictive performance of ordinal classification tree
than classification tree is supported by results of Friedman’s test and Wilcoxon

14
signed test (Appendix 4-5). Ordinal classification tree performs significantly
better than classification tree at =0.05 level. Ordinal splitting method with
applying quadratic misclassification cost is able to improve the predictive
performance of classification tree. However, the predictive performance of ordinal
classification tree as single classifier is still poor compared to boosting method.
Boosted Classification Tree
As the boosted classification tree has the highest predictive performance,
the school accreditation rank model is built using this method. Accreditation rank
of 151 schools, which haven’t been accredited, are predicted by this model. Table
7 reports the distribution of junior secondary schools by accreditation rank
including the prediction. Majority of the schools are predicted as Rank C or B by
this model. Almost half of the schools is predicted as Rank C (47%) and the
model predicts 43.7% of the schools as Rank B. Only few of the schools are
predicted as Rank A. At total level, half of the junior secondary schools in Banten
Province have rank B. While percentages of Rank A and Rank B are quite close.
Table 7

Distribution of accreditation rank of junior secondary schools, including
the prediction using boosted classification tree

Accreditation
rank
Rank C
Rank B
Rank A
Total

Accredited
schools
n
%
117 17.5%
336 50.4%
214 32.1%
667 100.0%

Prediction of schools which
haven’t been accredited
n
%
71
47.0%
66
43.7%
14
9.3%
151 100.0%

Total
n
188
402
228
818

%
23.0%
49.1%
27.9%
100.0%

Figure 3 illustrate the relative importance for each predictors, it shows the
contribution of each predictor variable in predicting the response. Clearly some
predictors are more important than others in classifying school accreditation rank.
The percentage of parents working as farmer/fisherman/cattleman (X22), lot size
owned by school (X29), percentage of parents have income  1 million (X20), and
percentage of parents working as government officer/army/police/private
employee (X23) are estimated to be the four most relevant predictors. Student and
teacher components that are estimated as top ten most important variable are ratio
student per class group (X18), ratio student per teacher (X17) and percentage of
certified teacher (X5). Number of class rooms (X31), has laboratory of science
(X35), and ratio class group per class room (X19) are estimated to have higher
relative importance compared to other school facilities (X33-X34 and X36-X38).
At the other end of the bar chart, percentage of students repeating a given grade
(X16), school located in natural disaster area (X43) and school located in social
disaster area (X44) have virtually no relevance. However it doesn’t mean that
students repeating a given grade are not important in assessing school
accreditation rank, the model uses the number of students repeating a given grade
(X13) instead of the percentage (X16). Schools located in natural and social
disaster area (X43-X44) are quite rare in Banten Province, so these predictors
don’t explain the school accreditation rank well.

15
8
7
Relative Importance

6
5
4
3
2
1

X22
X29
X20
X23
X30
X18
X24
X17
X25
X5
X3
X21
X12
X1
X31
X35
X2
X19
X4
X14
X37
X11
X38
X27
X41
X40
X15
X6
X36
X28
X9
X42
X34
X39
X32
X26
X33
X7
X10
X45
X8
X13
X16
X43
X44

0

Figure 3 Relative importance of the predictors

Figure 4 Partial dependence plot of school accreditation rank on four most
important variables. The red ticks at the base of the plots are quartiles of
the predictors.
Figure 4 shows the partial dependence of school accreditation rank on four
most important variables. In terms of X22, the probability of Rank C is increasing
with increasing percentage of parents working as farmer/cattleman/fisherman. On
the contrary, the probability of Rank A is decreasing with increasing of percentage
of parents working as farmer/cattleman/fisherman. It can be concluded that the
percentage of parents working as farmer/cattleman/fisherman is estimated to be
negatively associated with school accreditation rank. Negative association is also
found on percentage of parents have income  1 million (X20). Partial

16
dependence plot of school accreditation rank on X29 shows that probability of
Rank C decreases with an increase of lot size owned by school. This suggests that
lot size owned by school (X29) differentiates Rank C to the other rank groups.
Figure 4 also suggests that school accreditation rank is positively associated with
percentage of students with parents working as government officer
/army/police/private employee (X23), as an opposed of X22.

Figure 5 Partial dependence plot of school accreditation rank on X30 and X18.
The red ticks at the base of the plots are quartiles of the predictors.
Figure 5 shows that the averaged probability of Rank A is higher for older
school (X30). Ratio of students per class group (X18) is also positively associated
with school accreditation rank. In terms of parents’ occupation, percentage of
parents working as entrepreneur (X24) is positively associated with school
accreditation rank (Figure 6), while percentage of parents working as blue collar
(X25) is estimated to be negatively associated with school accreditation rank
(Figure 7).
In terms of teachers, the probability of Rank A is generally increasing with
increasing percentage of certified teachers except perhaps for percentage of
certified teachers less than 16% (Figure 7). It can be concluded that percentage of
certified