Minimum Constructive Back Propagation Neural Network Based on Fuzzy Logic for Pattern Recognition of Electronic Nose System
Network Based on Fuzzy Logic for Pattern Recognition of Electronic Nose System
Radi, Muhammad Rivai 1 , and Mauridhi Hery Purnomo
Abstract Constructive Back Propagation Neural Network (CBPNN) is a kind of back propagation neural network
trained with constructive algorithm. Training of CBPNN is mainly conducted by developing the network’s architecture which commonly done by adding a number of new neuron units on learning process. Training of the network usually implements fixed method to develop its structure gradually by adding new units constantly. Although this method is simple and able to create an adaptive network for data pattern complexity, but it is wasteful and inefficient for computing. New unit addition affects directly to the computational load of training, speed of convergence, and structure of the final neural network. While increases training load significantly, excessive addition of units also tends to generate a large size of final network. Moreover, addition pattern with small unit number tends to drop off the adaptability of the network and extends time of training. Therefore, there is important to design an adaptive structure development pattern for CBPNN in order to minimize computing load of training. This study proposes Fuzzy Logic (FL) algorithm to manage and develop structure of CBPNN. FL method was implemented on two models of CBPNN, i.e. designed with one and two hidden layers, used to recognize aroma patterns on an electronic nose system. The results showed that this method is effective to be applied due to
its capability to minimize time of training, to reduce load of computational learning, and generate small size of network.
Keywords CBPNN, structure development pattern, fuzzy logic, effective
Abstrak Constructive Back Propagation Neural Network (CBPNN) adalah jaring saraf perambatan balik galat yang dilatih
dengan algoritma konstruktif. Pelatihan CBPNN pada dasarnya dilakukan melalui metode pengembangan arsitektur jaring yang biasanya dilakukan dengan menambahkan sejumlah neuron baru pada lapis tersembunyi pada proses pelatihannya. Pelatihan jaring saraf ini dapat dilakukan dengan metode fixed, yaitu metode pengembangan struktur jaring dengan pola penambahan sejumlah neuron konstan secara bertahap. Meskipun cara ini mudah dilakukan dan telah mampu membangun struktur jaring yang adaptif terhadap kempleksitas data pelatihan, namun dari sisi komputasi dipandang kurang efisien. Penambahan neuron baru secara langsung berdampak terhadap beban komputasi pelatihan, kecepatan konvergensi, dan struktur jaring saraf yang terbentuk. Selain memberatkan beban komputasi, penambahan neuron yang terlalu banyak cenderung menghasilkan struktur jaring akhir yang besar. Di lain pihak, pola penambahan dengan sedikit neuron dapat menurunkan kemampuan adaptasi jaring saraf dan cenderung menambah waktu pelatihan. Oleh karena itu, pola pengembangan struktur jaring CBPNN yang adaptif perlu didesain untuk menurunkan beban komputasi pelatihan. Penelitian ini mengusulkan algoritma Fuzzy Logic (FL) untuk mengadaptifkan pengembangan struktur jaring CBPNN. Metode FL ini diterapkan pada pelatihan CBPNN dengan dua struktur, yaitu CBPNN dengan satu lapisan tersembunyi dan CBPNN dengan dua lapisan tersembunyi. Hasil penelitian menunjukan bahwa metode ini cukup efektif untuk diterapkan karena mampu meminimumkan waktu pelatihan, mengurangi beban komputasi dan menghasilkan struktur jaring saraf akhir yang lebih kecil.
Kata Kunci CBPNN, pola pengembangan struktur, logika fuzzy, efektif
I. I NTRODUCTION 4 recognition algorithms for product quality have been lectronic nose system is artificial olfaction
E technology, works as a representative of human olfactory system, usually used to analyze gaseous
conducted, such as applied to identify the quality of tea and spoiled beef [2-3].
ANN algorithm has been developed and mostly applied mixtures, discriminate different mixtures, or quantify the
in many disciplines especially those related to the concentration of gas component. This instrument
artificial intelligence systems. ANN is very helpful in the consists of four main parts, namely an odor sampling
field of control and pattern recognition systems. system, odor sensitive receptor, electronic circuitry, and
Implementation of ANN increased significantly in recent data analysis software. Part of odor receptor usually
decades, mainly supported by the development of pattern implements a number of gas sensors arranged in an array
recognition research for various object studies. One form. While a pattern recognition software is commonly
model of ANN usually used as pattern recognition is used to analyze the data [1]. Beside sensors, performance
Multi Layer Perceptron (MLP). MLP network can be of the electronic nose system is determined by pattern
trained with error back propagation method. Hence, this recognition system. One of the pattern recognition
network is often popular with Back Propagation Neural algorithm broadly used in electronic nose systems is
Network (BPNN) [4]. Study showed that compared to Artificial Neural Network (ANN). Researches on
other networks, BPNN has better performance [5]. electronic nose system by using ANN as pattern
BPNN has great ability to identify and recognize data patterns. Performance of the network is very dependent on the network architecture. The solution quality found
Radi, Muhammad Rivai, and Mauridhi Hery Purnomo are with Department of Electrical Engineering, FTI, Institut Teknologi Sepuluh
by general BPNN depends strongly on the network size
Nopember, Surabaya,
used. Size of network also determines learning process
radi10@mhs.ee.its.ac.id.
and its ability to recognize a given data pattern. Small
IPTEK, The Journal for Technology and Science, Vol. 22, No. 3, August 2011 identify complex patterns. Otherwise, BPNN with larger
structure has better performance to recognize difficult patterns accurately [6-8]. On the other hand, large-sized of BPNN is more difficult to be trained than the small one. Choosing large size of BPNNs in one case often does not improve their performance and tends to aggravate the computational burden of training. Moreover, large size neural network commonly contains of noise, which cause the network lose its ability to recognize patterns [7-8]. Therefore, properly network has to be designed carefully.
Application of small BPNN provides many advantages. Small BPNN can be easily trained with a light computational load. Besides, this network is also programmable. Implementation of this network requires only a minimum space of memory. Today, technology enables to design a portable electronic nose by applying microprocessor such as microcontroller or FPGA. In this system, application of small neural network is desirable. Therefore, designing minimum size of ANN is required. Researchers still investigate and review some algorithms in order to design minimum neural networks, such as finding minimum network by using pruning methods for business intelligent applications [9].
There are three methods commonly used to design neural network correspond to the patterns which want to
be classified [6]. First is trial and error method. This method is conducted by training various sizes of neural network for a training data set, and then selects an appropriate neural network. Beside difficult to be implemented, this method needs experiences. The second method is pruning algorithm. Firstly, we train large size network. During training, the network size is reduced by eliminating some neurons, which have less contribution to the network outputs. Elimination of unnecessary neuron is done continuously in order to obtain an appropriate neural network. The last is constructive method. Training of the neural network is started with simple architecture. Adding new unit progressively is required to develop the initial network. Training is finished when the network was able to classify the data pattern. Other methods are developed by combining some of these methods.
Constructive method has advantages compared to other methods [4, 6]. Constructive method applied on neural network is used to modify its previous structure in order to get a proper neural network. Constructive algorithm tries to find suitable network from very simple architecture, so this algorithm is easy to be computed. Besides, this method also tends to generate smaller network. In addition, it is also consistent to the provisions of CASA (Continuous Automation Structure Adaptation ) which commonly used as standard of adaptive neural network.
BPNN trained with constructive method is known as Constructive Back Propagation Neural Network (CBPNN). Training of CBPNN can be characterized by developing structure of the network gradually in order to create adaptive algorithm for various complexity of training data patterns. Before trained, the neural network is initiated by a simple structure. This initial network is trained to recognize a given training data. If the first network fails to recognize the patterns, developing structure has to be conducted by adding new neurons in
the hidden layer. Adding unit is approved continuously until the neural network produce accurate results on patterns outside its training set.
Pattern of structures development is the focus on training of CBPNN. Pattern by adding new unit constantly, which in this paper referred to fixed method, is structure development pattern that widely implemented to train of general CBPNN. In this method, number of new units added at each step of training should be managed fixedly. Although the training pattern can be implemented easily and creates adaptive CBPNN for the data patterns complexity, but it is wasteful and inefficient computing. Adding size of network will affect to the computational load of learning, speed of convergence, and structure of the final network. Change of structure due to new unit added tends to enlarge load of computational running. Adding new unit means increasing neuron number and network connection simultaneously. Lot of neurons needs hard effort to be computed. Besides, too many connections must be modified concurrently. In addition, excessive addition pattern while increasing computational load also tends to produce large-sized of final network. Furthermore, addition pattern by too small new unit number will reduce the network adaptability and extend time to learn. Therefore, there is important to design adaptive structure development pattern for CBPNN in order to minimize load of training process.
The aim of this study is to implement Fuzzy Logic algorithm (FL) in training of CBPNN. This algorithm will be used to modify structure development pattern of CBPNN. Modifying structure development is desired to minimize effort and to streamline training process of the network. We expect that this method is powerful to be implemented on training of CBPNN.
II. M ETHOD
A. Basic Training of CBPNN Learning process of CBPNN is closely related to the
learning process of mostly BPNN. As discussed in the previous section, BPNN is trained with error back propagation method. Generally, training process of BPNN can be described as follows. First, we should prepare a pair of input and target pattern into a set of training data and training parameters required (learning rate, momentum, maximum error). BPNN is early initialized with random weights. Input pattern of data set is fed to the input nodes of the BPNN. Learning process is started by forwarding this input signal through to all neurons in the hidden layer. All neurons in this layer will count all input signals based on the appropriate connection weight. Hidden neurons respond the signal by issuing output signals based on their activation function. These signals are forwarded to the output neurons through to their connection weight. Output neurons respond these signals by issuing signals as output signals of the network. After that, these signals are compared to the target. Network error is defined as deference of the target and the network output. The error has to be propagated back to the previous layer and then forwarded to the input layer. Interconnection weights of the network are modified based on the value of the back propagation of error related to the learning parameter.
137 Learning process is finished when the network converges
IPTEK, The Journal for Technology and Science, Vol. 22, No. 3, August 2011
e k t k Y k (5)
at the specified error threshold. Training network with back propagation involves three
Means Square Error (MSE) of the network can be steps, they are: step to feed-forward all input patterns,
formulated from the errors with the equation as follow:
MSE 2
step to calculate and back propagate the associated error, nOut put 1 2 1 nOut put
t k Y k and step to modify connection weight of the network [9]. (6) 2 k 1 2 k 1
Structure of one hidden layer of BPNN is illustrated on After that, this MSE has to be minimized by gradient Figure 1. This network consists of n input units of input
descent procedure. This method is conducted by back layer, n hidden neurons of hidden layer, n output neurons
propagating the error started from the output to the input of output layer, hidden bias, and output bias. w1 is
layer. It is used to modify the overall network connection weights between input layer and hidden
connections. Weight corrections of the network can be layer, b1 is connection weights of hidden bias and hidden
calculated from the derivative of the MSE for the layer, w2 is connection weight of hidden layer and
connection weights which are going to be modified. output layer, and b2 represents connection weight
Weight correction of w2 ( Δw2) can be determined with between output bias and output layer.
the following equation:
MSE Yin training data set. This data consists of n data pairs of
Before training is started, we ought to prepare a
MSE
w 2 Yin w 2 (7)
input data pattern x i (i = 1, 2, 3, … n input) and target t j , k
(j = 1, 2, …n output). These Data Patterns prepared (DP) w 2 j , k e k f ' ( Yin k ) V j can be formulated as follows:
where µ is learning rate parameter. While the weight
correction of b2 ( Δb2) is determined by the following w 1 ,b 1 ,w 2 ,b 2 are firstly initiated by randomize function.
DP {(x 1 , t 1 ), (x 2 , t 2 ), , (x nDat a , t nDat a )} (1)
Detail of these steps can be explained as follow:
1. Step to feed-forward the input patterns
Yin k
This step is also popular with forward step. This step
b 2 k e k f ' ( Yin k )
calculates all output signals of the network. Each pair of Weight correction of w1 ( Δw1) is calculated with the the training data is used to train the network. The first
following equation:
pair of the data that consist of n input data are used as
first input pattern of the neural network. In the input
layer, the data is forwarded to all hidden neurons with w 1 i , j e k f ' ( Yin k ) w 2 j , k f ' ( Vin j ) x i their related connection weight w1. Each hidden unit (Vj,
j = 1, .. n hidden) counts all input signals from the Then, equation 10 is used to calculate weight correction previous layer and connection weight of the bias (b1).
of b1 ( Δb1).
Mathematically, it can be formulated as follows:
Vin j b 1 j (10)
Vin j b 1 j x i w 1 i , j (2)
i 1 b 1 j e k f ' ( Yin k ) w 2 j , k f ' ( Vin j ) All neurons in the hidden layer will issue output
3. Step to modify connection weight of the network signals that appropriate to their activation function (f).
If the network is trained with momentum constant ( ⍺), Signals of all hidden unit neurons can be written with the
weight connection of the network (w1, w2, b1, and b2) following formula:
can be modified with the following equation:
V f ( Vin ) (3)
w 1 i , j ( new ) w 1 ( old ) w 1 ( new ) w 1 ( old )
b 1 j ( new ) b 1 j ( old ) b 1 j ( new ) b 1 j ( old ) (11) to all units of the output layer after multiplied with
jj
Then, output signals of the hidden layer are forwarded
w 2 j , k ( new ) w 2 j , k ( old ) w 2 j , k ( new ) w 2 j , k ( old ) suitable connection weights. Each output unit of the
b 2 k ( new ) b 2 k ( old ) b 2 k ( new ) b 2 k ( old ) network sums weighted signal of the previous layer and
its bias. Output neurons generate output signals (Yk, k = Training process is continued until the network is able
1, 2, ... n output) based on their activation function, to recognize the training data pattern, which can be which can be formulated as follows:
indicated by its ability to convergence at a specified error
nHidden
threshold. Training of BPNN with some hidden layers
Yin k b 2 k V j w 2 ,
can be adopted from the previous training algorithm.
j 1 (4)
Training of CBPNN is conducted with error back propagation algorithm as same as learning process of
Y k f ( Yin k ) BPNN. Constructive algorithm is added to modify
2. Step to calculate and back propagate the associated structure of the network. Principally, training algorithm error
of general CBPNN can be described into 4 steps, as This step is frequently called with backward step. This
follows:
step counts error of the network and propagates the error
a. Initialization step, the phase of initial network to all connections of the network. Each output neuron is
configuration. Initial network is constructed without going to receive target pattern t k accordance with the
hidden units. The weight of this configuration is training input patterns. Difference between the network
updated by error back propagation to minimize the outputs and the target is defined as error (e k ) of the
MSE
network. This error is calculated by the following
b. Structure development step by adding new hidden equation:
units, i.e. by connecting input units to the new hidden
138 IPTEK, The Journal for Technology and Science, Vol. 22, No. 3, August 2011
unit and connecting the new hidden unit to the output computational process of training. Fast and slow units of the network. All connection weights are
structure changes will be not effective in learning firstly initialized with random function.
process [6]. In order to minimize training process,
c. Training of new network. After adding new units, the structure development pattern has to be managed. new network is trained by error back propagation to
Number of new units ought to be adjusted based on the modify connection weight of the network in order to
error value at that step. If the network issue signal with minimize the new MSE. Correcting weight can be
big error then a number of hidden units must be added, done by modifying all network connections or only
but if the error is small means that the network has been limited to the new connection added.
closely to the expected structure. In this condition, the
d. Convergence test, that is by checking the network network only needs addition time to convergence and output. When the network yields an acceptable
even does not require new neurons. On the other cases, solution, then stop the training, otherwise return to
rapid dropping off error indicates that the network is step 2.
possible to recognize the training data set. Based on the Adding few hidden units can simultaneously accelerate
previous consideration, number of new units should be the formation of the neural network. Besides that, this
modified automatically and called with adaptive way tends to minimize the output error faster than
structure development pattern. FL algorithm is used to training of some units independently [4].
the algorithm. Implementation of FL on training of Constructive method effects on the number of hidden
CBPNN can be presented on Figure 2. unit neuron. The number is modified by pattern of
FL algorithm is designed by two inputs. Both inputs structure development. Constructive method modifies
are formulated on fuzzy rules to generate FL output that size of the network structure automatically. Hence,
represents new unit number should be added. Fuzzy learning equation of CBPNN can be rewritten as follows:
logic thinking involves three processes namely
nInput
fuzzification, inference, and defuzzification step. Input Vin j b 1 j x i w 1 i , j (12) signal must be converted into fuzzy variables. These
i 1 fuzzy variables are determined based on the membership
nHidden NU
Yin k b 2 k V j w 2 j , k (13)
functions of these variables. Inference is stage to
j 1 calculate fuzzy output based on the fuzzy rules defined. The last step is converting the fuzzy variable into desired
w 2 j , k e k f ' ( Yin k ) V j (14)
output number. Design of FL implemented on CBPNN
w 1 i , j e k f ' ( Yin k ) w 2 j , k f ' ( Vin j ) x (15)
can be outlined as follows:
1. Memberships function of input
FL algorithm is constructed with two inputs, i.e. value of error (MSE) and change of MSE (dMSE). MSE is where NU is number of new unit and j is used to
b 1 j e k f (' Yin k ) w 2 j , k f (' Vin j ) (16)
transformed into fuzzy variable based on the represent adaptive variable. NU is set constantly on fixed
membership functions, which can be presented on Figure method and is going to be modified by FL method. Then,
3. the new network weight can be determined as follow: There are five membership functions of MSE, namely
w VS for very small MSE, S for small MSE, M for medium 1
i , j ( new ) w 1 i , j ( old ) w 1 i , j ( new ) w 1 i , j ( old )
MSE, B for big MSE, and VB for very big MSE. While,
b 1 j ( new ) b 1 j ( old ) b 1 j ( new ) b 1 j ( old )
value of dMSE is grouped into 3 membership functions,
w 2 j , k ( new ) w 2 j , k ( old ) w 2 j , k ( new ) w 2 j , k ( old )
i.e. B for fast changes, M for medium changes, and S for
slowly changes. Value of these functions can be exposed
b 2 k ( new ) b 2 k ( old ) b 2 k ( new ) b 2 k ( old )
on Figure 4.
2. Rules of FL
B. Implementation of Fuzzy Logic on Training Of Rules are the key of FL operation. These rules connect CBPNN
the input variable and the output of FL. Fuzzy rules used CBPNN is firstly initiated with simplest structure,
in this study are described on Table 1. commonly without hidden unit. In this study, we design
3. Memberships function of output one neuron in the hidden layer for the first arrangement.
Output of FL is used to modify number of New Units This initial configuration is trained to recognize data
(NU). To simplify the case, we implement singleton pattern provided at limited iteration. If the network fails
function as membership function of the fuzzy output. to recognize the data set, some neurons are added to the
There are five membership functions used, namely VS hidden layer. Then the new network architecture is
for very small, S for small, M for medium, B for big, and trained with the previous pattern. Training process is
VB for very big. Value of each function can be shown on continued until we get suitable network for the training
Figure 5.
data set. Training method by modifying structure is
4. Inference
typical algorithm of CBPNN. Main part of fuzzy logic operation is inference. In this As already discussed, changing of network structure
study, we use min method, formulated as follow: effects to the computing process. The number of new
min MSE , dMSE (18)
units added has to be tuned carefully to simplify Output of this logic (z) is calculated with centroid function, which can be formulated as follow:
z D (19)
139 where D is decision index. Then NU can be formulated
IPTEK, The Journal for Technology and Science, Vol. 22, No. 3, August 2011
where VS is for very small (z ≤ 0.2), S is for small (0.2 < as follows:
z ≤ 0.4), M is for medium (0.4 < z ≤ 0.6), B is for big
NU f (z ) (20)
(0.6 < z ≤ 0.8), and VB is for very big (0.8 < z).
C. Experimental Design
D. Training Data Set
FL method is implemented in training of CBPNN. There are two structures of the network investigated in
Training data sets used in this study were aroma this study, i.e. One Hidden Layer of CBPNN (OHL-
patterns of tobacco. There were five samples of tobacco, CBPNN) and Double Hidden Layer of CBPNN (DHL-
captured by combination of 7-gas sensors, namely CBPNN). Both networks are commonly used on an
MQ135, MQ136, MQ137, MQ138, MQ3, TGS2620, and electronic nose system. The result will be compared to
TGS822. Average pattern of these samples are presented the fixed method implemented on the same network for
on Figure 6. We prepared five training data sets with the similar training data set.
different combination of these sensors. Data 1 Implementation of both methods can be detailed as
represented 5 patterns of tobacco captured by follows. Fixed method on OHL-CBPNN is conducted by
combination of 3 sensors. Data 2 was pattern of the adding fixed unit number. There are five variations of
sample odor with 4 sensors, data 3 with 5 sensors, data 4 unit number studied, i.e. 1 unit, 2 units, 3 units, 4 units,
with 6 sensors, and the last data was captured by and 5 units. Batch iteration is set up at 300 epochs,
combination of seven sensors.
related to the previous study conducted by Radi et.al. (2011). While, NU is arranged in the following section:
III. R ESULTS AND D ISCUSSION NU (unit) : 0 Both CBPNNs were trained to recognize all data sets.
VS Batch iteration : 300 Training was conducted for both neural networks for five
NU (unit) : 1 repetitions for each data set. Both of them were trained S with learning rate of 0.7, momentum of 0.2, and
Batch iteration : 300 maximum error threshold of 0.001. This study conducted NU (unit) : 2
M on laptop with specification of Intel Celeron M processor Batch iteration : 300 440 (1.86 GHz, 533 MHz FSB, 1 MB L2 cache) with
B Batch iteration : 300 Effectiveness of both algorithms (fixed and FL) have
NU (unit) : 3 memory of 1.5 MB DDR2.
been evaluated. There are three components used for this
NU (unit) : 4 VB
evaluation, i.e. size of final network, number of training Batch iteration : 300 iteration, and time of network learning. The network size
where VS is for very small (z ≤ 0.2), S is for small (0.2 < in general MLP network involves the number of layers z ≤ 0.4), M is for medium (0.4 < z ≤ 0.6), B is for big
and the number of hidden-units per layer. Both of them (0.6 < z ≤0.8), and VB is for v ry big (0.8 < z).
reflect the complexity of the network. Number of units in For DHL-CBPNN, we implemented the fixed method
each layer presents the numbers of connections in the by varying number of new units added on the first and network. Number of layers indicates the stage of
second hidden layer simultaneously. By equal internal computing process when it trained and used. Normally, iteration, number of new units is managed with range of
the number of units in the input and output layers can be
1 to 5 units gradually. Whereas, implementation of FL determined by the dimensionality of the problem or method on DHL-CBPNN training is performed by
training data set, so the number cannot be set up when modifying NU based on the following arrangement:
the network trained. Studying neural network which have determined configuration layer, observation of the NU (1 st hidden layer, unit) : 0 network size is focused on the number of neurons in the
hidden layer. In this study, the network size is
VS
nd NU (2 hidden layer, unit) : 0
represented by the number of neurons in the hidden Batch iteration : 300 layer. Time for training and number of iterations are used
NU (1 st hidden layer, unit) : 1 to represent the load of learning process. Result of this study is detailed as follows:
S NU (2 nd hidden layer, unit) : 1
A. Training of OHL-CBPNN
Batch iteration : 300 OHL-CBPNN was prepared and trained for all data
NU (1 st hidden layer, unit) : 2 sets. Observation involves size of final network, training
iteration, and training time. The result of this study is
nd
presented on Table 2. This result written on the table is Batch iteration : 300
M NU (2 hidden layer, unit) : 1
average value of five experimental data. From the Table 2, we clearly know that the smaller
NU (1 st hidden layer, unit) : 3 size of final network is generated by fixed 1 and FL
B NU (2 nd hidden layer, unit) : 2 methods. Based on size consideration, result showed that
Batch iteration : 300 adding more unit numbers simultaneously tends to generate bigger size of the final network. Comparison of
NU (1 st hidden layer, unit) : 4 both algorithms is clearly understood on Figure 7.
VB NU (2 nd hidden layer, unit) : 3 It follows from Figure 7 that size of final OHL-
Batch iteration : 300
CBPNN is determined by structure development pattern
140 IPTEK, The Journal for Technology and Science, Vol. 22, No. 3, August 2011
smaller structure is generated by applying smaller new iteration. Based on the results, FL method and big new unit number. The study shows that fixed 1 method can
unit number on fixed method can be recommended. construct final network with smaller size. Compare to
Training time of DHL-CBPNN for both algorithms fixed 1 method, FL method also generate small network.
can be clearly evaluated from Figure 12. The figure From this result, both methods can be recommended to
shows that training time of the network for all data train the OHL-CBPNN.
patterns was determined by implementation of training Beside size of network, effectiveness of the proposed
methods. From the figure, we can conclude that FL method is evaluated from number of training iteration.
method give the best performance due to minimum time Comparison of iteration number for both methods can be
of convergence.
shown on Figure 8. Figure 8 presents that application of Based on the three parameters studied before, we both methods does not significantly affect to the number
consider that FL method can be used as new strategy on of iteration. The biggest influence is only seen on the
training of DHL-CBPNN. Due to the same result on OHL-CBPNN with fixed 1 method. Result shows that
training of OHL-CBPNN, we sure that this method can this method tends to increase number of training
be implemented on training of general CBPNN. iteration. This is understandable because this method
C. Analyze of Both Algorithms on Training of OHL- cause the structure changed slowly. Based on these
CBPNN and DHL-CBPNN
results, the FL and fixed method with smaller number of iteration may be selected.
In this section, we focus on analyzing performance of Time of network training is also used to evaluate
FL method compared to fixed methods. In order to performance of the proposed algorithm. Training time of
evaluate effectiveness of FL method, we only compare the OHL-CBPNN with various data sets is presented on
the method to the best and average performance of fixed Figure 9. With this result, we know that number of new
method. Best performance in this section is defined as unit added in the fixed method determines speed of the
method, which is able to generate small size of final network convergence. Fixed method with small number
network with minimum iteration and time. Percentages of units tends to accelerate its training process. FL
of network size, training iteration, and training time are method also shows good performance. This method is
discussed. Network size comparison of FL method and able to minimize load of computation and results
related fixed method is presented on Table 4. network with high speed of convergence. Compare to the
Colum A in the Table 4 is smaller network size for all studied method, FL method can be recommended.
fixed methods implemented in training of CBPNN. Fixed Based on the previous parameters, we can conclude
1 method showed the best performance due to its ability that FL methods applied on OHL-CBPNN training
to generate smallest network for all data patterns. We can shows the best performance. This method can be used to
evaluate the performances of both algorithms by optimize training of the network. This is due to results of
comparing the data. Result shows that FL method training time, training iteration, and network size of the
implemented on OHL-CBPNN was able to generate network.
smaller network compare to fixed 1 method (94.98 %). Better performance of FL method is showed if we
B. Training of DHL-CBPNN compare it with the average size of final network of DHL-CBPNN is trained to recognize the same data
various fixed methods. FL method is able to produce less sets. The similar manner is applied in this experiment in
than a half size of final network with fixed method. accordance with the previous experimental design. In
Similar performance of FL method is also presented on this study, we also focus to evaluate 3 parameters, i.e.
training of DHL-CBPNN. FL method is effective to size of network, training time, and iteration. Study result
minimize final network size with average of 91.47 % shows on Table 3.
compared to the best fixed method and 48.71 % Based on Table 3, we can analyze that both, proposed
compared to average fixed method implemented. This method and comparison method, show similar
result showed that the proposed method is effective to performance with the previous study. Smaller size of
find minimum network architecture. network is also generated by applying of FL and Fixed 1
Beside network size, performance evaluation of both methods. Both FL and fixed 1 method may be chosen to
methods can be evaluated from training iteration. generate small CBPNN. Evaluation of network size for
Comparison of training iteration between FL method and both applied algorithm for all variations of data sets is
related fixed method is presented on Table 5. It shown presented on Figure 10. From this result, we consider
that fixed method (the best and average) has better that both algorithms show the similar characteristic.
performance than FL method for OHL-CBPNN and Observations of training iteration of DHL-CBPNN are
DHL-CBPNN. CBPNN trained by fixed method is able presented in Figure 11. The graph shows that variations
to convergence with less training iteration than FL of the fixed method employed are not exhibited
method implemented. It can be understood from the significantly affects to the network training iterations.
design of FL discussed in the previous section. The biggest influence can be understood on the
Effectiveness of both methods can also be evaluated implementation of the fixed 1 method. Applying of this
by comparing time of training process. Training time method for DHL-CBPNN tends to increase number of
assessment of both methods for OHL-CBPNN and DHL- training iteration. The result shows that the network
CBPNN are detailed on Table 6. The table shows that convergent with higher number of training iterations than
comparing to fixed method; FL method has better the other methods implemented. This is understandable
performance due to its capability to minimize time of because the structure development was too slow and
convergence. Based on the study result, training of OHL- each change of structure requires considerable training
CBPNN with FL method is able to convergence with
141 only 43.96 % of training time used by average fixed
IPTEK, The Journal for Technology and Science, Vol. 22, No. 3, August 2011
where I is number of training iteration and t is total time method and 14.78 % less than the best fixed method.
needed to train the network. Number of f shows the load The similar result is showed on training of DHL-
of computing process.
CBPNN. In this case, the proposed method also presents Comparison of f for both methods implemented on better performance. The study result shows that the
both CBPNN is presented on Table 7. It shown that network trained using FL method was able to recognize
proposed method has better performance than proper the data pattern more than two times faster than the
fixed method. Value of iteration frequency of CBPNN average performance of fixed methods studied. The data
trained by FL method is bigger than the similar neural shows that DHL-CBPNN with FL method is able to
network when fixed method is implemented. It means convergence with only 39.66 % of average time needed
that FL method tends to reduce load of computational when the fixed method applied and 73.88 % of time
process significantly. Implementation of FL shows that needed by the best fixed method implemented.
load of processor is lighter than the other methods Based on the study result, we can conclude that FL
implemented on training of CBPNN. method is suitable to train not only OHL-CBPNN but
These results prove that the addition of the algorithm also DHL-CBPNN. Although this method tends to add a
does not necessarily aggravate the computational load, few number of training iteration, FL method is able to
but in this case, the addition of FL algorithm can minimize the final network size and time of training.
decrease load the network training. Thus, we agree that
D. Computational Load of Training combining FL algorithm on CBPNN does not increase training load, but otherwise, it tends to drop off training
Computational load is mainly used to evaluate
load.
effectiveness of an algorithm or method. Computational load is closely related to the algorithm design. A
IV. C ONCLUSION program, which contains a lot of multiplier, is usually
difficult to be executed. In this case, the algorithm has This paper presented training of CBPNN by high computational load. We also can predict the
implementing FL method and fixed method. The neural computational load by evaluating difficulty level of the
network was used to recognize aroma pattern on an algorithm. Besides, computational load can also be
electronic nose system. The result shows that FL method valued from the computer load when the algorithm is
applied on OHL-CBPNN and DHL-CBPNN is able to running. Light algorithm is easy to be computed. In this
generate small final neural network. Besides that, FL section, we only focus on the effect showed on computer
method on both CBPNNs can minimize training time. FL when the both methods implemented on the OHL-
method is also possible to reduce load of computational CBPNN and DHL-CBPNN.
training. Combining FL method and CBPNN is possible In the previous section, we captured training iteration
to increase performance of the network. Therefore, this and training time data. Both are used to calculate
method can be recommended to arrange structure iteration frequency (f) of training of the neural network.
development of general CBPNN. Frequency of iteration means number of iteration for
each second, which can be formulated as follows: I
pattern Target
x nInput
t nOutput
b1 b2 layer
1 Hidden
Hidden
layer Output
bias
bias
Figure 1. Structure of one hidden layer of BPNN
142 IPTEK, The Journal for Technology and Science, Vol. 22, No. 3, August 2011
Start
Set of Training Data Network Training Parameter
Design of Initial Network
Network Training
Save Final Network w1 w2 w3
Design of New Network
Accepted Error
layer Output
hidden
Bias
output Bias
No FL Method
MSE dMSE
Figure 2. FL method implemented on CBPNN
10xErrLim 100xErrLim 1000xErrLim nOutput JumData -nOutput -1000xErrLim -10xErrLim
Figure 3. Memberships function of MSE Figure 4. Memberships function of dMSE
VS S M B VB
Figure 5. Membership function of FL output
IPTEK, The Journal for Technology and Science, Vol. 22, No. 3, August 2011
1 2 3 4 5 1 2 3 4 5 M Q135
M Q136
TGS2620
M Q135
M Q137
M Q136 TGS2620
M Q135 M Q3 M Q137
M Q136
(e) Figure 6. Patterns of training data set (a) data 1, (b) data 2, (c) data 3, (d) data 4, (e) data 5
Figure 7. Final network size of OHL-CBPNN with fixed and FL Figure 8. Training iteration of OHL-CBPNN with fixed and FL methods for all training data sets
methods for all training data sets
Figure 9. Training time of OHL-CBPNN with fixed and FL methods Figure 10. Final network size of DHL-CBPNN with fixed and FL for all training data sets
methods for all training data sets
144 IPTEK, The Journal for Technology and Science, Vol. 22, No. 3, August 2011
Training Iteration Training Time
p E 8,000
Fixed 5 Fixed 4
Fixed 2 Fixed 1 FL
Data 1 Data 2
Data 3 Data 4 Data 5
Figure 11. Training Iteration of DHL-CBPNN with fixed and FL Figure 12. Training time of OHL-CBPNN with fixed and FL methods methods for all training data sets
for all training data sets
T ABLE 1. R ULE OF FL
T ABLE 2. T ABLE 3. C OMPARISON OF T WO M ETHODS IN T RAINING OF OHL-CBPNN
C OMPARISON OF T WO M ETHODS IN T RAINING OF DHL-CBPNN Method
Network size Iteration Training Data Set
Training Network size
(units)* (epoch) Time (s) Data 1
(units)*
(epoch)
Time (s)
Data Set
49 5878 9.212 *) number of hidden neurons of the final networks
*) number of hidden neurons of the final networks
IPTEK, The Journal for Technology and Science, Vol. 22, No. 3, August 2011
T ABLE 4. F INAL NETWORK S IZE C OMPARISON OF FL AND R ELATED F IXED M ETHOD
Data set
A: Final network size of the best fixed method performance B: Average of final network size of fixed methods used
T ABLE 5. T RAINING I TERATION C OMPARISON OF FL AND R ELATED F IXED M ETHOD
Data Set
A: Training iteration of the best fixed method performance B: Average of training iteration of fixed methods used
T ABLE 6. T RAINING TIME COMPARISON OF FL AND R ELATED F IXED M ETHOD
Data Set
A: Training time of the best fixed method performance B: Average of training time of fixed methods used
146 IPTEK, The Journal for Technology and Science, Vol. 22, No. 3, August 2011
T ABLE 7. F REQUENCY ITERATION C OMPARISON OF FL AND R ELATED F IXED M ETHOD
Data Set
A: Iteration frequency of the best fixed method performance B: Average of f of fixed methods used
B. Lan, “Comparison of Algorithms for an El ctronic Nos in Id ntifying Liquors”, [1] T. C. Pearce, S. S. Schiffmen, H. T. Nagle, and J. W. Gardner,
R EFERENCES [5] Z. B. Shi, T. Yu, Q. Zhao, Y. Li, and Y.
Journal of Bionic Engineering, vol. 5, pp. 253 –257, 2008. Handbook of Machine Olfaction . New York: WILEY-VCH
M. L htokangas, “Mod ling ith constructiv backpropagation”, Verlag GmbH & Co. KgaA., 2003.
[6]
[2] H. Yu, J. Wang, C. Yao, H. Zhang, and –716, 1999.
Y. Yu, “Quality grad
Neural Networks , vol. 21, pp.707
M. L htokangas, “Modifi d constructiv backpropagation for identification of green tea using E- nos by CA and ANN”, LWT,
[7]
r gr ssion”, Neurocomputing, vol. 35, pp. 113-122, 2000. vol. 41, pp. 1268 –1273, 2008.
. S tiono, “Finding minimal n ural networks for business [3] S. Panigrahi, S. Balasubramanian, H. Gu, C. Lague, and M.
[8]
int llig nc application”. presented at IFSA-AFSS International March llo, “N ural-network-integrated electronic nose system for
id ntification of spoil d b f”, LWT, vol. 39, pp. 135–145, 2005,. Conference, Surabaya-Bali Indonesia, 2011. [9] T. S. Widodo, Sistem Neurofuzzy. Yogyakarta: Graha Ilmu, 2005.
[4] M. H. Purnomo and A. Kurniawan, Supervised Neural Networks [10] Radi, M. Rivai,M. H. Purnomo, “Multi-thread constructive back dan Aplikasinya . Yogyakarta: Graha Ilmu, 2006.
propagation n ural n t ork for aroma patt rn classification”, in Proc. 2011 IFSA-AFSS International Conference .