Minimum Constructive Back Propagation Neural Network Based on Fuzzy Logic for Pattern Recognition of Electronic Nose System

Network Based on Fuzzy Logic for Pattern Recognition of Electronic Nose System

Radi, Muhammad Rivai 1 , and Mauridhi Hery Purnomo

Abstract Constructive Back Propagation Neural Network (CBPNN) is a kind of back propagation neural network

trained with constructive algorithm. Training of CBPNN is mainly conducted by developing the network’s architecture which commonly done by adding a number of new neuron units on learning process. Training of the network usually implements fixed method to develop its structure gradually by adding new units constantly. Although this method is simple and able to create an adaptive network for data pattern complexity, but it is wasteful and inefficient for computing. New unit addition affects directly to the computational load of training, speed of convergence, and structure of the final neural network. While increases training load significantly, excessive addition of units also tends to generate a large size of final network. Moreover, addition pattern with small unit number tends to drop off the adaptability of the network and extends time of training. Therefore, there is important to design an adaptive structure development pattern for CBPNN in order to minimize computing load of training. This study proposes Fuzzy Logic (FL) algorithm to manage and develop structure of CBPNN. FL method was implemented on two models of CBPNN, i.e. designed with one and two hidden layers, used to recognize aroma patterns on an electronic nose system. The results showed that this method is effective to be applied due to

its capability to minimize time of training, to reduce load of computational learning, and generate small size of network.

Keywords CBPNN, structure development pattern, fuzzy logic, effective

Abstrak Constructive Back Propagation Neural Network (CBPNN) adalah jaring saraf perambatan balik galat yang dilatih

dengan algoritma konstruktif. Pelatihan CBPNN pada dasarnya dilakukan melalui metode pengembangan arsitektur jaring yang biasanya dilakukan dengan menambahkan sejumlah neuron baru pada lapis tersembunyi pada proses pelatihannya. Pelatihan jaring saraf ini dapat dilakukan dengan metode fixed, yaitu metode pengembangan struktur jaring dengan pola penambahan sejumlah neuron konstan secara bertahap. Meskipun cara ini mudah dilakukan dan telah mampu membangun struktur jaring yang adaptif terhadap kempleksitas data pelatihan, namun dari sisi komputasi dipandang kurang efisien. Penambahan neuron baru secara langsung berdampak terhadap beban komputasi pelatihan, kecepatan konvergensi, dan struktur jaring saraf yang terbentuk. Selain memberatkan beban komputasi, penambahan neuron yang terlalu banyak cenderung menghasilkan struktur jaring akhir yang besar. Di lain pihak, pola penambahan dengan sedikit neuron dapat menurunkan kemampuan adaptasi jaring saraf dan cenderung menambah waktu pelatihan. Oleh karena itu, pola pengembangan struktur jaring CBPNN yang adaptif perlu didesain untuk menurunkan beban komputasi pelatihan. Penelitian ini mengusulkan algoritma Fuzzy Logic (FL) untuk mengadaptifkan pengembangan struktur jaring CBPNN. Metode FL ini diterapkan pada pelatihan CBPNN dengan dua struktur, yaitu CBPNN dengan satu lapisan tersembunyi dan CBPNN dengan dua lapisan tersembunyi. Hasil penelitian menunjukan bahwa metode ini cukup efektif untuk diterapkan karena mampu meminimumkan waktu pelatihan, mengurangi beban komputasi dan menghasilkan struktur jaring saraf akhir yang lebih kecil.

Kata Kunci CBPNN, pola pengembangan struktur, logika fuzzy, efektif

I. I NTRODUCTION 4 recognition algorithms for product quality have been lectronic nose system is artificial olfaction

E technology, works as a representative of human olfactory system, usually used to analyze gaseous

conducted, such as applied to identify the quality of tea and spoiled beef [2-3].

ANN algorithm has been developed and mostly applied mixtures, discriminate different mixtures, or quantify the

in many disciplines especially those related to the concentration of gas component. This instrument

artificial intelligence systems. ANN is very helpful in the consists of four main parts, namely an odor sampling

field of control and pattern recognition systems. system, odor sensitive receptor, electronic circuitry, and

Implementation of ANN increased significantly in recent data analysis software. Part of odor receptor usually

decades, mainly supported by the development of pattern implements a number of gas sensors arranged in an array

recognition research for various object studies. One form. While a pattern recognition software is commonly

model of ANN usually used as pattern recognition is used to analyze the data [1]. Beside sensors, performance

Multi Layer Perceptron (MLP). MLP network can be of the electronic nose system is determined by pattern

trained with error back propagation method. Hence, this recognition system. One of the pattern recognition

network is often popular with Back Propagation Neural algorithm broadly used in electronic nose systems is

Network (BPNN) [4]. Study showed that compared to Artificial Neural Network (ANN). Researches on

other networks, BPNN has better performance [5]. electronic nose system by using ANN as pattern

BPNN has great ability to identify and recognize data patterns. Performance of the network is very dependent on the network architecture. The solution quality found

Radi, Muhammad Rivai, and Mauridhi Hery Purnomo are with Department of Electrical Engineering, FTI, Institut Teknologi Sepuluh

by general BPNN depends strongly on the network size

Nopember, Surabaya,

used. Size of network also determines learning process

radi10@mhs.ee.its.ac.id.

and its ability to recognize a given data pattern. Small

IPTEK, The Journal for Technology and Science, Vol. 22, No. 3, August 2011 identify complex patterns. Otherwise, BPNN with larger

structure has better performance to recognize difficult patterns accurately [6-8]. On the other hand, large-sized of BPNN is more difficult to be trained than the small one. Choosing large size of BPNNs in one case often does not improve their performance and tends to aggravate the computational burden of training. Moreover, large size neural network commonly contains of noise, which cause the network lose its ability to recognize patterns [7-8]. Therefore, properly network has to be designed carefully.

Application of small BPNN provides many advantages. Small BPNN can be easily trained with a light computational load. Besides, this network is also programmable. Implementation of this network requires only a minimum space of memory. Today, technology enables to design a portable electronic nose by applying microprocessor such as microcontroller or FPGA. In this system, application of small neural network is desirable. Therefore, designing minimum size of ANN is required. Researchers still investigate and review some algorithms in order to design minimum neural networks, such as finding minimum network by using pruning methods for business intelligent applications [9].

There are three methods commonly used to design neural network correspond to the patterns which want to

be classified [6]. First is trial and error method. This method is conducted by training various sizes of neural network for a training data set, and then selects an appropriate neural network. Beside difficult to be implemented, this method needs experiences. The second method is pruning algorithm. Firstly, we train large size network. During training, the network size is reduced by eliminating some neurons, which have less contribution to the network outputs. Elimination of unnecessary neuron is done continuously in order to obtain an appropriate neural network. The last is constructive method. Training of the neural network is started with simple architecture. Adding new unit progressively is required to develop the initial network. Training is finished when the network was able to classify the data pattern. Other methods are developed by combining some of these methods.

Constructive method has advantages compared to other methods [4, 6]. Constructive method applied on neural network is used to modify its previous structure in order to get a proper neural network. Constructive algorithm tries to find suitable network from very simple architecture, so this algorithm is easy to be computed. Besides, this method also tends to generate smaller network. In addition, it is also consistent to the provisions of CASA (Continuous Automation Structure Adaptation ) which commonly used as standard of adaptive neural network.

BPNN trained with constructive method is known as Constructive Back Propagation Neural Network (CBPNN). Training of CBPNN can be characterized by developing structure of the network gradually in order to create adaptive algorithm for various complexity of training data patterns. Before trained, the neural network is initiated by a simple structure. This initial network is trained to recognize a given training data. If the first network fails to recognize the patterns, developing structure has to be conducted by adding new neurons in

the hidden layer. Adding unit is approved continuously until the neural network produce accurate results on patterns outside its training set.

Pattern of structures development is the focus on training of CBPNN. Pattern by adding new unit constantly, which in this paper referred to fixed method, is structure development pattern that widely implemented to train of general CBPNN. In this method, number of new units added at each step of training should be managed fixedly. Although the training pattern can be implemented easily and creates adaptive CBPNN for the data patterns complexity, but it is wasteful and inefficient computing. Adding size of network will affect to the computational load of learning, speed of convergence, and structure of the final network. Change of structure due to new unit added tends to enlarge load of computational running. Adding new unit means increasing neuron number and network connection simultaneously. Lot of neurons needs hard effort to be computed. Besides, too many connections must be modified concurrently. In addition, excessive addition pattern while increasing computational load also tends to produce large-sized of final network. Furthermore, addition pattern by too small new unit number will reduce the network adaptability and extend time to learn. Therefore, there is important to design adaptive structure development pattern for CBPNN in order to minimize load of training process.

The aim of this study is to implement Fuzzy Logic algorithm (FL) in training of CBPNN. This algorithm will be used to modify structure development pattern of CBPNN. Modifying structure development is desired to minimize effort and to streamline training process of the network. We expect that this method is powerful to be implemented on training of CBPNN.

II. M ETHOD

A. Basic Training of CBPNN Learning process of CBPNN is closely related to the

learning process of mostly BPNN. As discussed in the previous section, BPNN is trained with error back propagation method. Generally, training process of BPNN can be described as follows. First, we should prepare a pair of input and target pattern into a set of training data and training parameters required (learning rate, momentum, maximum error). BPNN is early initialized with random weights. Input pattern of data set is fed to the input nodes of the BPNN. Learning process is started by forwarding this input signal through to all neurons in the hidden layer. All neurons in this layer will count all input signals based on the appropriate connection weight. Hidden neurons respond the signal by issuing output signals based on their activation function. These signals are forwarded to the output neurons through to their connection weight. Output neurons respond these signals by issuing signals as output signals of the network. After that, these signals are compared to the target. Network error is defined as deference of the target and the network output. The error has to be propagated back to the previous layer and then forwarded to the input layer. Interconnection weights of the network are modified based on the value of the back propagation of error related to the learning parameter.

137 Learning process is finished when the network converges

IPTEK, The Journal for Technology and Science, Vol. 22, No. 3, August 2011

e k  t k  Y k (5)

at the specified error threshold. Training network with back propagation involves three

Means Square Error (MSE) of the network can be steps, they are: step to feed-forward all input patterns,

formulated from the errors with the equation as follow:

MSE 2  

step to calculate and back propagate the associated error, nOut put 1 2 1 nOut put

  t k  Y k  and step to modify connection weight of the network [9]. (6) 2 k  1 2 k  1

Structure of one hidden layer of BPNN is illustrated on After that, this MSE has to be minimized by gradient Figure 1. This network consists of n input units of input

descent procedure. This method is conducted by back layer, n hidden neurons of hidden layer, n output neurons

propagating the error started from the output to the input of output layer, hidden bias, and output bias. w1 is

layer. It is used to modify the overall network connection weights between input layer and hidden

connections. Weight corrections of the network can be layer, b1 is connection weights of hidden bias and hidden

calculated from the derivative of the MSE for the layer, w2 is connection weight of hidden layer and

connection weights which are going to be modified. output layer, and b2 represents connection weight

Weight correction of w2 ( Δw2) can be determined with between output bias and output layer.

the following equation:

 MSE Yin training data set. This data consists of n data pairs of

Before training is started, we ought to prepare a

 MSE

 w 2  Yin  w 2 (7)

input data pattern x i (i = 1, 2, 3, … n input) and target t j , k

(j = 1, 2, …n output). These Data Patterns prepared (DP)  w 2 j , k    e k  f ' ( Yin k )  V j can be formulated as follows:

where µ is learning rate parameter. While the weight

correction of b2 ( Δb2) is determined by the following w 1 ,b 1 ,w 2 ,b 2 are firstly initiated by randomize function.

DP  {(x 1 , t 1 ), (x 2 , t 2 ),  , (x nDat a , t nDat a )} (1)

Detail of these steps can be explained as follow:

1. Step to feed-forward the input patterns

 Yin k

This step is also popular with forward step. This step

 b 2 k    e k  f ' ( Yin k )

calculates all output signals of the network. Each pair of Weight correction of w1 ( Δw1) is calculated with the the training data is used to train the network. The first

following equation:

pair of the data that consist of n input data are used as

first input pattern of the neural network. In the input

layer, the data is forwarded to all hidden neurons with  w 1 i , j    e k  f ' ( Yin k )  w 2 j , k  f ' ( Vin j )  x i their related connection weight w1. Each hidden unit (Vj,

j = 1, .. n hidden) counts all input signals from the Then, equation 10 is used to calculate weight correction previous layer and connection weight of the bias (b1).

of b1 ( Δb1).

Mathematically, it can be formulated as follows:

 Vin j  b 1 j (10)

Vin j  b 1 j   x i  w 1 i , j (2)

i  1  b 1 j    e k  f ' ( Yin k )  w 2 j , k  f ' ( Vin j ) All neurons in the hidden layer will issue output

3. Step to modify connection weight of the network signals that appropriate to their activation function (f).

If the network is trained with momentum constant ( ⍺), Signals of all hidden unit neurons can be written with the

weight connection of the network (w1, w2, b1, and b2) following formula:

can be modified with the following equation:

V  f ( Vin ) (3)

w 1 i , j ( new )  w 1 ( old )   w 1 ( new )   w 1 ( old )

b 1 j ( new )  b 1 j ( old )   b 1 j ( new )    b 1 j ( old ) (11) to all units of the output layer after multiplied with

jj

Then, output signals of the hidden layer are forwarded

w 2 j , k ( new )  w 2 j , k ( old )   w 2 j , k ( new )    w 2 j , k ( old ) suitable connection weights. Each output unit of the

b 2 k ( new )  b 2 k ( old )   b 2 k ( new )    b 2 k ( old ) network sums weighted signal of the previous layer and

its bias. Output neurons generate output signals (Yk, k = Training process is continued until the network is able

1, 2, ... n output) based on their activation function, to recognize the training data pattern, which can be which can be formulated as follows:

indicated by its ability to convergence at a specified error

nHidden

threshold. Training of BPNN with some hidden layers

Yin k  b 2 k   V j  w 2 ,

can be adopted from the previous training algorithm.

j  1 (4)

Training of CBPNN is conducted with error back propagation algorithm as same as learning process of

Y k  f ( Yin k ) BPNN. Constructive algorithm is added to modify

2. Step to calculate and back propagate the associated structure of the network. Principally, training algorithm error

of general CBPNN can be described into 4 steps, as This step is frequently called with backward step. This

follows:

step counts error of the network and propagates the error

a. Initialization step, the phase of initial network to all connections of the network. Each output neuron is

configuration. Initial network is constructed without going to receive target pattern t k accordance with the

hidden units. The weight of this configuration is training input patterns. Difference between the network

updated by error back propagation to minimize the outputs and the target is defined as error (e k ) of the

MSE

network. This error is calculated by the following

b. Structure development step by adding new hidden equation:

units, i.e. by connecting input units to the new hidden

138 IPTEK, The Journal for Technology and Science, Vol. 22, No. 3, August 2011

unit and connecting the new hidden unit to the output computational process of training. Fast and slow units of the network. All connection weights are

structure changes will be not effective in learning firstly initialized with random function.

process [6]. In order to minimize training process,

c. Training of new network. After adding new units, the structure development pattern has to be managed. new network is trained by error back propagation to

Number of new units ought to be adjusted based on the modify connection weight of the network in order to

error value at that step. If the network issue signal with minimize the new MSE. Correcting weight can be

big error then a number of hidden units must be added, done by modifying all network connections or only

but if the error is small means that the network has been limited to the new connection added.

closely to the expected structure. In this condition, the

d. Convergence test, that is by checking the network network only needs addition time to convergence and output. When the network yields an acceptable

even does not require new neurons. On the other cases, solution, then stop the training, otherwise return to

rapid dropping off error indicates that the network is step 2.

possible to recognize the training data set. Based on the Adding few hidden units can simultaneously accelerate

previous consideration, number of new units should be the formation of the neural network. Besides that, this

modified automatically and called with adaptive way tends to minimize the output error faster than

structure development pattern. FL algorithm is used to training of some units independently [4].

the algorithm. Implementation of FL on training of Constructive method effects on the number of hidden

CBPNN can be presented on Figure 2. unit neuron. The number is modified by pattern of

FL algorithm is designed by two inputs. Both inputs structure development. Constructive method modifies

are formulated on fuzzy rules to generate FL output that size of the network structure automatically. Hence,

represents new unit number should be added. Fuzzy learning equation of CBPNN can be rewritten as follows:

logic thinking involves three processes namely

nInput

fuzzification, inference, and defuzzification step. Input Vin j  b 1 j   x i  w 1 i , j (12) signal must be converted into fuzzy variables. These

i  1 fuzzy variables are determined based on the membership

nHidden  NU

Yin k  b 2 k   V j  w 2 j , k (13)

functions of these variables. Inference is stage to

j  1 calculate fuzzy output based on the fuzzy rules defined.  The last step is converting the fuzzy variable into desired

w 2 j , k    e k  f ' ( Yin k )  V j (14)

output number. Design of FL implemented on CBPNN

 w 1 i , j    e k  f ' ( Yin k )  w 2 j , k  f ' ( Vin j )  x (15)

can be outlined as follows:

1. Memberships function of input

FL algorithm is constructed with two inputs, i.e. value of error (MSE) and change of MSE (dMSE). MSE is where NU is number of new unit and j is used to

 b 1 j    e k  f (' Yin k )  w 2 j , k  f (' Vin j ) (16)

transformed into fuzzy variable based on the represent adaptive variable. NU is set constantly on fixed

membership functions, which can be presented on Figure method and is going to be modified by FL method. Then,

3. the new network weight can be determined as follow: There are five membership functions of MSE, namely

w VS for very small MSE, S for small MSE, M for medium 1

i , j ( new )  w 1 i , j ( old )   w 1 i , j ( new )    w 1 i , j ( old )

MSE, B for big MSE, and VB for very big MSE. While,

b 1 j ( new )  b 1 j ( old )   b 1 j ( new )    b 1 j ( old )

value of dMSE is grouped into 3 membership functions,

w 2 j , k ( new )  w 2 j , k ( old )   w 2 j , k ( new )    w 2 j , k ( old )

i.e. B for fast changes, M for medium changes, and S for

slowly changes. Value of these functions can be exposed

b 2 k ( new )  b 2 k ( old )   b 2 k ( new )    b 2 k ( old )

on Figure 4.

2. Rules of FL

B. Implementation of Fuzzy Logic on Training Of Rules are the key of FL operation. These rules connect CBPNN

the input variable and the output of FL. Fuzzy rules used CBPNN is firstly initiated with simplest structure,

in this study are described on Table 1. commonly without hidden unit. In this study, we design

3. Memberships function of output one neuron in the hidden layer for the first arrangement.

Output of FL is used to modify number of New Units This initial configuration is trained to recognize data

(NU). To simplify the case, we implement singleton pattern provided at limited iteration. If the network fails

function as membership function of the fuzzy output. to recognize the data set, some neurons are added to the

There are five membership functions used, namely VS hidden layer. Then the new network architecture is

for very small, S for small, M for medium, B for big, and trained with the previous pattern. Training process is

VB for very big. Value of each function can be shown on continued until we get suitable network for the training

Figure 5.

data set. Training method by modifying structure is

4. Inference

typical algorithm of CBPNN. Main part of fuzzy logic operation is inference. In this As already discussed, changing of network structure

study, we use min method, formulated as follow: effects to the computing process. The number of new

  min   MSE ,  dMSE  (18)

units added has to be tuned carefully to simplify Output of this logic (z) is calculated with centroid function, which can be formulated as follow:

   z D (19)

139 where D is decision index. Then NU can be formulated

IPTEK, The Journal for Technology and Science, Vol. 22, No. 3, August 2011

where VS is for very small (z ≤ 0.2), S is for small (0.2 < as follows:

z ≤ 0.4), M is for medium (0.4 < z ≤ 0.6), B is for big

NU  f (z ) (20)

(0.6 < z ≤ 0.8), and VB is for very big (0.8 < z).

C. Experimental Design

D. Training Data Set

FL method is implemented in training of CBPNN. There are two structures of the network investigated in

Training data sets used in this study were aroma this study, i.e. One Hidden Layer of CBPNN (OHL-

patterns of tobacco. There were five samples of tobacco, CBPNN) and Double Hidden Layer of CBPNN (DHL-

captured by combination of 7-gas sensors, namely CBPNN). Both networks are commonly used on an

MQ135, MQ136, MQ137, MQ138, MQ3, TGS2620, and electronic nose system. The result will be compared to

TGS822. Average pattern of these samples are presented the fixed method implemented on the same network for

on Figure 6. We prepared five training data sets with the similar training data set.

different combination of these sensors. Data 1 Implementation of both methods can be detailed as

represented 5 patterns of tobacco captured by follows. Fixed method on OHL-CBPNN is conducted by

combination of 3 sensors. Data 2 was pattern of the adding fixed unit number. There are five variations of

sample odor with 4 sensors, data 3 with 5 sensors, data 4 unit number studied, i.e. 1 unit, 2 units, 3 units, 4 units,

with 6 sensors, and the last data was captured by and 5 units. Batch iteration is set up at 300 epochs,

combination of seven sensors.

related to the previous study conducted by Radi et.al. (2011). While, NU is arranged in the following section:

III. R ESULTS AND D ISCUSSION  NU (unit) : 0 Both CBPNNs were trained to recognize all data sets.

VS   Batch iteration : 300 Training was conducted for both neural networks for five

 NU (unit) : 1 repetitions for each data set. Both of them were trained S  with learning rate of 0.7, momentum of 0.2, and

 Batch iteration : 300 maximum error threshold of 0.001. This study conducted  NU (unit) : 2

M  on laptop with specification of Intel Celeron M processor  Batch iteration : 300 440 (1.86 GHz, 533 MHz FSB, 1 MB L2 cache) with

B  Batch iteration : 300  Effectiveness of both algorithms (fixed and FL) have

 NU (unit) : 3 memory of 1.5 MB DDR2.

been evaluated. There are three components used for this

 NU (unit) : 4 VB 

evaluation, i.e. size of final network, number of training  Batch iteration : 300 iteration, and time of network learning. The network size

where VS is for very small (z ≤ 0.2), S is for small (0.2 < in general MLP network involves the number of layers z ≤ 0.4), M is for medium (0.4 < z ≤ 0.6), B is for big

and the number of hidden-units per layer. Both of them (0.6 < z ≤0.8), and VB is for v ry big (0.8 < z).

reflect the complexity of the network. Number of units in For DHL-CBPNN, we implemented the fixed method

each layer presents the numbers of connections in the by varying number of new units added on the first and network. Number of layers indicates the stage of

second hidden layer simultaneously. By equal internal computing process when it trained and used. Normally, iteration, number of new units is managed with range of

the number of units in the input and output layers can be

1 to 5 units gradually. Whereas, implementation of FL determined by the dimensionality of the problem or method on DHL-CBPNN training is performed by

training data set, so the number cannot be set up when modifying NU based on the following arrangement:

the network trained. Studying neural network which have determined configuration layer, observation of the  NU (1 st hidden layer, unit) : 0 network size is focused on the number of neurons in the

hidden layer. In this study, the network size is

VS

 nd NU (2 hidden layer, unit) : 0

represented by the number of neurons in the hidden   Batch iteration : 300 layer. Time for training and number of iterations are used

 NU (1 st hidden layer, unit) : 1 to represent the load of learning process. Result of this study is detailed as follows:

S   NU (2 nd hidden layer, unit) : 1

A. Training of OHL-CBPNN

 Batch iteration : 300  OHL-CBPNN was prepared and trained for all data

 NU (1 st hidden layer, unit) : 2 sets. Observation involves size of final network, training 

iteration, and training time. The result of this study is

nd

 presented on Table 2. This result written on the table is  Batch iteration : 300 

M  NU (2 hidden layer, unit) : 1

average value of five experimental data. From the Table 2, we clearly know that the smaller

 NU (1 st hidden layer, unit) : 3 size of final network is generated by fixed 1 and FL

B   NU (2 nd hidden layer, unit) : 2 methods. Based on size consideration, result showed that

  Batch iteration : 300  adding more unit numbers simultaneously tends to generate bigger size of the final network. Comparison of

 NU (1 st hidden layer, unit) : 4 both algorithms is clearly understood on Figure 7.

VB NU (2  nd hidden layer, unit) : 3 It follows from Figure 7 that size of final OHL-

   Batch iteration : 300

CBPNN is determined by structure development pattern

140 IPTEK, The Journal for Technology and Science, Vol. 22, No. 3, August 2011

smaller structure is generated by applying smaller new iteration. Based on the results, FL method and big new unit number. The study shows that fixed 1 method can

unit number on fixed method can be recommended. construct final network with smaller size. Compare to

Training time of DHL-CBPNN for both algorithms fixed 1 method, FL method also generate small network.

can be clearly evaluated from Figure 12. The figure From this result, both methods can be recommended to

shows that training time of the network for all data train the OHL-CBPNN.

patterns was determined by implementation of training Beside size of network, effectiveness of the proposed

methods. From the figure, we can conclude that FL method is evaluated from number of training iteration.

method give the best performance due to minimum time Comparison of iteration number for both methods can be

of convergence.

shown on Figure 8. Figure 8 presents that application of Based on the three parameters studied before, we both methods does not significantly affect to the number

consider that FL method can be used as new strategy on of iteration. The biggest influence is only seen on the

training of DHL-CBPNN. Due to the same result on OHL-CBPNN with fixed 1 method. Result shows that

training of OHL-CBPNN, we sure that this method can this method tends to increase number of training

be implemented on training of general CBPNN. iteration. This is understandable because this method

C. Analyze of Both Algorithms on Training of OHL- cause the structure changed slowly. Based on these

CBPNN and DHL-CBPNN

results, the FL and fixed method with smaller number of iteration may be selected.

In this section, we focus on analyzing performance of Time of network training is also used to evaluate

FL method compared to fixed methods. In order to performance of the proposed algorithm. Training time of

evaluate effectiveness of FL method, we only compare the OHL-CBPNN with various data sets is presented on

the method to the best and average performance of fixed Figure 9. With this result, we know that number of new

method. Best performance in this section is defined as unit added in the fixed method determines speed of the

method, which is able to generate small size of final network convergence. Fixed method with small number

network with minimum iteration and time. Percentages of units tends to accelerate its training process. FL

of network size, training iteration, and training time are method also shows good performance. This method is

discussed. Network size comparison of FL method and able to minimize load of computation and results

related fixed method is presented on Table 4. network with high speed of convergence. Compare to the

Colum A in the Table 4 is smaller network size for all studied method, FL method can be recommended.

fixed methods implemented in training of CBPNN. Fixed Based on the previous parameters, we can conclude

1 method showed the best performance due to its ability that FL methods applied on OHL-CBPNN training

to generate smallest network for all data patterns. We can shows the best performance. This method can be used to

evaluate the performances of both algorithms by optimize training of the network. This is due to results of

comparing the data. Result shows that FL method training time, training iteration, and network size of the

implemented on OHL-CBPNN was able to generate network.

smaller network compare to fixed 1 method (94.98 %). Better performance of FL method is showed if we

B. Training of DHL-CBPNN compare it with the average size of final network of DHL-CBPNN is trained to recognize the same data

various fixed methods. FL method is able to produce less sets. The similar manner is applied in this experiment in

than a half size of final network with fixed method. accordance with the previous experimental design. In

Similar performance of FL method is also presented on this study, we also focus to evaluate 3 parameters, i.e.

training of DHL-CBPNN. FL method is effective to size of network, training time, and iteration. Study result

minimize final network size with average of 91.47 % shows on Table 3.

compared to the best fixed method and 48.71 % Based on Table 3, we can analyze that both, proposed

compared to average fixed method implemented. This method and comparison method, show similar

result showed that the proposed method is effective to performance with the previous study. Smaller size of

find minimum network architecture. network is also generated by applying of FL and Fixed 1

Beside network size, performance evaluation of both methods. Both FL and fixed 1 method may be chosen to

methods can be evaluated from training iteration. generate small CBPNN. Evaluation of network size for

Comparison of training iteration between FL method and both applied algorithm for all variations of data sets is

related fixed method is presented on Table 5. It shown presented on Figure 10. From this result, we consider

that fixed method (the best and average) has better that both algorithms show the similar characteristic.

performance than FL method for OHL-CBPNN and Observations of training iteration of DHL-CBPNN are

DHL-CBPNN. CBPNN trained by fixed method is able presented in Figure 11. The graph shows that variations

to convergence with less training iteration than FL of the fixed method employed are not exhibited

method implemented. It can be understood from the significantly affects to the network training iterations.

design of FL discussed in the previous section. The biggest influence can be understood on the

Effectiveness of both methods can also be evaluated implementation of the fixed 1 method. Applying of this

by comparing time of training process. Training time method for DHL-CBPNN tends to increase number of

assessment of both methods for OHL-CBPNN and DHL- training iteration. The result shows that the network

CBPNN are detailed on Table 6. The table shows that convergent with higher number of training iterations than

comparing to fixed method; FL method has better the other methods implemented. This is understandable

performance due to its capability to minimize time of because the structure development was too slow and

convergence. Based on the study result, training of OHL- each change of structure requires considerable training

CBPNN with FL method is able to convergence with

141 only 43.96 % of training time used by average fixed

IPTEK, The Journal for Technology and Science, Vol. 22, No. 3, August 2011

where I is number of training iteration and t is total time method and 14.78 % less than the best fixed method.

needed to train the network. Number of f shows the load The similar result is showed on training of DHL-

of computing process.

CBPNN. In this case, the proposed method also presents Comparison of f for both methods implemented on better performance. The study result shows that the

both CBPNN is presented on Table 7. It shown that network trained using FL method was able to recognize

proposed method has better performance than proper the data pattern more than two times faster than the

fixed method. Value of iteration frequency of CBPNN average performance of fixed methods studied. The data

trained by FL method is bigger than the similar neural shows that DHL-CBPNN with FL method is able to

network when fixed method is implemented. It means convergence with only 39.66 % of average time needed

that FL method tends to reduce load of computational when the fixed method applied and 73.88 % of time

process significantly. Implementation of FL shows that needed by the best fixed method implemented.

load of processor is lighter than the other methods Based on the study result, we can conclude that FL

implemented on training of CBPNN. method is suitable to train not only OHL-CBPNN but

These results prove that the addition of the algorithm also DHL-CBPNN. Although this method tends to add a

does not necessarily aggravate the computational load, few number of training iteration, FL method is able to

but in this case, the addition of FL algorithm can minimize the final network size and time of training.

decrease load the network training. Thus, we agree that

D. Computational Load of Training combining FL algorithm on CBPNN does not increase training load, but otherwise, it tends to drop off training

Computational load is mainly used to evaluate

load.

effectiveness of an algorithm or method. Computational load is closely related to the algorithm design. A

IV. C ONCLUSION program, which contains a lot of multiplier, is usually

difficult to be executed. In this case, the algorithm has This paper presented training of CBPNN by high computational load. We also can predict the

implementing FL method and fixed method. The neural computational load by evaluating difficulty level of the

network was used to recognize aroma pattern on an algorithm. Besides, computational load can also be

electronic nose system. The result shows that FL method valued from the computer load when the algorithm is

applied on OHL-CBPNN and DHL-CBPNN is able to running. Light algorithm is easy to be computed. In this

generate small final neural network. Besides that, FL section, we only focus on the effect showed on computer

method on both CBPNNs can minimize training time. FL when the both methods implemented on the OHL-

method is also possible to reduce load of computational CBPNN and DHL-CBPNN.

training. Combining FL method and CBPNN is possible In the previous section, we captured training iteration

to increase performance of the network. Therefore, this and training time data. Both are used to calculate

method can be recommended to arrange structure iteration frequency (f) of training of the neural network.

development of general CBPNN. Frequency of iteration means number of iteration for

each second, which can be formulated as follows: I

pattern Target

x nInput

t nOutput

b1 b2 layer

1 Hidden

Hidden

layer Output

bias

bias

Figure 1. Structure of one hidden layer of BPNN

142 IPTEK, The Journal for Technology and Science, Vol. 22, No. 3, August 2011

Start

Set of Training Data Network Training Parameter

Design of Initial Network

Network Training

Save Final Network w1 w2 w3

Design of New Network

Accepted Error

layer Output

hidden

Bias

output Bias

No FL Method

MSE dMSE

Figure 2. FL method implemented on CBPNN

10xErrLim 100xErrLim 1000xErrLim nOutput JumData -nOutput -1000xErrLim -10xErrLim

Figure 3. Memberships function of MSE Figure 4. Memberships function of dMSE

VS S M B VB

Figure 5. Membership function of FL output

IPTEK, The Journal for Technology and Science, Vol. 22, No. 3, August 2011

1 2 3 4 5 1 2 3 4 5 M Q135

M Q136

TGS2620

M Q135

M Q137

M Q136 TGS2620

M Q135 M Q3 M Q137

M Q136

(e) Figure 6. Patterns of training data set (a) data 1, (b) data 2, (c) data 3, (d) data 4, (e) data 5

Figure 7. Final network size of OHL-CBPNN with fixed and FL Figure 8. Training iteration of OHL-CBPNN with fixed and FL methods for all training data sets

methods for all training data sets

Figure 9. Training time of OHL-CBPNN with fixed and FL methods Figure 10. Final network size of DHL-CBPNN with fixed and FL for all training data sets

methods for all training data sets

144 IPTEK, The Journal for Technology and Science, Vol. 22, No. 3, August 2011

Training Iteration Training Time

p E 8,000

Fixed 5 Fixed 4

Fixed 2 Fixed 1 FL

Data 1 Data 2

Data 3 Data 4 Data 5

Figure 11. Training Iteration of DHL-CBPNN with fixed and FL Figure 12. Training time of OHL-CBPNN with fixed and FL methods methods for all training data sets

for all training data sets

T ABLE 1. R ULE OF FL

T ABLE 2. T ABLE 3. C OMPARISON OF T WO M ETHODS IN T RAINING OF OHL-CBPNN

C OMPARISON OF T WO M ETHODS IN T RAINING OF DHL-CBPNN Method

Network size Iteration Training Data Set

Training Network size

(units)* (epoch) Time (s) Data 1

(units)*

(epoch)

Time (s)

Data Set

49 5878 9.212 *) number of hidden neurons of the final networks

*) number of hidden neurons of the final networks

IPTEK, The Journal for Technology and Science, Vol. 22, No. 3, August 2011

T ABLE 4. F INAL NETWORK S IZE C OMPARISON OF FL AND R ELATED F IXED M ETHOD

Data set

A: Final network size of the best fixed method performance B: Average of final network size of fixed methods used

T ABLE 5. T RAINING I TERATION C OMPARISON OF FL AND R ELATED F IXED M ETHOD

Data Set

A: Training iteration of the best fixed method performance B: Average of training iteration of fixed methods used

T ABLE 6. T RAINING TIME COMPARISON OF FL AND R ELATED F IXED M ETHOD

Data Set

A: Training time of the best fixed method performance B: Average of training time of fixed methods used

146 IPTEK, The Journal for Technology and Science, Vol. 22, No. 3, August 2011

T ABLE 7. F REQUENCY ITERATION C OMPARISON OF FL AND R ELATED F IXED M ETHOD

Data Set

A: Iteration frequency of the best fixed method performance B: Average of f of fixed methods used

B. Lan, “Comparison of Algorithms for an El ctronic Nos in Id ntifying Liquors”, [1] T. C. Pearce, S. S. Schiffmen, H. T. Nagle, and J. W. Gardner,

R EFERENCES [5] Z. B. Shi, T. Yu, Q. Zhao, Y. Li, and Y.

Journal of Bionic Engineering, vol. 5, pp. 253 –257, 2008. Handbook of Machine Olfaction . New York: WILEY-VCH

M. L htokangas, “Mod ling ith constructiv backpropagation”, Verlag GmbH & Co. KgaA., 2003.

[6]

[2] H. Yu, J. Wang, C. Yao, H. Zhang, and –716, 1999.

Y. Yu, “Quality grad

Neural Networks , vol. 21, pp.707

M. L htokangas, “Modifi d constructiv backpropagation for identification of green tea using E- nos by CA and ANN”, LWT,

[7]

r gr ssion”, Neurocomputing, vol. 35, pp. 113-122, 2000. vol. 41, pp. 1268 –1273, 2008.

. S tiono, “Finding minimal n ural networks for business [3] S. Panigrahi, S. Balasubramanian, H. Gu, C. Lague, and M.

[8]

int llig nc application”. presented at IFSA-AFSS International March llo, “N ural-network-integrated electronic nose system for

id ntification of spoil d b f”, LWT, vol. 39, pp. 135–145, 2005,. Conference, Surabaya-Bali Indonesia, 2011. [9] T. S. Widodo, Sistem Neurofuzzy. Yogyakarta: Graha Ilmu, 2005.

[4] M. H. Purnomo and A. Kurniawan, Supervised Neural Networks [10] Radi, M. Rivai,M. H. Purnomo, “Multi-thread constructive back dan Aplikasinya . Yogyakarta: Graha Ilmu, 2006.

propagation n ural n t ork for aroma patt rn classification”, in Proc. 2011 IFSA-AFSS International Conference .