Publication Repository endang icsiit 2012
Facial Emotional Expressions Synthesis using Radial
Basis Function Network
Endang Setyati
Sekolah Tinggi Teknik Surabaya
Informatic Department
Surabaya, Indonesia
+62315027920
[email protected]
Mauridhi Hery Purnomo
Yoyon K. Suprapto
Institut Teknologi Sepuluh Nopember Institut Teknologi Sepuluh Nopember
Electrical Engineering Department
Electrical Engineering Department
Surabaya, Indonesia
Surabaya, Indonesia
+62315994251-54
+62315994251-54
[email protected]
ABSTRACT
Emotion recognition through the computer-based of facial
expression has been an active area of research in the literature for
a long time. In this paper we develop a facial emotional
expression synthesis (FEES) techniques based on the facial
features extracted from facial characteristic points (FCPs) in
frontal image sequences. In order to synthesize such facial
expression, FCPs can be used as landmarks. These FCPs are
feature points that represent significant movements during the
generation of expression. We try to simulate people’s expressions
artificially using Radial Basis Function Network (RBFN. The
experimental result from classifier, success rate was about 91.57%
using RBFN classifiers.
Keywords
Facial emotional expression synthesis; facial features extracted;
facial characteristic points; radial basis function network
1. INTRODUCTION
Face plays an essential role in interpersonal communication.
Facial expressions play an important role in non-verbal
communications. Many applications for teleconferencing, human
computer interface and computer animation require realistic
reproduction of facial expressions.
The interest in computational models of emotion and emotional
expressions has been steadily growing in the agent research
community. Several psychologists have acknowledged the role of
emotions in intelligence [1].
In 1971, Ekman and Friesen [2] postulated six primary emotions
that each posses a distinctive content together with a unique facial
expression. These prototypic emotional displays are also referred
to as basic emotions. They seem to be universal across human
ethnicities and cultures and comprise happiness, sadness, fear,
disgust, surprise, and anger. The method of recognizing the 6
universal facial expressions using neural network is discussed.
Most emotion theorist [3] emphasize the involuntary nature of
emotional experience, ignoring those instances when people
choose to generate an emotion through reminiscence or by
adopting the physical actions associated with a particular emotion
(e.g., speaking more softly to deintesify anger or smiling to
generate enjoyment).
[4] believed there exist a relationship between facial expression
and emotional state. There is a small set of basic emotions that
can be expressed distinctively from one another by facial
expressions. For instance, when people are angry they frown and
when they are happy they smile.
[email protected]
Facial expressions [5] are the result of facial muscle actions
which are triggered by the nerve impulses generated by emotions.
The muscle actions cause the movement and deformation of facial
skin and facial features such as eyes, mouth, and nose. We can use
optical flow to estimate facial muscle actions which can then be
used in recognizing a facial expression.
[6] apply a 234x50x6 back-propagation neural network for
classification of expression into one of six basic emotion
categories and their strength. Then they generate the facial
position information and it is input into the input units of neural
network, networ learning is done by back-propagation algorithm
and recognition test is carried out. For six basic facial expressions,
the correct recognition ratio is found to about 90%.
Suppose we are given a 2D grayscale expressionless face image of
a person, i.e., a face image without any expression of emotion [7],
how can we synthesize different expression of that person? One of
the ways to synthesize facial expressions is to find the
approximate displacement of prominent facial feature points.
Moreover, [8] often cannot obtain accurate facial landmark
displacement information due to inherently inaccurate input data.
This is because: (1) it is hard to generate a set of standardized
expressions, e.g., each person may smile differently, (2) it is hard
to produce accurately the precise degree of a particular
expression, e.g., how to generate a 20% smile?, and (3) it is
difficult to mix various facial expressions, e.g., how to gesture a
happy and sad face?
[9] in synthesizing facial expressions include texture mapping
approach to 3D facial image synthesis and [10] use of 3D model
of facial muscles and tissues. An alternative approach has been
investigated by [11], which demonstrated the use of RBF in
interpolating the anchor points for 2D image warping, which can
be applied to synthesize facial expressions. However, it is does
not provide a mechanism to determine the appropriate destination
of the anchor points for each particular facial expression.
Human face has several unambiguous features: eyebrows, eyes,
mouth, nose, and face outline. [13] extract three main features:
eyebrows, eyes, and mouth. After extracting these feature, they
are able to get 30 points of the FCPs.
[14] proposed a hierarchical model of RBFN to classify and to
recognize facial expressions. This approach utilizes Principal
Component Analysis as the feature extraction process from static
images. This research is to develop a more efficient system to
discriminate 7 facial expressions. They achieved the correct
classification rate above 98.4% which is overwhelmingly
distinguished compared to other approaches.
[15] develop a facial expression recognition system, based on the
facial features extracted from FCPs in frontal image sequences.
Selected facial feature points were automatically tracked using a
cross-correlation based optical flow, and extracted feature vectors
were used to classify expressions, using RBFN and FIS. Success
rates were about 91.6% using RBF and 89.1% using FIS
classifiers.
In this paper, we proposed two systems for classifying of the
facial expressions from The Japanese Female Facial Expression
(JAFFE) Database [17]. 7 features extracted from 30 feature
points and from a feature vector for each expression. These
feature vectors were used to training a RBFN classifier to classify
input feature vectors into one of the six basic emotions.
2. INFORMATION OF FACIAL
EXPRESSION
2.1 Facial Characteristic Points
FCPs carry the information about the position and shape of these
three features. According to the study of the Ekman and Friesen
[12], almost all facial expressions of human face are described by
combination of 46 basic movements of facial muscles and these
basic movements are called Action Units (AUs). 30 AUs are
directly associated with movement of eyes, eyebrows, and mouth.
That is why the information expressing movement of eyes,
eyebrows, and mouth is desirable for machine recognition of
facial expressions. The information of each of 6 basic facial
expressions is obtained by subtracting the FCPs coordinates of
normal facial expression from those of facial expressions.
In [6] are confined in these three components and then determine
FCPs which are representative of the boundary between these
components and skin. A set of 30 facial landmarks located near
the eyes, eyebrows, and the mouth are defined as the FCPs. These
points are shown in Figure 1.
FCPs are the points in a face which can represent facial
characteristics. Figure 1 shows the FCPs and a i is a vector
expressing the coordinate of FCPs. ai is described as [7], [8]
a (x , y ), i 1,2,..., 30
i
i i
(1)
To normalize the face image, we introduce a quantity, base, which
is not varied for each of facial expressions, expressed as the origin
of which is assigned at the top of nose, is taken for the coordinate
of FCPs in this study. The information of these 30 FCPs are input
to a computer by using a mouse device.
In Figure 1, Xb-Yb coordinate system shows the absolute
coordinate system and X’-Y’ coordinate system is used for the
new coordinate of FCPs. The origin (originx, originy) of X’-Y’
coordinate system is chosen at the point of length base downward
of the mid point between the left and right eyes [5], [6], [7], [8].
2
2
base (X - X ) (Y - Y )
b2
b1
b2 b1
(2)
Then we Introduce , which is the inclination of face with
respect to the horizontal line, and defined as
θ tan 1
(Y - Y )
b2 b1
-X )
b2
b1
(X
Figure 1. Facial Characteristic Points [6], [7], [8]
The X and Y coordinates, (X0, Y0), of the mid point between the
left and right eyes are described as
X
0
Y X
X X
b2
b1
b2 and Y b1
0
2
2
(4)
The origin of the new coordinate system is calculated as
originx=X0+base*sin and originy=Y0 – base*cos
(5)
The coordinate (Xbi,Ybi) of a FCP ai is transformed into X-Y
coordinates system, (X i,Yi) by subtracting the origin coordinates
and then rotating an angle of orientation , and their relationships
are given by
Xi = Xbi - originx and Yi = Ybi - originy
(6)
Xir = Xi cos + Yi sin and Yir = -Xi sin + Yi cos
(7)
We normalize the input face input face image by dividing
(Xir,Yir) above by value base to compensate the distance effect
between the client face and the camera and the size of the client
face, given by
Y
X
X' ir and Y' ir
i base
i base
i = 1, 2, ..., 30
(8)
2.2 Feature Extraction from Feature Points
Seven features were extracted from the feature vector for each
expression, and were used to classify that expression to one of the
six basic emotions, using RBFN. Extracted features are as follows
[6], [8]:
Openness of Eyes:
oe
(Y - Y ) (Y - X )
n7 n5
n8
n6
2
(9)
Width of Eyes:
(3)
we
(X
n1
-X
n3
) (X
2
n4
-X
n2
)
(10)
from 10 Japanese female models. In Figure 3 are Example of
Facial Emotional Expressions of JAFFE Database [17].
Height of Eyebrows 1:
he1
(Y - Y
) (Y - X
)
0 n19
0
n20
2
(11)
Height of Eyebrows 2:
he2
(Y - Y
) (Y - Y
)
0
n17
0
n18
2
Angry
(12)
Surprise
Netral
Width of Mouth:
wm X
n24
X
(13)
n23
Disgust
Openness of Mouth:
Sad
Fear
om Y
Y
n26
n25
Nose Tip-Lip Corners Distance:
nl
(X - X
) (X - X
)
0
n23
0
n24
2
Happy
(14)
Figure 3. Example of Facial Emotional Expressions of
JAFFE Database
(15)
Figure 2 are example face image of feature extraction from feature
points. Standardization is needed to find out the relative
displacement of the facial landmarks from their normal position
[17].
The six basic emotions defined by [16] can be associated with a
set of facial expressions. In Table 1 shows textual description of
facial expressions as representations of basic emotions.
Table 1. Facial Expressions of Basic Emotions [16]
No
Basic
Emotion
1
Happy
The eyebrows are relaxed. The mouth is open
and the mouth corners pulled back toward the
ears.
2
Sad
The inner eyebrows are bent upward. The
eyes are slightly closed. The mouth is
relaxed.
3
Fear
The eyebrows are raised and pulled together.
The inner eyebrows are bent upward. The
eyes are tense and alert.
4
Angry
The inner eyebrows are pulled downward and
together. The eyes are wide open. The lips are
pressed against each other or opened to
expose the teeth.
5
Surprise
The eyebrows are raised. The upper eyelids
are wide open, the lower relaxed. The jaw is
opened.
6
Disgust
The eyebrows and eyelids are relaxed. The
upper lip is raised and curled, often
symmetrically.
Figure 2. (a) Example of Feature Extraction, (b)
Example of FCPs
3. FACIAL EMOTIONAL EXPRESSIONS
3.1 Facial Emotional Expression Synthesis
The synthesis of facial emotional expressions can be seen as a
reverse process of facial emotional expressions recognition [7]. In
recognition, we present the necessary information (the movements
of the landmarks) so as to classify for a particular facial
expression (an emotional label) in the order happy, sad, angry,
fear, suprised and disgusted.
But what if we reverse the question: given a particular facial
expression, can you tell what the necessary movements of the
landmarks are? This indeed can be seen as a reverse process to
which emotion labels are used and the outputs are the movements
of the landmarks.
The JAFFE Database have 213 face images of 7 facial expressions
(6 basic facial expressions and 1 neutral facial expression) taken
Textual Description of facial Expressions
3.2 Radial Basis Function Network
The basic idea is to find out the spatial differences between the
FCPs of the normal face and that of the expressive face. Thus,
differences of those 30 pairs of position information will
constitute the 60 inputs to the two-layered neural network as
Figure 4.
G(||d-1||)
w
G(||d-1||)
d
decided by width sigma such that a smooth interpolation over the
input space is allowed. The whole architecture is therefore fixed
by determing the hidden layer and the weights between the middle
and the output layers.
o
The number of input layer units must be equal to 7, equal to the
number of extracted features, and that of output layers is 6, which
corresponds to six kinds of facial expressions. The network
training is carried out by back propagation algorithm.
G(||d-1||)
4. EXPERIMENTAL RESULT
Figure 4. RBFN Structure [7]
The basic principle of synthesizing facial expressions is to find
out the necessary relative spatial shift of the FCPs for each
expressions of emotion. So what is initially the input to the neural
network in recognition will become the output in synthesis and
vice versa [7]. RBF have proven to be an effective tool in
interpolating data in multidimensional spaces.
The RBFN is ideal for interpolation since it uses a radial basis
function, for example Gaussian function, for smoothing out and
predict missing and inaccurate inputs [8].
We would consider interpolating functions of the form:
m
F d ω g d μ , d R n , k 1,..., n'
j
k
jk
j 1
Table 2. RBF Classifier Test Result
Degree of Expression
Basic
Emotion
Ha
Sa
Fe
An
Su
Di
Result
%
Happy
4.73
1.32
1.25
1.27
1.28
1.26
94.60
Sad
1.30
4.65
1.93
1.90
1.00
2.52
93.00
Fear
1.21
2.85
4.47
2.17
3.49
3.17
89.40
Angry
1.37
2.13
1.78
4.64
1.59
2.91
92.80
Surprise
1.81
1.49
2.54
1.62
4.24
1.00
84.80
Disgust
1.18
2.43
2.41
2.66
2.01
4.74
94.80
Average
91.57
(17)
5. CONCLUSION
where d is the input vector, is a set of weights and is the width
of the RBF.
Hence, the determination of the nonlinear map F(d) has been
reduced to the problem of solving the following set of linear
equations for the coefficients j,
f1k A11 A1m ω1k
. , k=1,2, ..., n’
f A
mk m1 A mm ω mk
Learning rate used is 0.01. The process is divided into two parts,
namely the process of calculating the hidden layer by using kmeans clustering and the training process the input. The total time
for facial feature extraction, pre-processing, neural network
calculation takes less than 15 seconds.
(16)
where . denotes the usual Euclidean norm on Rn and jRn’,
j=1, 2, ..., m denotes the centers of the radial-basis functions
which are given as the known data points. Often, the g(.) is the
normalized Gaussian activation function defined as
exp (d - μ ) 2 /2σ 2
j
j
g( d)
2
2
k exp (d - μ ) /2σ
k
k
A set of 256 x 256 grayscale images are used in our experiment.
In the RBFN classifier for FEES, we used 6 input layer, 10 hidden
layer, and 60 output in 60 sample. We just did a bit of input
samples and hidden units to be more easily studied. The more
input samples and the number of hidden units, then the result will
be better.
In this research we presented two systems for classifying of the
facial expressions from JAFFE Database. In the RBFN classifier,
7 feature extracted from 30 feature points were used as training
and test sequences. The trained RBFN was tested by features that
not used in training and we have obtained a high result rate of
91.57%.
6. REFERENCES
(18)
where A g d μ , i, j 1, 2, ..., m.
ij
j
RBFN is class of single hidden layer feedforward networks where
the activation functions for hidden units are defined as radially
symmetric basis functions phi such as the Gaussian function. The
fraction of overlap between each hidden unit and its neighbors is
[1]
M.S. El-Nasr, “Modeling Emotion Dynamics in Intelligent
Agents,” M.S. Thesis, May 1988.
[2]
Paul Ekman and W.V. Friesen, “Facial Action Coding
System (FACS),” Consulting Psychologists Press, Inc., 1978.
[3]
Paul Ekman, “Facial Expressions of Emotion: New Findings,
New Questions,” American Psychological Society, Vol. 3,
No. 1, pp. 34-38, January 1992.
[4]
Paul Ekman, “Facial Expressions and Emotion,” American
Psychological, Vol. 48, pp. 384-392, 1993.
[5]
Hiroshi Kobayashi and Fumio Hara, “The Recognition of
Basic Facial Expressions by Neural Network,” in
[6]
[7]
International Joint Conference on Neural Network, pp. 460466, 1991.
2009 American Psychological Association, Vol. 96, No. 1,
pp. 1-10, 2009.
Hiroshi Kobayashi and Fumio Hara, “Recognition of Six
Basic Facial Expressions and Their Strength by Neural
Network,” IEEE International Workshop on Robot and
Human Communication, pp. 381-386, 1992.
[21] Keith Anderson and Peter W. McOwan, “A Real-Time
Mary Y.Y. Leung, Hung Yen Hui and Irwin King, “Facial
Expression Synthesis by Radial Basis Function Network and
Image Warping,” In IEEE International Conference on
Neural Networks, Vol III, pp. 1400-1405, Washington D.C.,
1996. IEEE Computer Society.
[8]
I. King and H.T. Hou, “Radial Basis Function for Facial
Expression Synthesis,” In IEEE International Conference on
Neural Networks, 1996.
[9]
J. Yau and A. Duffy, “A Texture mapping approach to 3D
facial image synthesis, 1988.
[10] Fumio Hara and Hiroshi Kobayashi, “Computer Graphic for
Expressing Robot-Artificial Emotions,” IEEE International
Workshop on Robot and Human Communication, pp. 381386, 1992.
[11] D.R. Nur Arad, Nira Dyn and Y. Yeshurun. Image Warping
by Radial Basis Functions: Application to facial Expressions.
CVGIP: Graphical Models and Image Processing, 56, No.
2:161-172, 1994.
[12] P. Ekman and W.V. Friesen, “Unmasking the Face,”
Consulting Psychologists Press, Inc., 1975.
[13] Jyh-Yeong Chang and Jia-Lin Chen, “A Facial Expression
Recognition System Using Neural Networks,” IEEE, pp.
3511-3516, 1999.
Lin and Jam Chen, “Facial Expressions
Classification with Hierarchical Radial Basis Function
Networks,” IEEE, pp. 1202-1207, 1999.
[14] Daw-Tung
[15] Hadi
Seyedarabi, Ali Aghagolzadeh, and Sohrab
Khanmohammadi, “Recognition of Basic Facial Expressions
by Feature-Points Tracking using RBF Neural Network and
Fuzzy Inference System,” IEEE International Conference on
Multimedia and Expo (ICME), pp. 1219-1222, 2004.
[16] Surya Sumpeno, Mochamad Hariadi, and Mauridhi Hery
Purnomo, “Facial Emotional Expressions of Life-like
Character Based on Text Classifier and Fuzzy Logic,” in
IAENG International Journal of Computer Science, 38:2,
IJCS_38_2_04 [online], May 2011.
[17] The Japanese Female Facial Expression (JAFFE) Database.
Available: [Online] http://www.ksrl.org/jaffe.html.
[18] Muid Mufti, Assia Khanam, “Fuzzy Rule Based Facial
Expression Recognition,” IEEE Computer Society,
International Conference on Computational Intelligence for
Modelling Control and Automation, and International
Conference on Intelligent Agents, Web Technologies and
Internet Commerce (CIMCA-IAWTIC’06), 2006.
[19] Hadi
Seyedarabi, Ali Aghagolzadeh, and Sohrab
Khanmohammadi, “Facial Expression Recognition from
Static Images using RBF Neural Networks and Fuzzy
Logic,” 2nd Iranian Conference on Machine Vision and
Image Processing (MVIP 2003), Tehran, 2003.
[20] David Matsumoto and Bob Willingham, “Spontaneous Facial
Expressions of Congenitally and Noncongenitally Blind
Individuals,” Journal of Personality and Social Psychology,
Automated System for the Recognition of Human Facial
Expressions,” IEEE Transactions on Systems, Man and
Cybernetics-Part B: Cybernetics, Vol. 36, No. 1, February
2006.
[22] Kwang-Eun Ko and Kwee-Bo Sim, “Development of a
Facial Emotion Recognition Method based on combining
AAM with DBN”, IEEE Computer Society, 2010
International Conference on Cyberworlds, pp. 87-91, October
2010.
[23] Kwang-Eun Ko and Kwee-Bo Sim, “Facial Emotion
Recognition using a Combining AAM with DBN”,
International Conference on Control, Automation and
Systems 2010, ICROS in KINTEX, Gyeonggi-do, Korea, pp.
1436-1439, October 27-30, 2010.
[24] Marian Stewart Bartlett, Gwen Littlewort, Claudia Lainscsek,
Ian Fasel, Javier Movellan, “Machine Learning Methods for
Fully Automatic Recognition of Facial Expressions and
Facial Actions,” IEEE International Conference on Systems,
Man and Cybernetics, pp. 592-597, 2004.
[25] R. Ravi and Mahima S, “Study of the Changing Trends in
Facial Expression Recognition,” International Journal of
Computer Application (0975-8887) Vol. 21, No. 5, pp. 1016, May 2011.
[26] Spiros V. Ioannou, Amarylis T. Raouzaiou, Vasilis A.
Tzouvaras, Theofilos P. Mailis, Kostas C. Karpouzies,
Stefanos D. Kollias, “Emotion Recognition through Facial
Expression Analysis based on a Neurofuzzy Network,”
Elsevier
Ltd.
All
rights
reserved.
doi:10.1016/j.neunet.2005.03.004. Neural Networks 18
(2005), pp. 423-435, 2005, Available online at
www.sciencedirect.com and
[27] V. Radha, and N. Nallammal, “Neural Network Based Face
Recognition Using RBFN Classifier,” Proceedings of the
World Congress on Engineering and Computer Science,
WCECS 2011 in San Francisco, USA, Vol. I, October 19-21,
2011.
Basis Function Network
Endang Setyati
Sekolah Tinggi Teknik Surabaya
Informatic Department
Surabaya, Indonesia
+62315027920
[email protected]
Mauridhi Hery Purnomo
Yoyon K. Suprapto
Institut Teknologi Sepuluh Nopember Institut Teknologi Sepuluh Nopember
Electrical Engineering Department
Electrical Engineering Department
Surabaya, Indonesia
Surabaya, Indonesia
+62315994251-54
+62315994251-54
[email protected]
ABSTRACT
Emotion recognition through the computer-based of facial
expression has been an active area of research in the literature for
a long time. In this paper we develop a facial emotional
expression synthesis (FEES) techniques based on the facial
features extracted from facial characteristic points (FCPs) in
frontal image sequences. In order to synthesize such facial
expression, FCPs can be used as landmarks. These FCPs are
feature points that represent significant movements during the
generation of expression. We try to simulate people’s expressions
artificially using Radial Basis Function Network (RBFN. The
experimental result from classifier, success rate was about 91.57%
using RBFN classifiers.
Keywords
Facial emotional expression synthesis; facial features extracted;
facial characteristic points; radial basis function network
1. INTRODUCTION
Face plays an essential role in interpersonal communication.
Facial expressions play an important role in non-verbal
communications. Many applications for teleconferencing, human
computer interface and computer animation require realistic
reproduction of facial expressions.
The interest in computational models of emotion and emotional
expressions has been steadily growing in the agent research
community. Several psychologists have acknowledged the role of
emotions in intelligence [1].
In 1971, Ekman and Friesen [2] postulated six primary emotions
that each posses a distinctive content together with a unique facial
expression. These prototypic emotional displays are also referred
to as basic emotions. They seem to be universal across human
ethnicities and cultures and comprise happiness, sadness, fear,
disgust, surprise, and anger. The method of recognizing the 6
universal facial expressions using neural network is discussed.
Most emotion theorist [3] emphasize the involuntary nature of
emotional experience, ignoring those instances when people
choose to generate an emotion through reminiscence or by
adopting the physical actions associated with a particular emotion
(e.g., speaking more softly to deintesify anger or smiling to
generate enjoyment).
[4] believed there exist a relationship between facial expression
and emotional state. There is a small set of basic emotions that
can be expressed distinctively from one another by facial
expressions. For instance, when people are angry they frown and
when they are happy they smile.
[email protected]
Facial expressions [5] are the result of facial muscle actions
which are triggered by the nerve impulses generated by emotions.
The muscle actions cause the movement and deformation of facial
skin and facial features such as eyes, mouth, and nose. We can use
optical flow to estimate facial muscle actions which can then be
used in recognizing a facial expression.
[6] apply a 234x50x6 back-propagation neural network for
classification of expression into one of six basic emotion
categories and their strength. Then they generate the facial
position information and it is input into the input units of neural
network, networ learning is done by back-propagation algorithm
and recognition test is carried out. For six basic facial expressions,
the correct recognition ratio is found to about 90%.
Suppose we are given a 2D grayscale expressionless face image of
a person, i.e., a face image without any expression of emotion [7],
how can we synthesize different expression of that person? One of
the ways to synthesize facial expressions is to find the
approximate displacement of prominent facial feature points.
Moreover, [8] often cannot obtain accurate facial landmark
displacement information due to inherently inaccurate input data.
This is because: (1) it is hard to generate a set of standardized
expressions, e.g., each person may smile differently, (2) it is hard
to produce accurately the precise degree of a particular
expression, e.g., how to generate a 20% smile?, and (3) it is
difficult to mix various facial expressions, e.g., how to gesture a
happy and sad face?
[9] in synthesizing facial expressions include texture mapping
approach to 3D facial image synthesis and [10] use of 3D model
of facial muscles and tissues. An alternative approach has been
investigated by [11], which demonstrated the use of RBF in
interpolating the anchor points for 2D image warping, which can
be applied to synthesize facial expressions. However, it is does
not provide a mechanism to determine the appropriate destination
of the anchor points for each particular facial expression.
Human face has several unambiguous features: eyebrows, eyes,
mouth, nose, and face outline. [13] extract three main features:
eyebrows, eyes, and mouth. After extracting these feature, they
are able to get 30 points of the FCPs.
[14] proposed a hierarchical model of RBFN to classify and to
recognize facial expressions. This approach utilizes Principal
Component Analysis as the feature extraction process from static
images. This research is to develop a more efficient system to
discriminate 7 facial expressions. They achieved the correct
classification rate above 98.4% which is overwhelmingly
distinguished compared to other approaches.
[15] develop a facial expression recognition system, based on the
facial features extracted from FCPs in frontal image sequences.
Selected facial feature points were automatically tracked using a
cross-correlation based optical flow, and extracted feature vectors
were used to classify expressions, using RBFN and FIS. Success
rates were about 91.6% using RBF and 89.1% using FIS
classifiers.
In this paper, we proposed two systems for classifying of the
facial expressions from The Japanese Female Facial Expression
(JAFFE) Database [17]. 7 features extracted from 30 feature
points and from a feature vector for each expression. These
feature vectors were used to training a RBFN classifier to classify
input feature vectors into one of the six basic emotions.
2. INFORMATION OF FACIAL
EXPRESSION
2.1 Facial Characteristic Points
FCPs carry the information about the position and shape of these
three features. According to the study of the Ekman and Friesen
[12], almost all facial expressions of human face are described by
combination of 46 basic movements of facial muscles and these
basic movements are called Action Units (AUs). 30 AUs are
directly associated with movement of eyes, eyebrows, and mouth.
That is why the information expressing movement of eyes,
eyebrows, and mouth is desirable for machine recognition of
facial expressions. The information of each of 6 basic facial
expressions is obtained by subtracting the FCPs coordinates of
normal facial expression from those of facial expressions.
In [6] are confined in these three components and then determine
FCPs which are representative of the boundary between these
components and skin. A set of 30 facial landmarks located near
the eyes, eyebrows, and the mouth are defined as the FCPs. These
points are shown in Figure 1.
FCPs are the points in a face which can represent facial
characteristics. Figure 1 shows the FCPs and a i is a vector
expressing the coordinate of FCPs. ai is described as [7], [8]
a (x , y ), i 1,2,..., 30
i
i i
(1)
To normalize the face image, we introduce a quantity, base, which
is not varied for each of facial expressions, expressed as the origin
of which is assigned at the top of nose, is taken for the coordinate
of FCPs in this study. The information of these 30 FCPs are input
to a computer by using a mouse device.
In Figure 1, Xb-Yb coordinate system shows the absolute
coordinate system and X’-Y’ coordinate system is used for the
new coordinate of FCPs. The origin (originx, originy) of X’-Y’
coordinate system is chosen at the point of length base downward
of the mid point between the left and right eyes [5], [6], [7], [8].
2
2
base (X - X ) (Y - Y )
b2
b1
b2 b1
(2)
Then we Introduce , which is the inclination of face with
respect to the horizontal line, and defined as
θ tan 1
(Y - Y )
b2 b1
-X )
b2
b1
(X
Figure 1. Facial Characteristic Points [6], [7], [8]
The X and Y coordinates, (X0, Y0), of the mid point between the
left and right eyes are described as
X
0
Y X
X X
b2
b1
b2 and Y b1
0
2
2
(4)
The origin of the new coordinate system is calculated as
originx=X0+base*sin and originy=Y0 – base*cos
(5)
The coordinate (Xbi,Ybi) of a FCP ai is transformed into X-Y
coordinates system, (X i,Yi) by subtracting the origin coordinates
and then rotating an angle of orientation , and their relationships
are given by
Xi = Xbi - originx and Yi = Ybi - originy
(6)
Xir = Xi cos + Yi sin and Yir = -Xi sin + Yi cos
(7)
We normalize the input face input face image by dividing
(Xir,Yir) above by value base to compensate the distance effect
between the client face and the camera and the size of the client
face, given by
Y
X
X' ir and Y' ir
i base
i base
i = 1, 2, ..., 30
(8)
2.2 Feature Extraction from Feature Points
Seven features were extracted from the feature vector for each
expression, and were used to classify that expression to one of the
six basic emotions, using RBFN. Extracted features are as follows
[6], [8]:
Openness of Eyes:
oe
(Y - Y ) (Y - X )
n7 n5
n8
n6
2
(9)
Width of Eyes:
(3)
we
(X
n1
-X
n3
) (X
2
n4
-X
n2
)
(10)
from 10 Japanese female models. In Figure 3 are Example of
Facial Emotional Expressions of JAFFE Database [17].
Height of Eyebrows 1:
he1
(Y - Y
) (Y - X
)
0 n19
0
n20
2
(11)
Height of Eyebrows 2:
he2
(Y - Y
) (Y - Y
)
0
n17
0
n18
2
Angry
(12)
Surprise
Netral
Width of Mouth:
wm X
n24
X
(13)
n23
Disgust
Openness of Mouth:
Sad
Fear
om Y
Y
n26
n25
Nose Tip-Lip Corners Distance:
nl
(X - X
) (X - X
)
0
n23
0
n24
2
Happy
(14)
Figure 3. Example of Facial Emotional Expressions of
JAFFE Database
(15)
Figure 2 are example face image of feature extraction from feature
points. Standardization is needed to find out the relative
displacement of the facial landmarks from their normal position
[17].
The six basic emotions defined by [16] can be associated with a
set of facial expressions. In Table 1 shows textual description of
facial expressions as representations of basic emotions.
Table 1. Facial Expressions of Basic Emotions [16]
No
Basic
Emotion
1
Happy
The eyebrows are relaxed. The mouth is open
and the mouth corners pulled back toward the
ears.
2
Sad
The inner eyebrows are bent upward. The
eyes are slightly closed. The mouth is
relaxed.
3
Fear
The eyebrows are raised and pulled together.
The inner eyebrows are bent upward. The
eyes are tense and alert.
4
Angry
The inner eyebrows are pulled downward and
together. The eyes are wide open. The lips are
pressed against each other or opened to
expose the teeth.
5
Surprise
The eyebrows are raised. The upper eyelids
are wide open, the lower relaxed. The jaw is
opened.
6
Disgust
The eyebrows and eyelids are relaxed. The
upper lip is raised and curled, often
symmetrically.
Figure 2. (a) Example of Feature Extraction, (b)
Example of FCPs
3. FACIAL EMOTIONAL EXPRESSIONS
3.1 Facial Emotional Expression Synthesis
The synthesis of facial emotional expressions can be seen as a
reverse process of facial emotional expressions recognition [7]. In
recognition, we present the necessary information (the movements
of the landmarks) so as to classify for a particular facial
expression (an emotional label) in the order happy, sad, angry,
fear, suprised and disgusted.
But what if we reverse the question: given a particular facial
expression, can you tell what the necessary movements of the
landmarks are? This indeed can be seen as a reverse process to
which emotion labels are used and the outputs are the movements
of the landmarks.
The JAFFE Database have 213 face images of 7 facial expressions
(6 basic facial expressions and 1 neutral facial expression) taken
Textual Description of facial Expressions
3.2 Radial Basis Function Network
The basic idea is to find out the spatial differences between the
FCPs of the normal face and that of the expressive face. Thus,
differences of those 30 pairs of position information will
constitute the 60 inputs to the two-layered neural network as
Figure 4.
G(||d-1||)
w
G(||d-1||)
d
decided by width sigma such that a smooth interpolation over the
input space is allowed. The whole architecture is therefore fixed
by determing the hidden layer and the weights between the middle
and the output layers.
o
The number of input layer units must be equal to 7, equal to the
number of extracted features, and that of output layers is 6, which
corresponds to six kinds of facial expressions. The network
training is carried out by back propagation algorithm.
G(||d-1||)
4. EXPERIMENTAL RESULT
Figure 4. RBFN Structure [7]
The basic principle of synthesizing facial expressions is to find
out the necessary relative spatial shift of the FCPs for each
expressions of emotion. So what is initially the input to the neural
network in recognition will become the output in synthesis and
vice versa [7]. RBF have proven to be an effective tool in
interpolating data in multidimensional spaces.
The RBFN is ideal for interpolation since it uses a radial basis
function, for example Gaussian function, for smoothing out and
predict missing and inaccurate inputs [8].
We would consider interpolating functions of the form:
m
F d ω g d μ , d R n , k 1,..., n'
j
k
jk
j 1
Table 2. RBF Classifier Test Result
Degree of Expression
Basic
Emotion
Ha
Sa
Fe
An
Su
Di
Result
%
Happy
4.73
1.32
1.25
1.27
1.28
1.26
94.60
Sad
1.30
4.65
1.93
1.90
1.00
2.52
93.00
Fear
1.21
2.85
4.47
2.17
3.49
3.17
89.40
Angry
1.37
2.13
1.78
4.64
1.59
2.91
92.80
Surprise
1.81
1.49
2.54
1.62
4.24
1.00
84.80
Disgust
1.18
2.43
2.41
2.66
2.01
4.74
94.80
Average
91.57
(17)
5. CONCLUSION
where d is the input vector, is a set of weights and is the width
of the RBF.
Hence, the determination of the nonlinear map F(d) has been
reduced to the problem of solving the following set of linear
equations for the coefficients j,
f1k A11 A1m ω1k
. , k=1,2, ..., n’
f A
mk m1 A mm ω mk
Learning rate used is 0.01. The process is divided into two parts,
namely the process of calculating the hidden layer by using kmeans clustering and the training process the input. The total time
for facial feature extraction, pre-processing, neural network
calculation takes less than 15 seconds.
(16)
where . denotes the usual Euclidean norm on Rn and jRn’,
j=1, 2, ..., m denotes the centers of the radial-basis functions
which are given as the known data points. Often, the g(.) is the
normalized Gaussian activation function defined as
exp (d - μ ) 2 /2σ 2
j
j
g( d)
2
2
k exp (d - μ ) /2σ
k
k
A set of 256 x 256 grayscale images are used in our experiment.
In the RBFN classifier for FEES, we used 6 input layer, 10 hidden
layer, and 60 output in 60 sample. We just did a bit of input
samples and hidden units to be more easily studied. The more
input samples and the number of hidden units, then the result will
be better.
In this research we presented two systems for classifying of the
facial expressions from JAFFE Database. In the RBFN classifier,
7 feature extracted from 30 feature points were used as training
and test sequences. The trained RBFN was tested by features that
not used in training and we have obtained a high result rate of
91.57%.
6. REFERENCES
(18)
where A g d μ , i, j 1, 2, ..., m.
ij
j
RBFN is class of single hidden layer feedforward networks where
the activation functions for hidden units are defined as radially
symmetric basis functions phi such as the Gaussian function. The
fraction of overlap between each hidden unit and its neighbors is
[1]
M.S. El-Nasr, “Modeling Emotion Dynamics in Intelligent
Agents,” M.S. Thesis, May 1988.
[2]
Paul Ekman and W.V. Friesen, “Facial Action Coding
System (FACS),” Consulting Psychologists Press, Inc., 1978.
[3]
Paul Ekman, “Facial Expressions of Emotion: New Findings,
New Questions,” American Psychological Society, Vol. 3,
No. 1, pp. 34-38, January 1992.
[4]
Paul Ekman, “Facial Expressions and Emotion,” American
Psychological, Vol. 48, pp. 384-392, 1993.
[5]
Hiroshi Kobayashi and Fumio Hara, “The Recognition of
Basic Facial Expressions by Neural Network,” in
[6]
[7]
International Joint Conference on Neural Network, pp. 460466, 1991.
2009 American Psychological Association, Vol. 96, No. 1,
pp. 1-10, 2009.
Hiroshi Kobayashi and Fumio Hara, “Recognition of Six
Basic Facial Expressions and Their Strength by Neural
Network,” IEEE International Workshop on Robot and
Human Communication, pp. 381-386, 1992.
[21] Keith Anderson and Peter W. McOwan, “A Real-Time
Mary Y.Y. Leung, Hung Yen Hui and Irwin King, “Facial
Expression Synthesis by Radial Basis Function Network and
Image Warping,” In IEEE International Conference on
Neural Networks, Vol III, pp. 1400-1405, Washington D.C.,
1996. IEEE Computer Society.
[8]
I. King and H.T. Hou, “Radial Basis Function for Facial
Expression Synthesis,” In IEEE International Conference on
Neural Networks, 1996.
[9]
J. Yau and A. Duffy, “A Texture mapping approach to 3D
facial image synthesis, 1988.
[10] Fumio Hara and Hiroshi Kobayashi, “Computer Graphic for
Expressing Robot-Artificial Emotions,” IEEE International
Workshop on Robot and Human Communication, pp. 381386, 1992.
[11] D.R. Nur Arad, Nira Dyn and Y. Yeshurun. Image Warping
by Radial Basis Functions: Application to facial Expressions.
CVGIP: Graphical Models and Image Processing, 56, No.
2:161-172, 1994.
[12] P. Ekman and W.V. Friesen, “Unmasking the Face,”
Consulting Psychologists Press, Inc., 1975.
[13] Jyh-Yeong Chang and Jia-Lin Chen, “A Facial Expression
Recognition System Using Neural Networks,” IEEE, pp.
3511-3516, 1999.
Lin and Jam Chen, “Facial Expressions
Classification with Hierarchical Radial Basis Function
Networks,” IEEE, pp. 1202-1207, 1999.
[14] Daw-Tung
[15] Hadi
Seyedarabi, Ali Aghagolzadeh, and Sohrab
Khanmohammadi, “Recognition of Basic Facial Expressions
by Feature-Points Tracking using RBF Neural Network and
Fuzzy Inference System,” IEEE International Conference on
Multimedia and Expo (ICME), pp. 1219-1222, 2004.
[16] Surya Sumpeno, Mochamad Hariadi, and Mauridhi Hery
Purnomo, “Facial Emotional Expressions of Life-like
Character Based on Text Classifier and Fuzzy Logic,” in
IAENG International Journal of Computer Science, 38:2,
IJCS_38_2_04 [online], May 2011.
[17] The Japanese Female Facial Expression (JAFFE) Database.
Available: [Online] http://www.ksrl.org/jaffe.html.
[18] Muid Mufti, Assia Khanam, “Fuzzy Rule Based Facial
Expression Recognition,” IEEE Computer Society,
International Conference on Computational Intelligence for
Modelling Control and Automation, and International
Conference on Intelligent Agents, Web Technologies and
Internet Commerce (CIMCA-IAWTIC’06), 2006.
[19] Hadi
Seyedarabi, Ali Aghagolzadeh, and Sohrab
Khanmohammadi, “Facial Expression Recognition from
Static Images using RBF Neural Networks and Fuzzy
Logic,” 2nd Iranian Conference on Machine Vision and
Image Processing (MVIP 2003), Tehran, 2003.
[20] David Matsumoto and Bob Willingham, “Spontaneous Facial
Expressions of Congenitally and Noncongenitally Blind
Individuals,” Journal of Personality and Social Psychology,
Automated System for the Recognition of Human Facial
Expressions,” IEEE Transactions on Systems, Man and
Cybernetics-Part B: Cybernetics, Vol. 36, No. 1, February
2006.
[22] Kwang-Eun Ko and Kwee-Bo Sim, “Development of a
Facial Emotion Recognition Method based on combining
AAM with DBN”, IEEE Computer Society, 2010
International Conference on Cyberworlds, pp. 87-91, October
2010.
[23] Kwang-Eun Ko and Kwee-Bo Sim, “Facial Emotion
Recognition using a Combining AAM with DBN”,
International Conference on Control, Automation and
Systems 2010, ICROS in KINTEX, Gyeonggi-do, Korea, pp.
1436-1439, October 27-30, 2010.
[24] Marian Stewart Bartlett, Gwen Littlewort, Claudia Lainscsek,
Ian Fasel, Javier Movellan, “Machine Learning Methods for
Fully Automatic Recognition of Facial Expressions and
Facial Actions,” IEEE International Conference on Systems,
Man and Cybernetics, pp. 592-597, 2004.
[25] R. Ravi and Mahima S, “Study of the Changing Trends in
Facial Expression Recognition,” International Journal of
Computer Application (0975-8887) Vol. 21, No. 5, pp. 1016, May 2011.
[26] Spiros V. Ioannou, Amarylis T. Raouzaiou, Vasilis A.
Tzouvaras, Theofilos P. Mailis, Kostas C. Karpouzies,
Stefanos D. Kollias, “Emotion Recognition through Facial
Expression Analysis based on a Neurofuzzy Network,”
Elsevier
Ltd.
All
rights
reserved.
doi:10.1016/j.neunet.2005.03.004. Neural Networks 18
(2005), pp. 423-435, 2005, Available online at
www.sciencedirect.com and
[27] V. Radha, and N. Nallammal, “Neural Network Based Face
Recognition Using RBFN Classifier,” Proceedings of the
World Congress on Engineering and Computer Science,
WCECS 2011 in San Francisco, USA, Vol. I, October 19-21,
2011.