Existing Research

2. Existing Research

classifier using such physiological response, we cannot expect the classifier to extract the common

Section 2 describes the advantages and disadvantages patterns of physiological response. In this research,

of existing methods to determine emotions. A scale is the emotion is estimated using classifiers trained with

necessary to determine emotions. The scale includes the physiological response of students similar in their

psychological scales to examine subjective emotional personality type. Since a classifier is prepared for each

experiences, behavior scales based on external reactions, personality type, the emotion of a new student can be

and biological scales based on internal responses.

94 Estimating Emotion for Each Personality by Analyzing BVP

The psychological scale includes the introspective emotions in this research. Emotions are estimated method, the Likert scale, the rating scale method, the

from biological data considering personality. Students open-ended question method, the questionnaire take a personality test to select a classifier according method, and so on. Since these scale methods require

to their test result. The method estimates emotions students to answer big questionnaires, they have

from BVP using wearable wireless PPG difficulties to obtain data of many university students.

(Photoplethysmography) sensors. Heart rate and its The behavior scale needs permanent acquisition of

variability can be obtained by analyzing BVP. Various various nonverbal behaviors, such as facial expression,

wearable devices have been developed and operated in posture, attitude, gesture, and voice. Although the field of health care and sports in recent years [9]. behavior data can be obtained through a lot of

For students wearing PPG, an application on a measuring equipment in many places, it causes

smartphone estimates negative emotions. Once it finds privacy violation.

a student who often suffers from negative emotion, it The biological scale uses both of autonomous

notifies faculty and counselors of the information, to reaction by activities of the autonomic nervous system

provide the mental care for the student. Fig. 1 shows and voluntary reactions by activities of the central

an operational diagram of the method. nervous system. The former includes BVP (blood

3.2 Estimating Personality

volume pulse), blood flow rate, blood pressure, heart rate, skin electrical reflection and skin temperature,

In this section, a personality and the method to while the latter includes brain waves, estimate personality are described. There are typology electromyograms, and respiration. The wearable theory and property theory in the ways of grasping a device for determination equipment enables to obtain

personality. The typology theory applies a personality those biological data all the time. Emotions can be

to a stereotype set based on psychological or estimated in an online manner from them.

biological characteristics. The typology theory is

However, physiological responses when each intuitively easy to understand, but the intermediate emotion occurs often do not match. The reason is that

type is likely to be ignored in the theory. It is also the physiological response patterns are different

difficult for one type to move to another. On the among individuals [5]. Leon has optimized classifiers

contrary, in the characteristic theory, the personality is with training using biological data for each person to

composed of several characteristics. It expresses eliminate individual differences in physiological personality with multi-dimensions for quantitative response [6]. Since personalized classification method

comparison. The characteristic theory has the requires time for the training using individual

disadvantage that it is difficult to grasp the identity of biological data, new users cannot use it immediately.

personality. However, we can analyze personality In order to obtain a practical solution to know the

statistically. It is also possible to grasp the personality timing when each university student has a negative

in a typological way, if we classify subjects from a emotion, we need a method many users can use easily

certain viewpoint.

on the spot. These existing methods are not suitable The characteristic theory is used in this research. for easy classification of student emotions.

The Big Five test expressing personality with five

3. Emotion Estimation with Personality

factors is considered to be the most influential test in the characteristic theory. The five factors are

3.1 Use Case neuroticism, extraversion, openness, agreeableness, This section describes the method to estimate

and conscientiousness [10]. Neuroticism responds

Estimating Emotion for Each Personality by Analyzing BVP

Fig. 1 Estimating emotions considering personality.

sensitively to external stimuli and shows emotional low-frequency) component (frequency is 0.0033 to instability trend. Extroversion indicates a tendency to

0.04 Hz), LF component (frequency is 0.04 to 0.15 actively appeal to the outside world. People with high

Hz), HF component (frequency is 0.15 to 0.5 Hz), TP extroversion tend to have positive emotions. It means

(total power) component (the sum of the three openness to experience. It shows a rich tendency for

frequency components), SD (standard deviation) of thought and images. Agreeableness shows a tendency

PR (pulse record) interval, RMssd (whose the to synchronize with other people in relationship with

deviation of the difference between adjacent PR people. Conscientiousness shows a tendency to

intervals)

overcome things with clear purpose and intention. The value of HF indicates the enhancement of Conscientiousness is a dimension related to control of

parasympathetic nervous system, while the value of impulses.

LF/HF indicates the sympathetic nerve system. TP Emotion is presumed to be affected by neuroticism

indicates the activity degree of autonomic nervous related to anxiety causing emotional instability. It is

system. RMssd indicates the tension degree of the also likely to be affected by extraversion leading to

vagus nerve. This research uses wearable wireless positive emotions.

PPG sensors attached to the earlobe.

3.3 Classifiers Based on Heart Rate

3.4 Create Personality Model

The students use wearable wireless PPG sensors to This section describes how to classify personality. measure BVP. We measure the heart rate and the heart

The estimation system in this research uses TIPI-J rate variability from BVP. This section describes heart

invented by Japanese as a scale of Big Five [13]. rate and heart rate variability. The heart rate increases

Since TIPI-J is a simple scale that can measure five with anger, fear, and sadness, not with joy, surprise,

personality traits with each of two items, the burden and disgust [11]. The activity of the human autonomic

on students is small. The estimation system figures out nervous system changes when the emotion changes.

personality vector on the basis of the five personality The activity of the autonomic nervous system is

traits with TIPI-J. After it classifies them, it calculates measured from the frequency response of heart rate

the centroid vector of each cluster by the k-means variability [12]. From the heart rate variability, the

method [14], to create a personality model. A new following components can be obtained; VLF (very

student is classified into the nearest cluster based on

96 Estimating Emotion for Each Personality by Analyzing BVP

the distance from the personality vector of the student male and 10 female. Heart rate and heart rate variation to the centroid vector of the cluster. The number of

were calculated from BVP signals obtained by a personality clusters is determined to the most wireless earlobe PPG sensor, Vital Meter made by appropriate one, trying from 2 to 6 clusters in the

TAOS Institute [17].

experiment. Three types of emotions obtained through experiments were positive emotions, negative

3.5 Emotion Estimation along Personality Model emotions, and emotion during relaxation. After the

Supervised machine learning creates emotional subject recalled one of pleasure events, anger events, classifiers. The emotional classifier is created for each

and the others, we estimated their emotion. The recall personality model. The explanatory variable of the

time was 2 minutes. We conducted each of three types emotional classifier is heart rate and heart rate

of emotion estimation after recalls five times. variability, while the objective variable specifies

We used the random forest to create classifiers. whether the student has negative or positive emotion.

Explanatory variables are 17 variables in total. They To know the student has negative or positive

include the average, the minimum, and the maximum emotion, we use the circumplex model and the affect

of heart rate, beat count (heart rate at all measurement grid proposed by Russell [15, 16]. The circumplex

time), SDNN, RMssd, and the average of VLF, LF, model expresses all emotions in two dimensions of the

HF, LF/HF and TP. It also includes the SD of heart pleasant-unpleasant one and the arousal-sleepiness

rate, VLF, LF, HF, LF/HF, and TP. one. The affect grid is an evaluation method of

The objective variables are two variables; one emotions, based on a circumplex model. It is formed

means the student has positive emotion, while the in a square grid composed of 81 squares of 9 × 9.

other means negative emotion. To obtain the objective It is considered that the accuracy to estimate

variable, the subject’s emotion was attained with the emotion improves, if the biological data are classified

affect grid. In the pleasant-unpleasant dimension of into groups having common patterns. The personality

the affect grid, the center was set to 0. We regarded +1 is used as the scale for the classification. Section 3.2

to +4 as positive emotion, while -4 to 0 as negative shows that personality traits affect brain functions and

emotion.

body reactions. Some researches report that there is a To show that the personality model works difference in the balance of autonomic nervousness

effectively, the classifier created from 20 subjects depending on personality [7, 8]. It is expected that

without classification by personality model was common patterns of biological data can be extracted if

compared with the classifier trained for each students are classified according to personality models.

personality model.

The classifier for each personality model of the user In the former, data of 20 subjects were divided into would estimate the emotion more accurately than one

twenty pieces, one for each subject, and the 20-part ignoring the difference in personality.

cross-validation was carried out. In the latter, a

4. Recalling Experiment

classifier was created for each personality model resulting from clustering of the five personality

4.1 Purpose and Method of Experiment traits of 20 subjects with the k-means method. After

This section describes the experiment conducted in the data were divided into the number of subjects for this research. Experimental results show that the

each personality model, the cross validation for the personality model improves the accuracy of emotion

number of subjects was carried out for each estimation. Our subjects are 20 university students, 10

personality model.

Estimating Emotion for Each Personality by Analyzing BVP

4.2 Estimation Accuracy by Personality Model The rejection region of p-value was set to 0.1. As a result, the personality models A and B showed the

We compared the estimation accuracy by the significant difference in extroversion, A and C showed cross validation of classifiers that do not classify it in openness and conscientiousness, A and D showed personality models as well as classifiers for each of it in extraversion and conscientiousness, B and C two to six personality models. The f-measure, a showed it in conscientiousness and neuroticism, C and harmonic mean of the precision and the recall

D showed it in neuroticism.

was used as an evaluation index of the estimation Next, Figs. 3-5 show the comparison of the average accuracy. When the personality is not classified, the values and dispersion of f-measures in each f-measure is 0.501. The best of estimation accuracy is personality model when classified and not classified 0.557 when the personality model is classified into with the personality model. “Non” is the average four. value of the f-measure of persons belonging to each Fig. 2 shows the codebook of each personality personality model when 20-part crossing verification model when the personality model is classified into was carried out without considering personality. “CP” four. In the score of 5 personality traits by TIPI-J, the is an abbreviation considering personality, which is minimum is 2, while the maximum is 14. The gray the average of f-measure when cross-validation is marker shows the average values of all 20 subjects. carried out only for persons of each personality model Personality model A, B, C, and D involved 8, 5, 4, and

when personality is considered.

3 subjects, respectively. From Fig. 2, personality model A is sociable, strong in outstanding curiosity and self-control, because of the high extraversion, openness, and conscientiousness. Personality model B is introverted, strong in intention and diligence, because of the low extraversion and high conscientiousness. Personality model C believes to be sensitive to the stimulus, which has a solid idea, is

Fig. 2 Personality traits of each personality model.

unique, and accepts himself/herself as he/she is,

Table 1 P-value on each personality model.

because of the high neuroticism and low openness, Compared E O C A N agreeableness, and conscientiousness. Personality

0.032 0.345 0.947 0.776 0.861 model D understands the psychological state of the

A:B

0.100 0.018 0.015 0.713 0.154 others, which are insensitive to stimulation, and have

0.245 0.879 0.065 0.491 0.079 high impulsivity, because of high agreeableness, low

B:C

0.367 0.998 0.140 0.935 0.448 neuroticism and conscientiousness.

B:D

0.999 0.793 0.927 0.220 0.097 We applied the Steel-Dwass test, which is a multiple comparison test [18], assuming that the score

C:D

of personality traits of each character model can be described as “there is no difference between the average values of both groups”. Table 1 shows the obtained p-value as a result of the Steel-Dwass test. The row of “A:B” in the column of “Compared” in Table 1 shows the result of comparing personality

traits of personality model A and personality model B.

Fig. 3 F-measures of positive emotion.

98 Estimating Emotion for Each Personality by Analyzing BVP

Table 2 shows the range of positive emotion and negative emotion on the pleasant axis and the arousal axis for each personality model. On the pleasant axis,

1 to 4 corresponds to the positive emotion, while -4 to

0 to the negative emotion. Both of the emotions become stronger as the absolute value increases. Moreover, the range on the arousal axis is -4 to 4 for

Fig. 4 F-measures of negative emotion.

any emotion. As shown in Table 2, personality model

C seems to have a low arousal level. However, p -value with multiple comparison tests for the

questionnaire after recalling presented no significant difference, if the rejection area is set to 0.1.

Table 3 shows the top 5 of the variable importance in each personality model in the random forest. It turned out that the importance of the explanatory variables is different depending on each personality

Fig. 5 F-measures of positive and negative emotion.

model. In the comparison of personality model A with personality model D, A has at least 2.9, whereas D is

The f-measure of positive emotion in personality

less than 1 for every variable.

model C decreased when personality was considered, but f-measure in the other personality model increased.

5. Discussion

Personality model C with high neuroticism had the

5.1 Personality and Estimation Accuracy greatest CP of negative emotion among other personality

models, and D of low neuroticism is the greatest For each personality model, we investigate the Non-CP of negative emotion. Moreover, personality

estimation accuracy of the classifiers incorporating the model A with high extraversion had the greatest CP

personality models and the classifier neglecting the and Non-CP of positive emotion. Although dispersion

personality models. Let us denote the estimation of negative emotion decreased in all personality models,

accuracy of the former with CP, while that of the latter the dispersion of positive emotion, the average of

with Non. The increasing rate of the difference of CP dispersion of positive and negative emotion rose.

from Non for personality model A is the largest, 0.082, compared with other personality models. Personality

4.3 Emotions for Each Personality Model model A is the highest in the extraversion. Those who

Fig. 6 shows a graph of biological data for each are highly extroverted are said to feel positive personality model. The Steel-Dwass test is applied

emotions, which seem to increase the accuracy of assuming that there is no difference between the

positive emotion. Personality model B has low value average values of both groups regarding biological

both in Non and CP.

data of each personality model. If the rejection area of The activity of autonomic nerves was mild in the p-value is 0.1, a significant difference was

personality model B, because the value of TP observed in the average value and SD of LF/HF of

indicating the degree of activity of autonomic nerves personality models A and D.

was low. Personality model B is considered to be Table 2 shows the values of the questionnaire after

unlikely to change biological data because of its low recalling. The value is represented by the effect grid.

extraversion as well as introverted and calm personality.

Estimating Emotion for Each Personality by Analyzing BVP

Fig. 6 Biological data of each personality model.

Table 2 Ranges on pleasant and arousal axes.

Personality model B Emotion Positive Negative Positive Negative

Personality model A

Pleasant axis 2.352 -1.439 1.786 -1.281 Arousal axis

Personality model D Emotion Positive Negative Positive Negative

Personality model C

Pleasant axis 2.333 -1.167 1.696 -1.591 Arousal axis

Table 3 Top 5 variable importance.

Rank Personality model A

Personality model B

1 st LF_AVG 5.002 VLF_AVG 1.820 2 nd HR_AVG 4.282 HF_SD 1.594 3 rd HR_Min 3.892 RMssd 1.341 4 th LF/HF_AVG 2.985 VLF_SD 1.330 5 th LF_SD 2.949 LF/HF_AVG 1.274

Estimating Emotion for Each Personality by Analyzing BVP

Table 3 to be continued Rank

Personality model D 1 st HF_AVG 2.604 LF_SD 0.908 2 nd HF_SD 2.310 SDNN 0.713 3 rd LF_AVG 2.205 RMssd 0.711 4 th HR_Min 1.958 LF/HF_AVG 0.650 5 th TP_AVG 1.542 LF_AVG 0.633

Personality model C

In personality model C, CP of positive emotion influenced the average value and SD of LF/HF with decreased, while CP of negative emotion was 0.644,

respect to personality model A and personality model which was the largest among all personality models.

D. Eysenck states that extroverts have a high level of Since it has high TP, autonomic nervous activities are

restraint of the cerebral cortex caused by the reticular assumed to be intense in personality model C.

activating system, while introverts have a high arousal Personality model C is sensitive to stimulation and

level [8]. Extroverts are insensitive to stimulation. feels anxiety because of its high neuroticism, which

Their cerebral cortex awakening is late, or the seems to affect the estimation accuracy of negative

awakening falls quickly even if it gets awaken. On the emotion. Since personality model D is introverted and

other hand, introverts are sensitive to stimuli. Their has a low neuroticism, it is considered that its emotion

cerebral cortex tends to awaken excessively even with is calm and stable. It is thought that the estimation

a small stimulus. Therefore, the extrovert type is accuracy of positive emotion was low. People with

considered to have low physiological excitement. high agreeableness are reported to have high-stress

Buck et al. [20] also showed that an extrovert person values [19]. The estimation accuracy of negative

has a weak autonomic nervous system reaction. From emotion was good because of stress accumulation.

the above, it is expected that the value of LF/HF will As a countermeasure to the personality model with

be low in personality model A who is high in the low estimation accuracy, we can add behavior data,

extraversion and insensitive to stimulation. On the such as GPS logs and acceleration to explanatory

contrary, the value of LF/HF would be high because variables. Since changes in biological data are

personality model D is low in the extraversion and unlikely to occur in low-accuracy models, it is

sensitive to simulation. However, the result was considered that the accuracy can be improved with

different. Personality model A with high extraversion behavior data.

had a high LF/HF value, while personality model D with low extraversion had a low LF/HF value.

5.2 Influence of Personality and Biological Data Although this research evaluates the extraversion as

As a result of multiple comparisons of each five factors in the Big Five, Eysenck evaluates it in the biological data for each personality model, a extrovert-introvert dimension. Because of it, the significant difference was found in the average value

difference seems to have occurred. and SD of LF/HF of personality model A and

The extrovert described by Eysenck has traits, such personality model D. There was no significant

as sociability and impulsivity. On the other hand, difference in the questionnaire after a recall by each

extraversion by the Big Five is considered to be personality model. However, there was a significant

cautious with identifying it as sociability. In the Big difference in extraversion and conscientiousness in the

Five, shyness is not extraversion. It considers the personality traits of personality model A and shyness corresponds to high anxiety and neuroticism personality model D. From the above, it is considered

in almost all cases [10]. In addition, the impulsivity of that extraversion and conscientiousness have Eysenck’s extrovert can be seen as conscientiousness

Estimating Emotion for Each Personality by Analyzing BVP

of the Big Five. Therefore, although personality model estimation accuracy improves, if the classifier is

A has high extraversion, it has high conscientiousness trained for each personality model. Personality traits and low impulsivity. It is presumed to be different

of each personality model were suggested to be related from extrovert described by Eysenck. The expectation

to biological data and variable importance. From the that the value of LF/HF will be high is not adapted

above, individual differences in physiological because personality model D with low extraversion is

response differ from each personality type. sensitive to stimulation.

As future work, it is necessary to look at the

Eysenck also states that people with high correlation to see how the personality traits exactly neuroticism are more likely to arouse the autonomic

affect biological data. The incorporation of gender nervous system while people with low neuroticism are

difference could be one way to improve the accuracy. less likely to be aroused. Therefore, it seems that the

Linking time and location information to the estimated value of the SD of LF/HF was low because of the low

emotions, faculties and staffs can prevent school neuroticism in personality model D.

dropout of students with mental care, such as Regarding the fact that explanatory variables

emotional support at a good timing. emphasized in each personality model shown in Table