Analysis And Development Of A Control Strategy For Robotic Wheelchair Controlled Using Single Channel Eeg Headset.
FACULTY OF ELECTRICAL ENGINEERING
UNIVERSITI TEKNIKAL MALAYSIA MELAKA
FINAL YEAR PROJECT
REPORT 2 (FYP 2)
ANALYSIS AND DEVELOPMENT OF A CONTROL STRATEGY FOR ROBOTIC WHEELCHAIR CONTROLLED USING SINGLE CHANNEL EEG HEADSET
Mohamad Amirul Bin Ariffin @ Aripin
Bachelor of Electrical Engineering
(Control, Instrumentation & Automation) June 2015
(2)
“I hereby declare that I have read through this report entitle “Analysis And Development Of A Control Strategy For Robotic Wheelchair Controlled Using Single Channel EEG Headset” and found that it has comply the partial fulfillment for
awarding the degree of Bachelor of Electrical Engineering (Control, Instrumentation and Automation)”
Signature :
Supervisor name :
Date :
a
……….
Tarmizi Bin Ahmad Izzuddin ……….
24/6/2015
(3)
ANALYSIS AND DEVELOPMENT OF A CONTROL STRATEGY FOR ROBOTIC WHEELCHAIR CONTROLLED USING SINGLE CHANNEL EEG HEADSET
MOHAMAD AMIRUL BIN ARIFFIN @ ARIPIN
Areportsubmittedinpartialfulfillmentoftherequirementsforthedegreeof BachelorinElectricalEngineering(Control,Instrumentation&Automation)
FacultyofElectricalEngineering
UNIVERSITITEKNIKALMALAYSIAMELAKA
(4)
I declare that this report entitle “Analysis And Development Of A Control Strategy For Robotic Wheelchair Controlled Using Single Channel Eeg Headset” is the outcome of my own study
except as cited in references. The report has not been accepted for any degree is not concurrently submitted in application of ano ther degree.
Signature :
Name :
Date :
a
……….
Mohamad Amirul Bin Ariffin @ Aripin ……….
24/6/2015
(5)
ACKNOWLEDGEMENT
Alhamdulilah, I am grateful to Allah s.w.t. for the good health and wellbeing that were necessary for me to complete this thesis.
First of all I would like to express my grateful to Mr. Tarmizi Bin Ahmad Izzuddin for guiding me in this study. I would also give my gratefulness, especially to my family for their assistance, advices and inspiration. Without their continued support and interest, this project would not have been same as presented here. Then, I would also like to express my appreciation to Ahmad Suhaizan Bin Pandak and fa.mily, Hafiy Syazwan bin Zainoddin and his Master’s research colleagues and of course to the entire parties that is involved in this project. They have given their full commitment without asking for any rewards.
Besides that, I would like to compliment my special thanks to Universiti Teknikal Malaysia Melaka (UTeM) especially to Fakulti Kejuruteraan Elektrik (FKE) and my fellow friends under the same laboratory which is Rehabilitation Engineering and Assistive Technology (REAT) lab for giving me an opportunity to apply the engineering understanding and develop skills of electrical field in this project.
(6)
ABSTRACT
Nowadays, with modern days of new generation many improvement and new innovative of machine, system and devices have been made. These developments also take account of in refining in quality life of people especially in medical. Biomedical signal lately have been a hot topic for researchers, as many journals and books related to it have been publish. In this paper, the control strategy to help damaged motor patient using BCI on basis of EEG signal was used. BCI is a technology that obtain user thought to control a machine or device. This technology has regained ability for quadriplegic or in other words a person that lost capability of his four limbs to move by himself again. Within the past years, many researchers have come out with a new method and investigation to develop a machine that can fulfill the objective for quadriplegic patient to move again. Besides that, due to the development of bio-medical and healthcare application, there are several ways that can be used to extract signal from brain. One of them is by using Electroencephalography (EEG). This research is carried out in order to detect the brain signal to controlling the movement of the wheelchair. A group of 5 healthy people will be chosen in order to determine performances of the machine during dynamic focusing activity are to focus on a stimulus. From the result that been collected during experiment, neural network configurations will be implemented to classify the signal. Data collected will be extracted and will be used to set a threshold for the machine to active. As a conclusion, a good neural network configuration and a decent method of extracting EEG signal will lead to give a command to control robotic wheelchair.
(7)
ABSTRAK
Generasi moden hari ini telah melakukan pelbagai inovasi dan penambahbaikan terhadap mesin, sistem dan peralatan. Transformasi ini termasuklah penambahbaikan kepada kualiti dalam pelbagai bidang terutamanya bidang perubatan. Isyarat bio-perubatan muktahir ini telah menjadi topik hangat dalam kalangan penyelidik. Ini kerana terdapat banyak jurnal dan buku yang berkaitan dengannya telah diterbitkan. Projek yang dihasilkan ini, strategi kawalan akan membantu pesakit yang lumpuh dengan menggunakan Otak-Komputer Penghubung (BCI) berpandukan isyarat EEG yang akan dikumpul. BCI adalah teknologi yang digunapakai untuk mendapatkan signal (isyarat) daripada otak untuk mengendalikan mesin atau peranti. Teknologi ini akan mengembalikan kemampuan pesakit untuk bergerak sendiri. Dalam tahun kebelakangan ini, ramai penyelidik telah menghasilkan kaedah baru untuk membangunkan mesin yang dapat memenuhi objektif untuk pesakit lumpuh bergerak semula. Melalui pembangunan aplikasi bio-perubatan dan penjagaan kesihatan, terdapat beberapa kaedah boleh digunakan untuk mengambil isyarat dari otak. Salah satunya melalui penggunaan Electroencephalography. Kajian ini dilakukan untuk mengesan isyarat otak yang akan digunapakai bagi mengawal pergerakan kerusi roda. Sekumpulan 5 orang yang sihat dipilih untuk menentukan keupayaan mesin. Aktiviti fokus adalah aktiviti untuk memberi tumpuan kepada bahan rangsangan yang akan diberikan bertujuan untuk menghasilkan isyarat ‘tumpuan’ yang tinggi. Data yang dikumpul akan dikeluarkan dan digunapakai untuk menetapkan had keaktifan mesin. Sebagai kesimpulan, konfigurasi rangkaian neural yang baik dan kaedah yang cekap untuk mendapatkan isyarat EEG akan menghasilkan kawalan yang baik untuk menggerakkan kerusi roda robot.
(8)
TABLE OF CONTENTS
CHAPTER TITTLE PAGE
ACKNOWLEDGEMENT i
ABSTRACT ii
ABSTRAK iii
TABLE OF CONTENTS iv
LIST OF FIGURES vii
LIST OF TABLES x
LIST OF APPENDICES xi
LIST OF ABBREVIATION xii
LIST OF SYMBOL xiii
1 INTRODUCTION 1
1.1 Research Background 1
1.2 Motivation 2
1.3 Problem Statement 3
1.4 Objectives 3
1.5 Scope of Work 4
1.6 Expected Results 4
(9)
2 LITERATURE REVIEW 6
2.1 Theory and Basic Principles 6
2.1.1 Introduction 6
2.1.2 Robotic Wheelchair 7
2.1.3 Electroencephalography 8
2.1.4 Brain-Computer Interfaces 9
2.2 Review of Related Study 10
2.3 Summary and Discussion of the Review 12
3 RESEARCH METHODOLOGY 15
3.1 Summary of the Methods Used In Related Works 15
3.2 Methodology of Collecting Data 18
3.2.1 Control Strategy 19
3.2.2 Neurosky Mindwave Headset 22
3.2.3 Scalp preparation 24
3.2.4 Placement of electrodes 25
3.2.5 Before Experiment 26
3.2.6 Experimental Procedure 27
3.2.7 After Experiment 32
3.2.8 Type of Analysis 32
(10)
4 RESULTS 34
4.1 EEG Analysis 34
4.2 Features Extraction 37
4.3 Neural Network Analysis 38
4.4 Feed Forward Back Propagation with Scaled
Gradient Conjugate Training Function 38
4.5 Pattern Recognition and Classification Result 40
4.5.1 Matrix Confusion 41
4.5.2 Mean Square Error (MSE) & Percent Error 44 4.5.3 Receiver Operating Characteristics (ROC) 47
5 CONCLUSION 50
5.1 Conclusion 50
5.2 Recommendation 52
REFERENCES 53
(11)
LIST OF FIGURES
FIGURE TITLE PAGE
2.1 Wheelchair system setup 7
2.2 Cross section of the cerebrum through the sensory motor region 8 with the motor and sensory sequence indicated
2.3 Basic structure of BCI 9
2.4 A person wearing an EEG-cap and navigating the Bremen
Autonomous Wheelchair Rolland. 10
3.1 Flow of the project 18
3.2 Overall data process 19
3.3 An interval at certain time is taken on Normalized EEG signal 19
3.4 Scaled view of normalized data of 1 second 20
3.5 Fast-Fourier Transform was carried out at the selected time
Interval 20
3.6 Diagram of neural network configuration 21
3.7 NeuroSky Mindwave headset 22
3.8 Alcohol swabs 24
3.9 Neurosky Mindwave headset placement on head 25
3.10 (a) Track was setup for the experiment 27
(12)
FIGURE TITLE PAGE
3.11 (a) Wheelchair joystick to control the movement 29
3.11 (b) Speed set to level 3 29
3.12 (a) Unwanted data recording 30
3.12 (b) Good data recording 30
3.13 Overall experiment setup 31
4.1 Normalized EEG signal 34
4.2 (a) Frequency spectrum of Normalized EEG signal
(Delta, Theta & Alpha) 36
4.2 (b) Frequency spectrum of Normalized EEG signal
(Beta & Gamma) 36
4.3 Illustration of the ranges of EEG signal ranged from
0.1-80 Hz 37
4.4 Matrix confusion result from Pattern Recognition tools 42 in MALTAB software
4.5 Graph of Matrix Confusion Performance 43
4.6 Example of Mean Square Error graph at epoch 15 45
4.7 Graph of Mean Square Error versus No. of Hidden Neurons 46 4.8 Graph of Percentage Error Versus No. of Hidden Neurons 46
(13)
FIGURE TITLE PAGE
4.10 ROC curve of 500 hidden neurons 48
(14)
LIST OF TABLE
NO. TITLE PAGE
2.1 Review of previous related works 12
3.1 Review of the methodology of related works 15
3.2 Neurosky Mindwave headset specifications and hardware overview 23
3.3 Time for subject to move and stop the wheelchair 28
3.4 Gantt chart table 33
4.1 Features of EEG signal 37
4.2 Example of Input data arrangement from average of each wave 39
4.3 Example of output data arrangement 39
4.4 Confusion Matrix arrangements 41
4.5 Final result of Confusion Matrix 43
(15)
LIST OF APPENDICES
APPENDICES TITLE PAGE
A Overview of the BCI where the user selects the destination 57 on the TV
B Correct position to place electrode i.e. on left frontal lobe
(16)
LIST OF ABBREVIATION
BCI - Brain Computer Interface ECG - Electrocardiogram
EEG - Electroencephalography EMG - Electromyography
FP - False Positive
FFBPNN - Feed Forward Back Propagation Neural Network FFT - Fast-Fourier Transform
Hz - Hertz
LED - Light Emitting Diode MATLAB - Matrix Laboratory
ms - Millisecond
MSE - Mean Square Error
REAT - Rehabilitation Engineering and Assistive Technology ROC - Receiver Operating Characteristics
SSVEP - Steady- State Visual Evokes Potentials
TP - True Positive
UTeM - Universiti Teknikal Malaysia Melaka
(17)
LIST OF SYMBOL
α - Alpha
β - Beta
δ - Delta
γ - Gamma
(18)
CHAPTER 1
INTRODUCTION
This section gives data in regards to basis of the study, issue explanations, and goals of the study. This part additionally clarifies the extent of the study. The arrangement of how the study will be led likewise introduced in this section.
1.1 Research Background
Currently, the use of bio-signals such as EMG , EEG and ECG to help the life of elderly people and disable are rising. People who are severely immobilized, quadriplegics in particular, may not be capable to easily control an electric wheelchair and are thus confined to a push-chair, depend on external support [1]. However, researcher has developed tools to use EEG as human command to control machines. At the same time, many ways have been develop to help this kind of patient to recover their ability to move. One of the ways to regain the ability is by using robotic wheelchair. This project introduces a robotic wheelchair controlled by using single channel EEG headset to control the movement of stop and forward and gaze tracking algorithm to control the direction of the wheelchair. It means the purpose of this project is to develop an effective robotic wheelchair using single channel EEG headset
(19)
which can be used without being implanted in the user body. The significance of this research had been supported by many academic publications. One of the academic papers [2] written by Khalil Ullah and co.
1.2Motivation
Nowadays, people suffer from quadriplegic are assumed to be ended their life alone on bed without an ability to move by theirselves again. Recently, several companies have invented equipment’s that can be used to obtained brain activity that can convert desired in thought into actions in real life. EEG, ECG and EMG are some of the well-known technique to collect data from patients. All of these equipment’s functions are wave and resonance based. Simple, user-friendly, effective, and accurate are the reason these equipment’s is having a broad intensively used in determining and analyzing of human illness. The numbers of people with the quadriplegic difficulties are growing not only in Malaysia but all over the world. Based on that fact, the mounting demand and awareness of people suffer of quadriplegic from a health care provider are also increasing. Basically quadriplegic problem may due to several factors such as accidents, ageing and other influences. This problem believes due to injury of neck that cost all motor sensory to be impaired. In order to identify the brain signal, Neurosky Mindwave headset was used to detect the electrical signal produced by the brain and from the collected data [3]. After that, we can apply a neural network control technique to classify the person intention that has a quadriplegic to move. This research carried out in order to gain the knowledge about EEG and experience on how EEG data can be used in applications. Besides that, this research also will identify the significant activities that will help quadriplegic patient to regain moving ability by their own self and how to apply EEG signal for other purpose. Although lack of knowledge and information, by doing research, it can motivate students and other researchers to study and explore more problems related to any EEG signals.
(20)
1.3 Problem Statement
Quadriplegic are the patient that loss the control to move the human motor functions. They suffer this disorder regularly due to neck injury that leads to limb paralysis. Paraplegic is an illness that quite similar with quadriplegic but paraplegic only cost the patient to lost the movement the organs on waist and below. While, quadriplegic lost the ability to control organs starting from neck to below.
However, this patient have still have the ability of their upper organ such as hearing ears, seeing with eyes and moving their mouth. This is includes brain that can control the whole body activity. Brain activity also preserved for quadriplegic patients. Thus, detection of EEG signal is possible. By applying BCI concept this signal can be obtained by placing a correct spot of electrode on subject head. By using EEG signal that can be turned into command a quadriplegic person can make their desired action into real life just by imagine in their heads.
By taking advantages of robotic wheelchair facility, quadriplegic patient can regain their movement by their own selves with assistance of single channel EEG headset as a medium between robotic wheelchair and controller on the wheelchair. Robotic wheelchair basically consists of motor and controller, in certain cases includes sensor for safety of the user.
1.4 Objectives
The main objectives for this project are:
i) To implement neural network control technique in order to move and stop robotic wheelchair.
ii) To analyze and evaluate which is the best EEG signal pattern for moving and stopping the wheelchair.
(21)
1.5 Scope Of Work
In order to achieve the objectives of this research, there are several scopes as listed. To determine the EEG signal, there are several criteria need to be considered. Firstly, the hardware that will be used as intermediate between quadriplegic intention and robotic wheelchair is Neurosky Mindwave headset. This headset type use non-invasive BCI method to obtain EEG signal. Apart from that, this headset also a single channel EEG headset. The EEG signal will classify, validate and tested by random subject that will be selected. This project’s subject selections are 5 healthy people. Selections of participant are chosen by random among student in UTeM. This is because demanding and complex procedures are needed to get participants that are quadriplegic person. Next, to identify the electrical signal yield by the brain, subject will going to complete the experiments in a controlled surroundings. This is to ensure no mental distractions, wandering thoughts, lack of focus, or anxiety that may lower the performance of the machine.
1.6 Expected Results
Based, from the objectives some of the funding listed below are supposed to be obtained at the end of the project.
1. Brain signal from Neurosky Mindwave Headset to be used for robotic wheelchair command.
2. Neural network configuration that consist of brain signal as input and moving and stopping the robotic wheelchair as output of the system.
3. A suitable method for subjects to focus in order to avoid subjects from suffering fatigue.
(22)
1.7 Report Structure
This report comprises of five sections which are Chapter 1: Introduction, Chapter 2: Literature Review, Chapter 3: Methodology, Chapter 4: Result and Chapter 5: Conclusion.
Chapter 1 is Introduction, which review about the undertaking study framework, objectives, problem statement and extent of the study. Part 2 is Literature review which is assessment the past undertaking and all material concept use in this study. Chapter 3 is about Methodology where procedure of equipment and programming utilized in this project procedure. Chapter 4 will consists of the discussion of findings of the experiment. Lastly Chapter 5, the conclusion section which is the part that concludes all the results obtained in Chapter 4 and recommendation for future research.
(23)
CHAPTER 2
LITERATURE REVIEW
This part displays the literature review of the analysis and improvement of a control strategy for robotic wheelchair controlled using single channel EEG headset.
2.1 Theory And Basic Principles
2.1.1 Introduction
This chapter concentrated on the factual and also theoretical aspects of the project. It is relating to the fundamental of controlling robotic wheelchair using Neurosky Mindwave headset. In order to obtain complete and accurate information, in-depth research should be done especially through reading articles and journals.
(24)
2.1.2 Robotic Wheelchair
Wheelchair is a four wheeled transportation used by people that unable to walk due to accidents of illness. An external support often needed for those people that severely immobilized, quadriplegics in particular, rely on help of others to move but researchers have developed tools that translate the patient desired into command to control machine. Robotic wheelchair is a wheelchair that moved by motor instead of an assist by someone or by the patient own hands. A robotic wheelchair is usually are motorized that stimulated by the patient intention. EEG signal was used to deliver the patient intended location to move. In other words, this is an effort to utilize brain signal to control motorized devices. A personal computer is attached to the back of the seat to plays the part as the intelligent controller for this system [4]. Figure 2.1 shows the previous design of wheelchair setup [5].
(1)
had been supported by many academic publications. One of the academic papers [2] written by Khalil Ullah and co.
1.2Motivation
Nowadays, people suffer from quadriplegic are assumed to be ended their life alone on bed without an ability to move by theirselves again. Recently, several companies have invented equipment’s that can be used to obtained brain activity that can convert desired in thought into actions in real life. EEG, ECG and EMG are some of the well-known technique to collect data from patients. All of these equipment’s functions are wave and resonance based. Simple, user-friendly, effective, and accurate are the reason these equipment’s is having a broad intensively used in determining and analyzing of human illness. The numbers of people with the quadriplegic difficulties are growing not only in Malaysia but all over the world. Based on that fact, the mounting demand and awareness of people suffer of quadriplegic from a health care provider are also increasing. Basically quadriplegic problem may due to several factors such as accidents, ageing and other influences. This problem believes due to injury of neck that cost all motor sensory to be impaired. In order to identify the brain signal, Neurosky Mindwave headset was used to detect the electrical signal produced by the brain and from the collected data [3]. After that, we can apply a neural network control technique to classify the person intention that has a quadriplegic to move. This research carried out in order to gain the knowledge about EEG and experience on how EEG data can be used in applications. Besides that, this research also will identify the significant activities that will help quadriplegic patient to regain moving ability by their own self and how to apply EEG signal for other purpose. Although lack of knowledge and information, by doing research, it can motivate students and other researchers to study and explore more problems related to any EEG signals.
(2)
1.3 Problem Statement
Quadriplegic are the patient that loss the control to move the human motor functions. They suffer this disorder regularly due to neck injury that leads to limb paralysis. Paraplegic is an illness that quite similar with quadriplegic but paraplegic only cost the patient to lost the movement the organs on waist and below. While, quadriplegic lost the ability to control organs starting from neck to below.
However, this patient have still have the ability of their upper organ such as hearing ears, seeing with eyes and moving their mouth. This is includes brain that can control the whole body activity. Brain activity also preserved for quadriplegic patients. Thus, detection of EEG signal is possible. By applying BCI concept this signal can be obtained by placing a correct spot of electrode on subject head. By using EEG signal that can be turned into command a quadriplegic person can make their desired action into real life just by imagine in their heads.
By taking advantages of robotic wheelchair facility, quadriplegic patient can regain their movement by their own selves with assistance of single channel EEG headset as a medium between robotic wheelchair and controller on the wheelchair. Robotic wheelchair basically consists of motor and controller, in certain cases includes sensor for safety of the user.
1.4 Objectives
The main objectives for this project are:
i) To implement neural network control technique in order to move and stop robotic wheelchair.
ii) To analyze and evaluate which is the best EEG signal pattern for moving and stopping the wheelchair.
(3)
In order to achieve the objectives of this research, there are several scopes as listed. To determine the EEG signal, there are several criteria need to be considered. Firstly, the hardware that will be used as intermediate between quadriplegic intention and robotic wheelchair is Neurosky Mindwave headset. This headset type use non-invasive BCI method to obtain EEG signal. Apart from that, this headset also a single channel EEG headset. The EEG signal will classify, validate and tested by random subject that will be selected. This project’s subject selections are 5 healthy people. Selections of participant are chosen by random among student in UTeM. This is because demanding and complex procedures are needed to get participants that are quadriplegic person. Next, to identify the electrical signal yield by the brain, subject will going to complete the experiments in a controlled surroundings. This is to ensure no mental distractions, wandering thoughts, lack of focus, or anxiety that may lower the performance of the machine.
1.6 Expected Results
Based, from the objectives some of the funding listed below are supposed to be obtained at the end of the project.
1. Brain signal from Neurosky Mindwave Headset to be used for robotic wheelchair command.
2. Neural network configuration that consist of brain signal as input and moving and stopping the robotic wheelchair as output of the system.
3. A suitable method for subjects to focus in order to avoid subjects from suffering fatigue.
(4)
1.7 Report Structure
This report comprises of five sections which are Chapter 1: Introduction, Chapter 2: Literature Review, Chapter 3: Methodology, Chapter 4: Result and Chapter 5: Conclusion.
Chapter 1 is Introduction, which review about the undertaking study framework, objectives, problem statement and extent of the study. Part 2 is Literature review which is assessment the past undertaking and all material concept use in this study. Chapter 3 is about Methodology where procedure of equipment and programming utilized in this project procedure. Chapter 4 will consists of the discussion of findings of the experiment. Lastly Chapter 5, the conclusion section which is the part that concludes all the results obtained in Chapter 4 and recommendation for future research.
(5)
CHAPTER 2
LITERATURE REVIEW
This part displays the literature review of the analysis and improvement of a control strategy for robotic wheelchair controlled using single channel EEG headset.
2.1 Theory And Basic Principles
2.1.1 Introduction
This chapter concentrated on the factual and also theoretical aspects of the project. It is relating to the fundamental of controlling robotic wheelchair using Neurosky Mindwave headset. In order to obtain complete and accurate information, in-depth research should be done especially through reading articles and journals.
(6)
2.1.2 Robotic Wheelchair
Wheelchair is a four wheeled transportation used by people that unable to walk due to accidents of illness. An external support often needed for those people that severely immobilized, quadriplegics in particular, rely on help of others to move but researchers have developed tools that translate the patient desired into command to control machine. Robotic wheelchair is a wheelchair that moved by motor instead of an assist by someone or by the patient own hands. A robotic wheelchair is usually are motorized that stimulated by the patient intention. EEG signal was used to deliver the patient intended location to move. In other words, this is an effort to utilize brain signal to control motorized devices. A personal computer is attached to the back of the seat to plays the part as the intelligent controller for this system [4]. Figure 2.1 shows the previous design of wheelchair setup [5].