Motion Tracking System Using Computer Vision With Opencv Source Software.

(1)

UNIVERSITI TEKNIKAL MALAYSIA MELAKA

MOTION TRACKING SYSTEM USING COMPUTER VISION

WITH OPENCV SOURCE SOFTWARE

This report in accordance with requirement of the Universiti Teknikal Malaysia Melaka (UTeM) for Bachelor Degree of Manufacturing Engineering (Robotic and Automation)

with Honours.

WIRADANI BIN MD ALI

FACULTY OF MANUFACTURING ENGINEERING 2009

(2)

UNIVERSITI TEKNIKAL MALAYSIA MELAKA

BORANG PENGESAHAN STATUS LAPORAN PSM

TAJUK:

‘Motion Tracking System Using Computer Vision With OpenCV Source Software’ SESI PENGAJIAN:

2009/2010 Semester 2

Saya WIRADANI BIN MD ALI

mengaku membenarkan laporan PSM ini disimpan di Perpustakaan Universiti Teknikal Malaysia Melaka (UTeM) dengan syarat-syarat kegunaan seperti berikut:

1. Laporan PSM / tesis adalah hak milik Universiti Teknikal Malaysia Melaka dan

penulis.

2. Perpustakaan Universiti Teknikal Malaysia Melaka dibenarkan membuat salinan

untuk tujuan pengajian sahaja dengan izin penulis.

3. Perpustakaan dibenarkan membuat salinan laporan PSM / tesis ini sebagai bahan

pertukaran antara institusi pengajian tinggi.

4. *Sila tandakan (√)

SULIT

TERHAD

⁄ TIDAK TERHAD

(Mengandungi maklumat yang berdarjah keselamatan atau

kepentingan Malaysia yang termaktub di dalam AKTA RAHSIA RASMI 1972)

(Mengandungi maklumat TERHAD yang telah ditentukan oleh organisasi/badan di mana penyelidikan dijalankan)

No 68, Jalan P 16 D, Persint 16,

621510 Putrajaya,

Wilayah Persekutuan Putrajaya.

Tarikh: _______________________

Cop Rasmi:

Tarikh: _______________________

* Jika laporan PSM ini SULIT atau TERHAD, sila lampirkan surat daripada pihak organisasi berkenaan dengan menyatakan sekali sebab dan tempoh tesis ini perlu dikelaskan sebagai SULIT atau TERHAD.

(3)

APPROVAL

This report is submitted to the Faculty of Manufacturing Engineering of UTeM as a partial fulfillment of the requirements for the degree of Bachelor of Manufacturing Engineering (Robotic and Automation) with Honours. The members of the supervisory committee are as follow:

(Signature of Supervisor)

……… (Official Stamp of Supervisor)

(4)

DECLARATION

I hereby declared this report entitled “Motion Tracking System Using Computer Vision With OpenCV Source Software” is the result of my own research except as cited in the references.

Signature : ………..

Author’s Name : ………..

(5)

ABSTRACT

Nowadays motion tracking widely used for some field, especially autonomous field. The motion tracking also has been used for securities. At the Oversea for example United States of America use the motion tracking in army usage. This project purpose is to do the motion tracking using the OpenCV software. The motion tracking is the camera will detect and track the object movement. This project will include the research for the related previous work as a reference to done the motion tracking method. The related previous work will cover in the literature review, the method used are research based on the scopes of this project. This scopes project is only cover the color based, optical flow and feature extraction image processing technique. The methodology chooses for this motion tracking use is edge detection under the optical flow technique and the color based will used the segmentation technique. The methodology chooses is based on the research has been done and the techniques is efficiencies and many of the work done by using this techniques.

(6)

ABSTRAK

Pada era sekarang “motion tracking” ini digunakan secara meluas untuk beberapa bidang, terutama bidang autonomi. “motion tracking” juga telah digunakan untuk system keselamatan. Di luar Negara misalnya Amerika Syarikat telah menggunakan “motion tracking” untuk kegunaan tentera. Objektif projek ini adalah untuk melakukan “motion tracking” perisian menggunakan OpenCV. “motion tracking” kamera akan mengesan dan menjejaki pergerakan objek. Projek ini merangkumi kajian sebelumnya yang berkaitan dengan tajuk sebagai rujukan untuk melakukan “motion tracking” cara kerja. Hasil kerja sebelumnya yang berkaitan akan merangkumi dalam kajian lapangan, cara kerja dan penelitian yang digunakan adalah berdasarkan pada takat projek ini. Skop projek ini hanya merangkumi berasaskan melalui cara warna, optik aliran dan ekstraksi ciri pengolahan gambar teknik. Cara kerja dipilh untuk menggunakan “motion tracking” ini adalah deteksi tepi aliran di bawah teknik optik dan warna yang didasarkan akan menggunakan teknik segmentasi. Memilih cara kerja berdasarkan kajian yang telah dilakukan dan teknik yg digunakan berpandukan daripada kecekapan dan daripada hasil kerja lepas yang telah dilakukan.

(7)

iii

DEDICATION

Specially dedicated to my beloved father Md Ali Sukardi and my mother Zaitom Abdul Hamidwho are very concern, understanding, patient, and supporting. Thanks for everything to my supervisor Mr. Shariman Bin Abdullah for his constructive guidance, encouragement and patient in fulfilling our aspiration in completing this project, to my sister, and all my friends. I also would like to say thanks for everything. The work and success will never be achieved without all of you.

(8)

ACKNOWLEDGEMENT

I would like thank to my supervisor Mr. Shariman Abdullah for helping, guide me and encourage me to finish this motion tracking project. Without guide from my supervisor I could not finish this project successfully. Then I would like to thank to my entire friend that give me a guide to finish the report with the correct format and teach me how to write in English with correct grammar. Not forget I would like to thank to my parent that not tired give me encouragement to finish my studies and always be my backbone of my successful. Lastly I would like to thank to all that directly or indirectly involve finishing this project together with me.

(9)

TABLE OF CONTENT

Abstract i

Abstrak ii

Dedication iii

Acknowledgement iv

Table of Content v

List of Tables viii

List of Figures ix

List Abbreviations xii

1 INTRODUCTION 1

1.1 Background 1

1.2 Motion Tracking 1

1.3 Open Computer Vision 2

1.4 Problem Statement 2

1.5 Research Objectives 3

1.6 Research Scopes 3

1.7 Importance of the Research 3

1.8 Gantt Chart of the Motion Tracking 4

2 LITERATURE REVIEW 6

2.1 Background 6

2.2 Motion Tracking 6

2.2.1 Definition to Motion Tracking 7

2.3 Motion Tracking Techniques 9

2.3.1 Optical Flow Based 10

2.3.1.1 Tracking with Background Subtraction 12

(10)

2.3.2.1 Color Based Motion Tracking Technique (The RGB Space) 17 2.3.2.2 Color Based Motion Tracking Technique (The Greyscale Space) 18 2.3.2.3 Color Based Motion Tracking Technique (The Hue, Saturation and

Lightness Space)

2.3.2.4 Color Based Motion Tracking Technique (Chromatic Mapping) 20

2.4 Feature Extraction and Image Processing 21

2.4.1 Image Filtering 21

2.5 Previous Related Work 23

2.5.1 Color, texture, and motion in level set based segmentation and tracking

2.5.2 Object tracking using SIFT features and mean shift 28 2.5.3 Reliable and Fast Human Body Tracking under Information

Deficiency

2.5.4 Detection and Tracking of Human Subjects 37

2.5.5 Region-based Pose Tracking 42

2.5.6 Statistical Cue Estimation for Model-Based Shape and Motion 46 2.5.7 A Hybrid Color-Based Foreground Object Detection Method for

Automated Marine Surveillance

2.5.8 Tracking a Tennis Ball Using Image Processing Techniques 55

2.6 Open Computer Vision 62

2.6.1 Open Computer Vision Implementation 62

3 METHODOLOGY 65

3.0 Background 65

3.1 Flow Chart of the Process 66

3.2 Motion Tracking Methodology 67

3.3 Optical Flow 67

3.3.1 Background Subtraction 68

3.3.1.1 Edge Detection 69

(11)

vii

3.4.1 Red, Green, Blue (RGB) 70

3.4.1.1 Segmentation 71

3.5 Mean Shift (camshift) 72

3.6 Motion Tracking Technique Combination 73

4 DEVELOPMENT 75

4.0 Introduction 75

4.1 Flowchart 75

4.2 Camshift 77

4.3 Back Projection 78

4.4 Histogram 79

4.5 Color Converting 81

4.6 Object Tracking In OpenCV 82

4.7 The Procedure 83

5 RESULT AND DISCUSSION 87

5.0 Background 87

5.1 Moving Slowly Toward and Away From the Camera 87

5.2 Swaying Side to Side 88

5.3 Rotational 89

5.4 All Over the Camera with Back Projection 90

6 CONCLUSION AND RECOMMENDATION 91

6.0 Background 91

6.1 Conclusion 91

6.2 Recommendation 92

(12)

viii

LIST OF TABLES

1.1 Gantt Chart for Motion Tracking Project PSM 1 4

1.2 Gantt Chart for Motion Tracking Project PSM 2 5

(13)

LIST OF FIGURES

2.1 The Class of the Motion Tracking 7

2.2 The Motion Tracking of Face Detection 8

2.3 The Object (face) Has Been Identified and Recognized 9

2.4 The Original Image 16

2.5 The Segmented Target Object 17

2.4 The RGB Cube 17

2.5 The Grayscale Space 18

2.6 The HSV Cone 19

2.7 The HSL Cone Inside the RGB Cube 20

2.8 The Chromatic Mapping 21

2.9 Show the Texture has been Applied 24

2.11 Segmentation Results for Synthetic Texture Images 25 2.12 Segmentation Results with Some Real-World Images Using Texture and

Color Information

2.13 Benefits of Using Joint Color and Texture Information 26 2.14 Tracking of Small Objects in a Cutout of the Traffic Sequence 26

2.15 Tracking of Three Overlapping Balls 27

2.16 Tracking of Three Players in a Soccer Sequence 27

2.17 Test Sequences Used in Current Evaluation 29

2.18 Tracking Comparison of the Classical Mean Shift 31 2.19 Tracking Comparison of the Classical Mean Shift 32 2.20 Performance Comparisons of Classical Mean Shift 33

2.21 Output Images 35

2.22 Face Detection Result 36

2.23 The Image of Walking Woman has been Change to Gray Scale 38 2.24 Subtracting Background Frame from Current Frame 39 2.25 Binary Image Binary image Generated with a Threshold of 8%. 40

(14)

2.26 Center-of-Mass The first Frame with the Center-of-Mass Displayed 41 2.27 Tracking Rectangle The same Frame with the Rectangle Visualization 42 2.28 Problems Often Occurring in Variation Segmentation Algorithms 43

2.29 The Giraffe Images has been Detect 44

2.30 The Images of Box 45

2.31 An Example of the Results of a Canny Edge Detection and the Distance Field of the Edges

2.32 Sequence of the Motion Tracking 49

2.32 The Sequence of Motion Tracking Using the Canon Camera with Cue Algorithm Method

2.32 Block Diagram of the Proposed Foreground Segmentation System 52 2.33 The Experiment Performed on a Video Containing Typical Marine

Surveillance Footage

2.34 Results Obtained for Representative Frames 54

2.35 The Sample of Background Image has been Done the Edges 56

2.36 The Foreground Object Detection Algorithm 56

2.37 The Sequence of the Object Detection 57

2.38 The Illustrated of Segmentation Approach 58

2.39 The Sequence Image Tennis Ball Detection Using the Colour Segmentation Approach

2.40 The Image of Tennis Ball with Shape Recognition Approach 60

2.41 The Illustrated of the Classifier Creation 61

2.42 Image of Tennis Ball Detection Using the Classifier Creation 61

2.43 The Example of the Threshold Processing 63

3.1 Methodology Motion Tracking Flow Chart 67

3.2 The Example of the Background Subtraction 69

3.3 The Result of the Edge Detection 70

3.4 The RGB Technique 72

(15)

3.6 An example of the camshift face tracking 73

3.7 Example of the Visual Studio C++ 2008 Express Edition with C++

(16)

xii

LIST OF ABBREVIATIONS

2D - Two Dimension

3D - Three Dimension

EKF - Extended Kalman filter

fps - frames per second

HS - Hue and Saturation

HSL - hue, saturation, and lightness

HSV - hue, saturation, and value

IEKF - Iterative Extended Kalman Filter

IIR - Impulse Response

OpenCV - Open Computer Vision

RGB - Red, Blue, and Green

(17)

CHAPTER 1

INTRODUCTION

1.1Background

In this chapter will describes and simple briefly about the motion tracking and OpenCV definition. In this chapter also include the problem statement of the motion tracking, objective of this project, research scopes, and lastly important of the research.

1.2Motion Tracking

What is motion tracking? Motion tracking is the process where the vision of image or videos detected. The movement on the camera vision range will be tracking. Many things can be used of motion tracking functional. Below statement is the definition of the motion tracking.

Motion tracking also used in media sectors. According to J. Sturman, 2001, motion tracking is the recording of human body movement for analysis and playback. The information of the tracking can be as simple as the position of the body in space or as complex as the deformations of the face and muscle masses. Motion tracking for computer character animation involves the mapping of human motion onto the motion of a computer character. The mapping can be direct, such as human arm motion controlling a character's arm motion, or indirect, such as human hand and finger patterns controlling a character's skin color or emotional state.

(18)

2 1.3Open Computer Vision

The history, in 1999 Open Computer Vision (OpenCV) was started at Intel create by Gary Bradski by purposes of accelerating research and commercial applications of computer vision. OpenCV is one of kind software installed to create, to program, and to run with a device to process an image or videos. OpenCV can be viewed as signal processing applied to higher dimensions of image and videos. OpenCV library usually used the C and C++ to programming the database. The OpenCV is optimized and intend for real time applications.

1.4Problem Statement

The motion tracking is not completely used in the autonomous robotic technologies. The motion tracking other problem is to implement to the future technologies where it can be detected and tracking the motion of the object or human. Then motion tracking methodologies applied to the real time world also is the problem, how the motion tacking can be done with the right method and can be implementing to the real time functionality. Next, the motion tracking functionality, what must be detected, how wants to detected, for a moving object which is, the robot car, color detection and thus recognition of the color object is of a more flexible concept where no database is inserted instead the object is recognized through a predefined color. On the same basic, they hold the same idea that is, to recognize a moving object.

1.5Research Objectives

The objective of the project is to track and detect the motion or movement of object selected. Means the object selected must be followed to anywhere of the movement of selected object as long as the selected object in webcam viewer range parameters. Besides of that the motion tracking of object hopefully is successfully.

(19)

1.6Research Scopes

The motion tracking have many type and ways to be done. However for this project the scope of project is to do the motion tracking of object based on the colours based technique, the camshift technique with the histogram and back projection also the feature extraction image processing technique for the this project of motion tracking methodology.

1.7Importance of the Research

Research processes is to Programming the motion tracking by using the OpenCV software. The research will help student to understand about motion tracking. Then apply to the real time function maybe as a security system to track and detect the movement of suspicious object. The motion tracking also can be applied in industrial function where the technologies of motion tracking can be as a helpful system to track and detect the production or anything. The motion tracking also is used in automation and robotic sector. Nowadays the motion tracking mostly can be applied to the army system for defense of country. For example satellite communication to detect the stranger or opponents movement.

(20)

4 1.8Gantt Chart of the Motion Tracking

Table 1.1: Gantt chart for motion tracking project PSM 1

History Actual

(21)

Table 1.2: Gantt chart for motion tracking project PSM 2

History Actual

(22)

CHAPTER 2

LITERATURE REVIEW

2.1 Background

This chapter will discuss and review available literature on Motion Tracking using OpenCV software. The reviews begin with motion tracking. This section will be including the motion tracking technique and related previous projects. Then, the OpenCV will be discussed in this chapter.

2.2 Motion Tracking

This title will discuss about the motion tracking definition. The motion tracking technique also will be describe in this topic. The technique will be divide into a few sub topic depend on the scope of the motion tracking research. And many more will be discuss in this topic.

A central thread of computer vision research is the development of algorithm or system to track the position and orientation of a target object or objects within images or image sequences. Object tracking, while a simple task for humans is monumentally more challenging for computer vision systems. Over the years, a vast number of algorithms have been proposed for object tracking, and there are large numbers of applications that

(23)

require such algorithms to track different target in different conditions (Maurin et al. 2005). For example, to guide an autonomous vehicle in a simple or complex environments (Kia and Arshad 2005, and Asif et al. 2005) or it may be used to track vehicle for collecting the traffic data from highway scenes (Kastrinaki et al. 2003) or even to detect human in a surveillance system (Collins et al. 2000a). Tracking may also be used in robot arm applications either to provide guidance to surgical robot (Ginhoux et al. 2003, and Zhang and Payandeh 2002) or to select an optimal grasp for picking-up object (Han and Kuc 1998). General techniques for tracking are independent from any particular application.

2.2.1 Definition to Motion Tracking

According to the Anonymous, (1991) the motion tracking definition is to track an object over a sequence of images. Object tracking in general, is a challenging problem. Difficulties in tracking objects can arise due to abrupt object motion, changing appearance patterns of the object and the scene, non-rigid object structures, object to object and object to scene occlusions, and camera motion. Tracking is usually performed in the context of higher-level applications that require the location or shape of the object in every frame. Generally there are three types of tracking algorithm first is point tracking, then kernel tracking and third is silhouette tracking. The type of the motion tracking is shown in figure below. The object tracking is the general and main title then is divided into three types.

(24)

Paulus, (2009) said that the aim of motion tracking is to detect and track moving objects through a sequence of images. Motion tracking is not only useful for monitoring activity in public places, but is becoming a key ingredient for further analysis of video imagery. For instance, information about the location and identity of objects at different points in time is the basis of detecting unusual object movements or coordinated activities. With motion tracking, you can track the movement of an object and then apply the tracking data for that movement to another object such as another layer or an effect control point to create compositions in which images and effects follow the motion.

Matsuyama, (2008) believe that the motion tracking is a common requirement for many real world applications, such as video surveillance, games, cultural and medical applications example like using for motion and behavior study. Simple object can be detected and tracked using various image features such as color regions, edges, contours, or texture. In the other hand, complex objects such as human faces require more sophisticated features to handle the multiple possible instances of the object class. Statistical methods are a good alternative. A statistical model learns different patterns related to the object of interest like motion tracking of the different views of human faces, including good and bad samples. And then the system is able to estimate whether a region contains an object of interest or not.

Figure 2.2: Show that the motion tracking of face detection. The red box is the box of reorganization progress where the face have been detect and not identified yet. The green box shows that the object has

(1)