A Robot Learns How to Entice an Insect
ROBOTICS
A Robot Learns How
to Entice an Insect
Ji-Hwan Son and Hyo-Sung Ahn, Gwangju Institute of Science and Technology
R
obot technology has achieved significant growth in recent years, to the
point where many expect robots to interact seamlessly with human envi-
ronments. To perfect this ability, robots must recognize various elements, make
feasible decisions, and conduct proper actuations. However, the real world
consists of uncertain and unpredictable elements that make it difficult for robots to
properly interact with our environment. The
key issue in this challenging task is linked to
Using a camera to
how a robot learns: it can learn by imitation
recognize a biological or demonstration from its own observation
or teacher,1,2 through trial and error, 3 or
insect and its heading via its own evolutionary skill4 or reasoning
ability based on acquired data.5 But regardangle, a robot can
less of learning method, the robot needs the
ability to recognize and concentrate on a
spread a specific
target, make a decision on how to achieve
the optimal outcome, and exercise control
odor source to entice to execute its chosen actions. On top of this,
the robot should consistently and continuthe insect to travel a
ously acquire useful knowledge as part of its
learning process.
given trajectory.
Motivated by these challenges, we were
curious as to whether a learning capability could be embedded into a robot. It’s
extremely difficult to realize a robot with
learning abilities in a real environment because the robot could be affected by that
environment as it learns. Moreover, the environment could be changed through its
interactions with the robot and by other
stimulations, such as wind, flavor, and temperature. Thus, instead of providing a universal solution for general robot intelligence,
we seek to implement a learning ability in
54
an ideal situation. Specifically, we studied
the interaction between a robot and a living, biological insect in a regulated space,
with the robot spreading a specific odor to
indirectly attract the insect. Unlike humans,
insect behavior is much simpler, making
it relatively easy for a robot to understand
and interact with. Nevertheless, an insect’s
behavior includes uncertain and unpredictable elements—it has enough intelligence to
survive in nature—so this wasn’t a trivial
problem.
Various researchers have studied the interaction between robots and animals that
we can divide into two classes. The first is
physical contact–based interaction—for example, researchers have installed electrodes
into the nervous systems of a moth,6,7 a beetle, 8 and a cockroach,9 and then controlled
the insects’ motions via electric stimuli. The
second class is indirect stimuli–based interaction—robots rely on indirect stimuli, such
as using sex pheromones to interact with
a moth,10 using pheromones to influence
a group of cockroaches,11 and interacting
with a living cricket via internally embedded cricket substances.12 Further examples
of indirect stimuli-based interaction include
mobile robot movement achieved by dragging a flock of ducks toward a specific goal
1541-1672/15/$31.00 © 2015 IEEE
Published by the IEEE Computer Society
IEEE INTELLIGENT SYSTEMS
position13 and keeping chickens via
an algorithm.14 Specific locomotion
behaviors using an animal-shaped robot can affect socialization in fish15
and interaction with rats,16 and visual stimuli can control a beetle’s
fl ight direction via LEDs attached to
its head.17 Attaching movable cylinders to a turtle’s shell can affect its
movement: the obstacle moves close
to the turtle’s head, making the turtle
move as desired to avoid it.18 However, all these interaction mechanisms
depend on programmed commands
or human operators.
Some of our earlier works studied insect and robot interactions.19,20
In our previous experiments, we assumed that the insect’s position and
heading angle were exactly known
from a camera attached to the top of
a platform, and the robot only needed
to entice the insect to a specific spot
in a defi ned area. However, in this article, the robot must fi nd the insect
using a camera attached to the robot
itself to recognize the insect’s position and heading angle. The robot
needs to know the insect’s position at
all times to entice it to a predefi ned
trajectory. Previous experiments
used fuzzy-logic-based reinforcement
learning and expertise measurement
for cooperative learning, but here, we
entice the insect to follow the given
trajectory by using hierarchical reinforcement learning.
Methodologies
In our case, reinforcement learning 21,22 means a trial-and-error-based
algorithm inspired by the learning
behavior of animals. To implement
reinforcement learning, we use Qlearning, 23 which is an optimal action-selection policy algorithm used
in reinforcement learning. This tool
is most suitable for model-free systems, such as that of an insect. The
main Q-learning equation is
JULY/AUGUST 2015
Q(s, a) ← (1 − a )Q(s, a)
+ a (t + Γ maxQ(sˆ , aˆ )) ,
(1)
aˆ
where a is the learning rate (0 ≤ a <
1), Γ is the discount factor (0 ≤ Γ < 1),
and t is the immediate reward.
From the current state, the robot
fi nds the next state under action a to
maximize the Q-value, which is the
total expected accumulated reward.
Based on the learning algorithm, the
robot fi nds its own goal and updates
the state to complete the exploitation
process. However, to efficiently fi nd a
goal, the learning process frequently
requires the robot to choose an action
A wireless camera
attached to the robot
detects the insect and
computes its position and
heading information with
respect to the robot.
randomly to explore the states (the
exploration process). Thus, the exploitation and exploration processes
have some trade-off issues. As a criterion for selecting between the two,
the learning process generates a random value e in every repetition, where
0 ≤ e < 1. If the random value is less
than the predefi ned e value, the learning process chooses an action randomly—if not, the learning process
chooses an action to maximize the
Q-value. Here, the predefi ned value e
decreases with the increasing number
of iterations, meaning that the probability of choosing a random action
also decreases.
www.computer.org/intelligent
To set up our experimental environment, we built the platform illustrated in Figures 1b and 1d, which
is 196 cm × 147 cm and contains a
camera (1,024 × 768 resolutions) and
a computer. To verify that the robot
can entice the insect in various directions, we chose a trajectory composed
of two circles (see Figure 1c). The robot attempts to entice the insect toward the left direction in the left
circle, the right direction in the right
circle, and forward at the intersection
point. Using this trajectory, the robot
learns how to entice the insect in various ways through repetition.
To entice the insect along the trajectory, the robot needs to know its
own position, so we attached a camera that faces the experimental platform to the ceiling. A wireless camera
attached to the robot detects the insect and computes its position and
heading information with respect to
the robot. In its interaction with the
insect, the robot’s size is a crucial factor—we needed a robot whose size
was similar to the insect’s, which
meant the robot couldn’t equip its
own computing structure. This is
why we borrow those abilities from
the host computer, which controls the
robot’s movement remotely by wireless signal, conducts real-time image processing, and stores acquired
data. Humans were only involved in
constructing the robot, the experimental platform, and any related
programs in advance. All the robot’s
processes, such as recognition, decision, learning, and control, were fully
performed autonomously during the
experiment without human aid.
We chose two types of living stag
beetles for the insect: Dorcus titanus castanicolor and Dorcus hopei
binodulosus (see Figure 1a). These
insects are strong enough to endure
several experiments, have good mobility over flat surfaces, and exhibit
55
ROBOTICS
Landmark
Wireless
camera
Insect
1 cm
Battery
(a)
(b)
Odor
source
Servo
motors
Air duct
Camera
Moving
direction
Trajectory
Wireless
LAN card
147 cm
Experimental
platform
Pump
motor
28.7 cm
(c)
Insect
Computer
Artificial
robot
Moving
direction
Experimental
platform
(d)
Figure 1. Experimental setup. (a) The stag beetles: Dorcus titanus castanicolor (left) and Dorcus hopei binodulosus (right);
(b) the robot, which contains a wireless camera mounted on servo motors to detect the insect, two servomotors to help the
camera track the insect, two air pump motors to spread the odor source, an e-puck robot to move onto specific positions, a
landmark for the robot to detect its own position, and a Li-Po battery; (c) the experimental platform and the shape of the given
trajectory; and (d) the experimental environment. To entice the insect on the trajectory, the robot needs position data, so a
camera that faces the experimental platform is attached to the ceiling.
a two- to three-year life span. To determine interaction mechanisms between the insect and the robot, we
performed various experiments using stimuli such as light, vibration,
air flow, robot movement, physical
contact, and sound. The insects’ reactions to these variables weren’t strong
enough to achieve our goal, but we
observed that they used three groups
of antennae on their heads to monitor the environment. After conducting more experiments, we found that
these insects react strongly to the specific odor of sawdust from their own
habitats.19
To get the robot to entice the insect
through odor, we equipped it with
56
two air pump motors and two bottles
containing specific odor sources. The
wireless camera mounted on the two
servomotors watches and tracks the
insect under study to recognize and
track it in real time. The air pump
motors and servomotors are controlled by the Atmega 128 microprocessor; a 7.4-v Li-Po battery supplies
electricity to the whole robot system.
the insect’s position and heading angle (see Figure 2). Based on the acquired position data of the robot (rx,
r y), the insect position is calculated as
Experiment
At the beginning, the robot doesn’t
know where the insect is: it tries to
find it by rotating its heading and
increasing the wireless camera’s
elevation angle. If the robot finds the
insect, it approaches and recognizes
where r2 = l cosq3r, h2 = l sinq3r, r3 =
www.computer.org/intelligent
bx = rx + r1 cos q1r
+ (r2 + r3) cos(q1r + q 2r )
and
by = ry + r1 sin q1r
+ (r2 + r3) sin(q1r + q 2r )
,
h1 + h2
tan(90 − q3r )
,
q1r is the robot’s heading angle, q 2r
and q3r are the camera’s azimuth and
elevation angle, and h1, h2 , l, r 1, r 2 ,
and r 3 are the distance values, as illustrated in Figures 2a and 2b.
IEEE INTELLIGENT SYSTEMS
m
bt
Insect
d ≥m
r2+r3
r
θ2
d
θ 1r
θ
bt
d
A Robot Learns How
to Entice an Insect
Ji-Hwan Son and Hyo-Sung Ahn, Gwangju Institute of Science and Technology
R
obot technology has achieved significant growth in recent years, to the
point where many expect robots to interact seamlessly with human envi-
ronments. To perfect this ability, robots must recognize various elements, make
feasible decisions, and conduct proper actuations. However, the real world
consists of uncertain and unpredictable elements that make it difficult for robots to
properly interact with our environment. The
key issue in this challenging task is linked to
Using a camera to
how a robot learns: it can learn by imitation
recognize a biological or demonstration from its own observation
or teacher,1,2 through trial and error, 3 or
insect and its heading via its own evolutionary skill4 or reasoning
ability based on acquired data.5 But regardangle, a robot can
less of learning method, the robot needs the
ability to recognize and concentrate on a
spread a specific
target, make a decision on how to achieve
the optimal outcome, and exercise control
odor source to entice to execute its chosen actions. On top of this,
the robot should consistently and continuthe insect to travel a
ously acquire useful knowledge as part of its
learning process.
given trajectory.
Motivated by these challenges, we were
curious as to whether a learning capability could be embedded into a robot. It’s
extremely difficult to realize a robot with
learning abilities in a real environment because the robot could be affected by that
environment as it learns. Moreover, the environment could be changed through its
interactions with the robot and by other
stimulations, such as wind, flavor, and temperature. Thus, instead of providing a universal solution for general robot intelligence,
we seek to implement a learning ability in
54
an ideal situation. Specifically, we studied
the interaction between a robot and a living, biological insect in a regulated space,
with the robot spreading a specific odor to
indirectly attract the insect. Unlike humans,
insect behavior is much simpler, making
it relatively easy for a robot to understand
and interact with. Nevertheless, an insect’s
behavior includes uncertain and unpredictable elements—it has enough intelligence to
survive in nature—so this wasn’t a trivial
problem.
Various researchers have studied the interaction between robots and animals that
we can divide into two classes. The first is
physical contact–based interaction—for example, researchers have installed electrodes
into the nervous systems of a moth,6,7 a beetle, 8 and a cockroach,9 and then controlled
the insects’ motions via electric stimuli. The
second class is indirect stimuli–based interaction—robots rely on indirect stimuli, such
as using sex pheromones to interact with
a moth,10 using pheromones to influence
a group of cockroaches,11 and interacting
with a living cricket via internally embedded cricket substances.12 Further examples
of indirect stimuli-based interaction include
mobile robot movement achieved by dragging a flock of ducks toward a specific goal
1541-1672/15/$31.00 © 2015 IEEE
Published by the IEEE Computer Society
IEEE INTELLIGENT SYSTEMS
position13 and keeping chickens via
an algorithm.14 Specific locomotion
behaviors using an animal-shaped robot can affect socialization in fish15
and interaction with rats,16 and visual stimuli can control a beetle’s
fl ight direction via LEDs attached to
its head.17 Attaching movable cylinders to a turtle’s shell can affect its
movement: the obstacle moves close
to the turtle’s head, making the turtle
move as desired to avoid it.18 However, all these interaction mechanisms
depend on programmed commands
or human operators.
Some of our earlier works studied insect and robot interactions.19,20
In our previous experiments, we assumed that the insect’s position and
heading angle were exactly known
from a camera attached to the top of
a platform, and the robot only needed
to entice the insect to a specific spot
in a defi ned area. However, in this article, the robot must fi nd the insect
using a camera attached to the robot
itself to recognize the insect’s position and heading angle. The robot
needs to know the insect’s position at
all times to entice it to a predefi ned
trajectory. Previous experiments
used fuzzy-logic-based reinforcement
learning and expertise measurement
for cooperative learning, but here, we
entice the insect to follow the given
trajectory by using hierarchical reinforcement learning.
Methodologies
In our case, reinforcement learning 21,22 means a trial-and-error-based
algorithm inspired by the learning
behavior of animals. To implement
reinforcement learning, we use Qlearning, 23 which is an optimal action-selection policy algorithm used
in reinforcement learning. This tool
is most suitable for model-free systems, such as that of an insect. The
main Q-learning equation is
JULY/AUGUST 2015
Q(s, a) ← (1 − a )Q(s, a)
+ a (t + Γ maxQ(sˆ , aˆ )) ,
(1)
aˆ
where a is the learning rate (0 ≤ a <
1), Γ is the discount factor (0 ≤ Γ < 1),
and t is the immediate reward.
From the current state, the robot
fi nds the next state under action a to
maximize the Q-value, which is the
total expected accumulated reward.
Based on the learning algorithm, the
robot fi nds its own goal and updates
the state to complete the exploitation
process. However, to efficiently fi nd a
goal, the learning process frequently
requires the robot to choose an action
A wireless camera
attached to the robot
detects the insect and
computes its position and
heading information with
respect to the robot.
randomly to explore the states (the
exploration process). Thus, the exploitation and exploration processes
have some trade-off issues. As a criterion for selecting between the two,
the learning process generates a random value e in every repetition, where
0 ≤ e < 1. If the random value is less
than the predefi ned e value, the learning process chooses an action randomly—if not, the learning process
chooses an action to maximize the
Q-value. Here, the predefi ned value e
decreases with the increasing number
of iterations, meaning that the probability of choosing a random action
also decreases.
www.computer.org/intelligent
To set up our experimental environment, we built the platform illustrated in Figures 1b and 1d, which
is 196 cm × 147 cm and contains a
camera (1,024 × 768 resolutions) and
a computer. To verify that the robot
can entice the insect in various directions, we chose a trajectory composed
of two circles (see Figure 1c). The robot attempts to entice the insect toward the left direction in the left
circle, the right direction in the right
circle, and forward at the intersection
point. Using this trajectory, the robot
learns how to entice the insect in various ways through repetition.
To entice the insect along the trajectory, the robot needs to know its
own position, so we attached a camera that faces the experimental platform to the ceiling. A wireless camera
attached to the robot detects the insect and computes its position and
heading information with respect to
the robot. In its interaction with the
insect, the robot’s size is a crucial factor—we needed a robot whose size
was similar to the insect’s, which
meant the robot couldn’t equip its
own computing structure. This is
why we borrow those abilities from
the host computer, which controls the
robot’s movement remotely by wireless signal, conducts real-time image processing, and stores acquired
data. Humans were only involved in
constructing the robot, the experimental platform, and any related
programs in advance. All the robot’s
processes, such as recognition, decision, learning, and control, were fully
performed autonomously during the
experiment without human aid.
We chose two types of living stag
beetles for the insect: Dorcus titanus castanicolor and Dorcus hopei
binodulosus (see Figure 1a). These
insects are strong enough to endure
several experiments, have good mobility over flat surfaces, and exhibit
55
ROBOTICS
Landmark
Wireless
camera
Insect
1 cm
Battery
(a)
(b)
Odor
source
Servo
motors
Air duct
Camera
Moving
direction
Trajectory
Wireless
LAN card
147 cm
Experimental
platform
Pump
motor
28.7 cm
(c)
Insect
Computer
Artificial
robot
Moving
direction
Experimental
platform
(d)
Figure 1. Experimental setup. (a) The stag beetles: Dorcus titanus castanicolor (left) and Dorcus hopei binodulosus (right);
(b) the robot, which contains a wireless camera mounted on servo motors to detect the insect, two servomotors to help the
camera track the insect, two air pump motors to spread the odor source, an e-puck robot to move onto specific positions, a
landmark for the robot to detect its own position, and a Li-Po battery; (c) the experimental platform and the shape of the given
trajectory; and (d) the experimental environment. To entice the insect on the trajectory, the robot needs position data, so a
camera that faces the experimental platform is attached to the ceiling.
a two- to three-year life span. To determine interaction mechanisms between the insect and the robot, we
performed various experiments using stimuli such as light, vibration,
air flow, robot movement, physical
contact, and sound. The insects’ reactions to these variables weren’t strong
enough to achieve our goal, but we
observed that they used three groups
of antennae on their heads to monitor the environment. After conducting more experiments, we found that
these insects react strongly to the specific odor of sawdust from their own
habitats.19
To get the robot to entice the insect
through odor, we equipped it with
56
two air pump motors and two bottles
containing specific odor sources. The
wireless camera mounted on the two
servomotors watches and tracks the
insect under study to recognize and
track it in real time. The air pump
motors and servomotors are controlled by the Atmega 128 microprocessor; a 7.4-v Li-Po battery supplies
electricity to the whole robot system.
the insect’s position and heading angle (see Figure 2). Based on the acquired position data of the robot (rx,
r y), the insect position is calculated as
Experiment
At the beginning, the robot doesn’t
know where the insect is: it tries to
find it by rotating its heading and
increasing the wireless camera’s
elevation angle. If the robot finds the
insect, it approaches and recognizes
where r2 = l cosq3r, h2 = l sinq3r, r3 =
www.computer.org/intelligent
bx = rx + r1 cos q1r
+ (r2 + r3) cos(q1r + q 2r )
and
by = ry + r1 sin q1r
+ (r2 + r3) sin(q1r + q 2r )
,
h1 + h2
tan(90 − q3r )
,
q1r is the robot’s heading angle, q 2r
and q3r are the camera’s azimuth and
elevation angle, and h1, h2 , l, r 1, r 2 ,
and r 3 are the distance values, as illustrated in Figures 2a and 2b.
IEEE INTELLIGENT SYSTEMS
m
bt
Insect
d ≥m
r2+r3
r
θ2
d
θ 1r
θ
bt
d