R
oc
= R
κ
· R
ω
· R
α
, 1
where R
α
– rotation matrix around the axis Y , R
ω
– rotation ma-
trix around the axis X, R
κ
– rotation matrix around the axis Z.
dX Y
r1
X
r1
Y
r2
X
r2
X
O
Y
O
Figure 2. Coordinate systems
3.2 Algorithm pipeline
The algorithm consists of four main steps. Firstly optical flow is estimated using infrared and visible spectrum image sequences.
Secondly robot’s motion is estimated using the data from robot’s encoders and the state-space model. The block diagram of the
algorithm is presented in figure 3. 3.3 Multispectral optical flow fusion
The estimation of multispectral optical flow is performed sepa- rately for images in visible spectrum and infrared images. Vari-
ous optical flow estimations algorithms were evaluated Revaud et al., 2015, Liu, 2009, Farnebäck, 2003, Dosovitskiy et al., 2015.
The FlowNet deep neural network based algorithm showed the best performance during the evaluation of the algorithms using
the ground truth optical flow. However only computational com- plexity of the algorithm proposed in Farnebäck, 2003 was low
enough for implementation on a single-board computer. Various methods of optical flow fusion were studied. Firstly
the most simple fusion model based on the selection of maximum flow among the channels were considered. Let
u
T V
= u
T V
, v
T V
be the visible spectrum optical flow,
u
IR
= u
T V
, v
T V
infrared optical flow. Then the maximum fusion method is given by
u =
maxu
T V
, u
IR
, v = maxv
T V
, v
IR
2 Also fusion method based on the mean value of the optical flows
was studied. The mean fusion method is given by u
= u
T V
+ u
IR
2 , v
= v
T V
+ v
IR
2 3
Finally the adaptive relay threshold method introduced in Man et al., 2007 was studied. The fused optical flow for the relay
method is given by A
i
= ∑
j∈N
u
2 T V j
+ v
2 T V j
u
2 IRj
+ v
2 IRj
4 Figure 3. Block diagram of the BipedDetect algorithm
u
i
= {
max u
T V
, u
IR
if A
i
≥ T
, min
u
T V
, u
IR
if A
i
T All three methods were implemented in the dedicated software
and their performance was studied using the BipedFlow dataset. The evaluation have showed that the fusion based on the maxi-
mum has the minimal root mean square error.
3.4 State-space model of the mobile robot
A state-space model of the Hercules robot was developed during previous research Kniaz, 2015, Kniaz, 2016. The state-space
dynamic model of the robot is given by: ˙
Xt = A · Xt + B · Ut
5
Yt = C · Xt + D · Ut 6
Where output vector Y t of the robot consists of longitudinal speed v
x
and rotational speed ˙ϑ:
Yt =
[v
x
˙ ϑ
] The input vector Ut consists of average and difference motor
speeds for left and right wheel motors:
Ut =
[a d
] a
= p
R
+ p
L
2 7
This contribution has been peer-reviewed. doi:10.5194isprs-archives-XLII-2-W4-73-2017
75
Algorithm 1: BipedDetect Input: Camera external orientation parameters for each frame
X
oc1
, X
oc2
, R
oc1
, R
oc2
, images in visible and infrared
spectrum I
i T V
, I
i IR
.
Output: Bounding boxes for detected pedestrians. for i := 1 to N
f rames
do Optical flow loop Estimate optical flow u
i T V
I
i−1 T V
, I
i T V
,
u
i IR
I
i−1 IR
, I
i IR
;
Perform the fusion of u
i T V
and u
i IR
using eq. 2 to
obtain u
i
;
Generate the synthetic optical flow u
i m
using eq. 9 and
X
oc1
, X
oc2
, R
oc1
, R
oc2
;
u
i d
= u
i
- u
i m
;
Threshold u
i d
using Otsu’s method to obtain u
i t
;
Perform the segmentation of u
i t
to obtain the bounding boxes for moving pedestrians;
end
d =
p
R
− p
L
2 8
Where p
R
, p
L
- are motor speeds for right and left wheel motors.
3.5 Synthetic optical flow generation The synthetic optical flow is generated using ray tracing. Let x
i p
be a projection of a point of the scene in i-th frame. Then the optical flow is given by
u
i m
=
x
i p
−
x
i−1 p
, 9
where x
i p
is given by
x
i−1 p
= µ
i−1
· R
i−1 oc
· X
p
− X
i−1
, 10
x
i p
= µ
i
· R
i oc
· X
p
− X
i
, 11
where X
p
– 3D coordinate of a point of the scene, µ
i
– scale factor,
X
i
– position of the camera. The difference of two last equations
provides the final equation for the synthetic optical flow u
i m
.
3.6 Pedestrian detection With the given synthetic optical flow u
i m
and a real optical flow
u
i
their difference u
i d
is calculated
u
i d
=
u
i
−
u
i m
. 12
As u
i d
doesn’t contain optical flow induced by robot’s egomotion pedestrians could be detected by thresholding the image using
Otsu’s method. The final bounding boxes for the pedestrians are provided by segmentation. The BipedDetect algorithm is sum-
marised in the algorithm 1.
4. ALGORITHM EVALUATION
