identification was based on shape analysis and since surface texture and color were irrelevant, background lighting could be used to improve the contrast between the
fish image and background and thus improve image segmentation. Following that work it was decided to test the ability of the method to identify
the three species while the fish were alive. Rather than inserting a camera into the water in a waterproof casing, fish were placed in an aquarium and their images
were taken by a CCD camera mounted outside. In a possible practical application fish will have to be ‘delivered’ into the field of
view of the inspection system a video camera and lighting system in the present case, preferably one at a time and side-on to the camera. Fish have been effectively
trained to approach feeders from long distances in both marine Fujiya et al., 1980 and freshwater environments Abbott, 1972; Levin and Levin, 1994 in response to
acoustic signals.
Sorting of large fish populations on the basis of fish performance depends upon the large-scale transfer of information from trained to naive fish, since only a small
fraction of the population can be effectively trained. Rapid transfer of information from trained to naive fish, based on imitation and social facilitation in mixed
schools of trained and naive fish has been demonstrated for various training tasks Hale, 1956; Fujiya et al., 1980; Suboski and Templeton, 1989. In the case of
Pagrus major, for example, which were trained in the laboratory to approach feeders and released in the ocean for sea ranching, trained fish were frequently
accompanied by naive specimens when captured Fujiya et al., 1980.
In addition to the development and testing of the computer vision part, a preliminary experiment was conducted to test the ability of fish to be trained to
swim through a narrow Plexiglas channel. Such a channel could be a part of a sorting system, through which fish images could possibly be acquired.
2. Materials and methods
2
.
1
. Species recognition Carp Cyprinus carpio, St. Peter’s fish Oreochromis sp. and grey mullet Mugil
cephalus were harvested from a local commercial fish farm and transported to the laboratory in containers. Fish mass ranges were 90 – 675 g, 80 – 210 g, and 15 – 70 g
for carp, St. Peter’s fish and grey mullet, respectively. The fish were kept in a plastic container approximately 500 l and moved in small groups between three and six
fish to a glass aquarium 150 cm long, 30 cm wide and 50 cm deep; 8 mm glass thickness for an imaging session.
Fish images were acquired by a computer vision system consisting of a colour video camera JVC ky-F30B 3CCD and a computer SUN SPARCstation IPX
equipped with a frame grabber Data Cell Limited, Model S2200 with 24-bit colour resolution. The camera viewed the aquarium from the side Fig. 1. On the
other side of the aquarium, between it and a white wall which formed the background, were mounted two pairs of standard 40 W fluorescent lamps, 120 cm
long.
The fish were allowed to swim freely in the aquarium and were viewed on a video monitor. When a fish was in the camera’s field of view, with its side to the camera
and without other fish in the foreground or background, its image was grabbed. When enough images had been taken, the fish were replaced by other fish for a new
imaging session. Since the laboratory was not equipped with means for holding each group of fish separately, they were returned to the plastic container with some
chance of the same fish being imaged again. Approximately ten fish of each of the three species were used in each imaging session of the experiment. More than a
single image of each fish was taken. Images which were grabbed while the fish were not side-on to the camera or which included other fish in the foreground or
background were discarded. This was done because the aquarium in which fish were swimming was large and the fish could have been in any possible orientation while
being imaged. In a future sorting system fish will have to be manipulated to swim through a narrow channel, with their side to the camera so that their typical shape
could be viewed.
The 143 acquired images 47 grey mullet, 43 St. Peter’s fish and 53 carp were divided into two sets: 20 grey mullet, 20 St. Peter’s fish and 20 carp images in one
set and the rest of the images in the other set. Each of these two sets was used as a training set for selection of feature thresholds which were then applied to the
other set for testing the performance in species recognition two-fold cross-valida- tion test. The average of the errors made by the classification procedure in
Fig. 1. Schematic of the experimental set-up used for live-fish imaging.
classifying the two test sets is an indication of the ability of the procedure to identify the three fish species.
Only the green band of the colour images was used by the image processing algorithm since there was no need for colour information and because it was
observed that the green band produced better contrast between the fish images and the background than the other two bands. Henceforth, this will be referred to as a
grey-scale image. Shape analysis procedures were applied to the binary fish images constructed from the grey-scale images, by applying a background thresholding and
labeling algorithm, as described below.
Assuming that fish images should always be fully within the camera field of view, a background sample of the first and last rows and columns constructing the
‘frame’ of the image was used for adaptive determination of the background threshold level for each image. Since image background was nonuniform, a
background grey level matrix was generated and used instead of a simple threshold value. The means and standard deviations of the grey values of each of the two
rows and two columns of the background frame were calculated and indexed as ‘left’, ‘right’, ‘up’ and ‘down’. A representative grey level for each of those sides was
calculated by subtracting since the background grey level was higher than that of the fish a fraction of the standard deviation from the corresponding mean e.g.
THleft = Meanleft − KStdevleft, where TH is the representative value and K is an empirical factor. The ‘left’ and ‘right’ representative values were used for
calculation of the horizontal distribution of the background grey level by simple linear interpolation. Similarly, the ‘up’ and ‘down’ representative values were used
for calculation of the vertical distribution of the background grey level. The value of background pixel i,j was calculated by averaging the two values calculated by
the horizontal and vertical interpolations for that pixel. The size of the background matrix matched the size of the image and the values of its elements were used as
background threshold values for the corresponding image pixels. K was initially set to 0, the background matrix was calculated and a standard labeling algorithm was
applied to the image. If the number of image segments labels was greater than N N was experimentally determined, K was increased to decrease the background
threshold values, the background matrix was recalculated and the labeling al- gorithm was reapplied. This iterative procedure was terminated when the number of
segments was less than or equal to N. At that stage, the largest image segment was considered to be the fish and the other segments were eliminated. A cropped binary
image, which was expected to contain only the fish image, was thus generated. The segmentation algorithm could be expressed as following:
compute grey scale average AV and standard deviation SD of first and last columns and rows l for left, r for right, u for up and d for down:
AVl, SD1, AVr, SDr, AVu, SDu, AVd, SDd
set N, the target number of binary image segments
for K = 0:0.1:2, compute representative threshold values for first and last columns and rows:
Fig. 2. Typical shapes of St. Peter’s fish A,B; carp C,D; and grey mullet E,F, while swimming in the aquarium.
THl = AVl − KSD1 THr = AVr − KSDr
THu = AVu − KSDu THd = AVd − KSDd
compute a background threshold matrix Bi,j whose dimensions N,M are the same to the original image:
Bi, j = 0.5 THr − THlj
N +
THu − THdi M
n
apply threshold values to original image pixels and create a binary image: apply labeling algorithm to binary image and count no. of segments, ns
if ns 0 N, – select the largest segment as the fish image and delete other segments
– stop else
end Fig. 2 shows typical images of the three fish species acquired while the fish were
swimming in the aquarium. It is clear that the typical shape of grey mullet is quite different from that of the other two species. It is a long and narrow fish whereas St.
Peter’s fish and carp have a higher width-to-length ratio. There are also some differences between the typical shapes of carp and St. Peter’s fish: their fins and tails
are different. However, with the live fish it was noted that sometimes their fins were wide open and other times tightly closed, therefore, the shape of the fish silhouette
could change drastically. It was also noted that with live fish in water, the fish tail clearly shows its characteristic shape in contrast to dead fish, whose tails are
sometimes bent or closed. It was, therefore, decided to use the shape of the whole fish for discrimination between grey mullet and the other two species, and to
analyze the shapes of the tails of carp and St. Peter’s fish to differentiate between these two species. Note that the tails of grey mullet are similar in shape to those of
carp, therefore, it was impossible to base species identification solely on tail shape.
Since fish come in different sizes and may approach the imaging system at different orientation angles, it was decided to use moment-invariants MI as the
shape features. Identification of grey mullet was based on the MI of the whole fish. Discrimination between carp and St. Peter’s fish was based on the MI of the tails.
In a live-fish sorting application images will be grabbed while the fish swim through a transparent channel in front of the camera, in a known direction. In this case the
orientation of the fish will obviously be head first. In the present work, however, fish were swimming freely in the aquarium and approached the field of view from
varying directions, so that the location of the tail was unpredictable. To overcome this problem, the fish angle was determined as described by Jain 1989 and its
length was calculated. The two ends of the fish image were cropped 10 of fish length on both head and tail ends, the MI of each end was calculated, and the
larger of the two values identified the tail Zion et al., 1999.
Once the tail of a fish was identified, two features were calculated and used separately by the classification procedure. The first feature was the square-root of
the sum of squares [F
t
= sqrtf
1t 2
+ f
2t 2
] of the two MI of the tail end. Higher order MIs, calculated from the third-order moments, were two orders of magnitude
smaller than the MIs calculated from the second-order moments. They also showed no significant difference between the species and therefore did not contribute to the
classification process. The second feature was involved with modeling the shape of the tail of a St. Peter’s fish as a rectangle whose aspect ratio widthlength was the
same as that of the tail segment which was extracted from the image Fig. 3. Using the expressions for moment-invariants Hu, 1962, the analytical expressions for the
first two MIs f
1
and f
2
of a rectangle whose length is ‘a’ and width ‘b’ are: f
1
= 1
12 a
b +
b a
and f
2
= 1
144 a
b −
b a
2
It was expected that the MI of a tail of a St. Peter’s fish would be close to that of a rectangle, whereas the MI of a tail of a carp was expected to be greater.
Therefore, the percentage of deviation df of the MI of a given tail f
t
from the MI of a rectangle f
r
with the same aspect ratio was calculated df = f
r
− f
t
f
r
100 and used as a classification feature. The aspect ratio was determined by counting the number of rows and columns within which the extracted tail segment
was included. The complete species recognition procedure was set as follows:
Calculate whole-body MI value f
1
from first order moments
If f
1
\ TH1 TH1 is the threshold value for grey mullet
identify fish as a grey mullet
Else crop 10 of fish length on both head and tail ends indexed h and t
calculate MI value for head and tail f
1h
, f
2h
,f
1t
and f
2t
identify tail side by [F
t
= sqrtf
1t 2
+ f
2t 2
] \ [F
h
= sqrtf
1h 2
+ f
2h 2
] If F
t
B TH2 F
t
is the feature being tested and TH2 is its threshold value – identify as a St. Peter’s fish
Else – identify as carp
END
2
.
2
. Fish handling and training A preliminary study of the ability of fish to be trained to move, upon command,
through an elongated narrow and transparent channel was conducted. The channel simulated a part of our future device for sorting live fish.
Ten red Oreochromis niloticus mass range of 200 – 300 g were used. The fish were maintained in a large rectangular concrete pool 6 × 1.5 × 2 m and fed ad
libitum on a diet of 35 protein feed pellets. Two manually operated feeding devices were installed at the two opposite sides of the pool Fig. 4a, each
comprising a white submerged feeding platform 30 cm diameter at a depth of 25 cm. A blue lamp 50 W inside a submerged transparent plastic cylinder Fig. 4b
was positioned 10 cm above the outer edge of each feeding platform. The two lamps were connected to an electronic blinker whose frequency was approximately
2 Hz.
Feed pellets were placed manually at the centre of the feeding platforms, with the aid of a 2 m long PVC pipe 2 cm diameter. The entire pool was surrounded by
a black screen to decrease visual disturbances and to avoid conditioning the fish to the presence of humans.
Fig. 3. A St. Peter’s fish showing the tail segment from which tail shape features were calculated. One feature was the MI of this segment. The other feature was a comparison of the MI of the tail with the
analytical MI of the rectangle aspect ratio = ab within which the tail was included.
Fig. 4. Schematic of the fish pond in which fish training experiments were conducted. a Concrete pool; b feeding station with blinking light; c partitions; d Plexiglas channel.
The training of the fish was carried out in three phases. During the first phase fish were trained to move to the feeding platforms in response to a visual signal. Each
training session lasted 1 min. Food was placed on one platform at a time, and a blinking blue light was operated above that platform for 20 s prior to food placing,
for 30 s during feeding and 10 s after feeding.
Four training sessions were carried out every day. The total amount of food provided was 1 of the fish biomass, and no food was given outside the training
sessions. During the second phase, fish were trained to approach the feeding platforms by
passing through a small opening in a mesh partition. The pool was divided into two equal sections by two identical affixed wire partitions each of which had a small
20 × 20 cm opening. The openings of the two partitions coincided so that they formed a single passage which fish could use to move from one section of the pool
to the other. During the third phase training continued with the two partitions Fig. 4c moved away from each other and connected by a 100 cm-long transparent
Plexiglas channel, through their 20 × 20 cm rectangular openings Fig. 4d.
3. Results and discussion
