Optimal Subpixel Matching of Contour Chains and Segments
Optimal Subpixel Matching
of Contour Chains and Segments
Bruno Serra y Marc BerthodINRIA Sophia-Antipolis, Projet Pastis, BP 93, 06902 Sophia-Antipolis Cedex, FRANCE y
AEROSPATIALE, E/ETRV, 1 Rue Pablo Picasso, 78114 Magny-les-Hameaux, FRANCE
Abstract This paper introduces a new general purpose algo- rithm that allows the optimal geometricmatch between contours to be determined, that is the transformation yielding a minimal deformation is obtained. The al- gorithm relies only on the geometric properties of the contours and does not call for any other constraint, so that it is particularly suitable when no parameteriza- tion of the deformation is available or desirable. Con- tour deformation is explicitly incorporated in the com- putation, allowing for a thorough use of all geometric information available. Moreover, no discretization is involved in the computation, resulting in two main ad- vantages: rst, the algorithm is robust to di erences in the segmentation of contours and allows the matching of polygonal approximations of contours with very little loss of precision, second, subpixel precision matching can be achieved.To establish point-to-point correspondences be- tween contours, most methods rely on (1) strong geo- metric constraints such as epipolar lines or parameter- ized deformations (2) heuristic criteria such as inten- sity similarity or local gradient orientation. While the former constraints are not always available (e.g. the motion between the two images might be unknown), the latter prove inapplicable in many cases: changes in luminosity cause the local intensities to vary, per- spective deformations a ect the gradient orientation.
Most existing methods for matching images with weak constraints iteratively distort one image to t the second [2] [3] [7]. The main drawbacks of these ap- proaches are the need of a global preregistration pro- cess and their iterative implementation which results in large computational requirements.
The approach we propose is aimed at exploiting to its best the geometric information provided by the contours themselves, using the only reliable informa- tion between contours, i.e. shape similarity.
The rst step is to de ne a
deformation measure
to compare two contours. This measure should be ro- bust to di erences in the segmentation of contours, especially when dealing with polygonal contours: two contours might be very similar even though not seg- mented in the same way (di erent number of segments with their endpoints at di erent positions). By de n- ing the correspondence as a continuous function and by using a variational formulation for the deformation measure, we are able to tackle this problem. At the same time, subpixel matching is achieved, since the continuous nature of the correspondence function im- plies that matches are available for any point on the contours, being an endpoint or not.
The second step involves the minimization of the deformation measure over the possible correspon- dences. This minimization is achieved in two stages. By considering only two linear portions of contours, we rst derive an analytical expression for the cor- respondence with minimal deformation. This expres- sion is then used as a local cost function in a dynamic programming algorithm which ensures global optimal- ity over the whole contour. The cost function being given as an analytical expression and the correspon- dence function being continuous, a novel scheme called
1 Introduction
Continuous Dynamic Programming
was devised. This new technique allows an optimal continuous path in a graph to be determined in an e cient fashion.
2 A geometric deformation measure
Given two contours de ned by their curvilinear co- ordinates C 1 (r ) and C 2 (s) (Fig. 1), let = ( r s ) rep- resent a match between point C 1 (r ) and point C 2 (s) , and D ( ) = C 2 (s) ? C 1 (r ) be the displacement vec- tor between the two points. We can associate to
D C (r) 1 2 C (s) C 1 C 2 s s 1 r r 1 Figure 1: Displacement vector.
This equation can of course not be solved in the gen- eral case, so that it is usually solved by an iterative nite elements or nite di erences method. However, it may be solved given an analytic expression of D : considering two linear contour portions: C 1
= min N X i=0 D i where is piecewise linear and D i is the optimal de- formation between two contour segments (eq. 4).
Given the analytical expression for the optimal matching of linear contour portions, we will now take advantage of the fact that contours can be viewed as chains of linear portions (either contour segments or "micro-segments" between contour points) to derive a global optimum for the matching of whole contours. Considering two contours, the optimal deformation can be expressed as: D
3 Optimal matching of contours using continuous dynamic programming
[( r 1 ? r ) 2 + ( s 1 ? s ) 2 ] 1=2 (4)
= ka 2 ( s 1 ? s ) ? a 1 ( r 1 ? r ) k 2
= r 1 ? r t 1 ? t ; = s 1 ? s t 1 ? t and the constraint on (Eq. 1) becomes: ( t 1 ? t ) 2 = ( r 1 ? r ) 2 + ( s 1 ? s ) 2 Substituting the previous equations in the de nition of the deformation measure (Eq. 2) nally yields the analytical expression of the optimal deformation: D
( t ) = t + ; s ( t ) = t + and can be determined by using the boundary conditions ( t ) and ( t 1 ) :
( r ) = a 1 r + b 1 ; C 2 ( s ) = a 2 s + b 2 ; ka 1 k = ka 2 k = 1 eq. 3 reduces to: ( t ) = 0 so that the optimal match is a linear function. We can now determine the associated deformation mea- sure (Eq. 2). Since is linear, we can write: r
Expanding this equation yields: [ D ( ( t )) ( t ) + D ( ( t ))_ ( t ) 2 ] T D ( ( t )) = 0 (3)
any match between the two contours a correspondence function ( t ) = ( r ( t ) s
Euler-Lagrange equation: F ? d dt F _ = 0
( ) = Z t 1 t F ( ; t ) dt where : F ( ; t ) = k dD ( ( t )) dt k 2 A classical approach involves the resolution of the
Finding the optimal deformation measure can now be stated as a variational problem where the following functional has to be minimized: D
(2) This expression measures the variation of the displace- ment vector along the contours, and thus the deforma- tion from one contour to the other. It is similar to the one used by Hildreth [5] to compute the velocity eld between images, and can be related to the gural con- tinuity and disparity gradient constraints introduced by Mayhew and Frisby [8].
= Z t 1 t k dD ( ( t )) dt k 2 dt
_ ( t ) k = 1 (1) Given two contours and a correspondencefunction, we can now de ne a deformation measure as: D
( t ) is unique, since for any function f the function such that ( t ) = ( f ( t )) will also be consistent with the de nition. The following constraint is thus added to ensure uniqueness, by imposing that the correspon- dence function varies in a uniform fashion over the contours (the matches should not accumulate in some portions of the contours or be sparse in over portions): k
( t ) ) de ned over an interval [ t ; t 1 ] such that ( t ) = ( r s ), ( t 1 ) = ( r 1 s 1 ), and for any t , C 1 ( r ( t )) is matched with C 2 ( s ( t )). This de ni- tion is however not enough to ensure that the function
This expression can be viewed as the search for an optimal path in a graph constructed by placing on each axis the arc length on each contour (Fig. 2). Any square portion of the graph corresponds to two seg- ments of contour, so that the optimal path is linear in each square of the graph, and we can associate with each path portion the cost function D i given by eq. 4.
- The optimal path ending in s
- ds 2 ?<
- with dr
) = min
i
( s
;i
This minimization can be simpli ed as follows: D
i
= + r
= s
i ? s i ?1 ;
i ? r i ?1 ; ds
= r
= 2 ? dr
2 a 1 :a 2 dr ds [ dr 2 + ds 2 ] 1
dr dr 2
) = min
i
z f
( z ) ds + with : z = dr ds and : f ( z ) = z 2 ?
2 a 1 :a 2 z + 1 p 1 + z 2 ? z Which shows that the optimal path ending in s
i
min
) = f ( z
i
( s
;i
is simply D
minimizes function f , the cost of the optimal path end- ing in s
i
min which
1. It can be calculated numer- ically in the other cases. Given the value z
2, and for z = 1 if
a 1 a 2 ?
2. Under that condition, the second derivative of f can easily be proved positive, which implies that there is a single minimum. Further- more, the minimum is obtained for z = 0 if
(note that z is simply the inverse of the slope of the path in the graph). Unfortunately, f does not have a single minimum in the general case. However, this di culty can be overcome if we impose the following simple constraint: a 1 :a 2 and z should have the same sign. This condition is not very restrictive, since it is equivalent to imposing that the angle between the two segments is less than =
is linear. We still need to nd the minimum of function f ( z )
( s
;i
(Eq. 4) yields: D
Considering a single square of the graph (Fig. 3), we will suppose that the cost of the optimal paths ending on the lower side of the square are linear, and we will show that under that condition the cost of the optimal paths ending on the right side of the square are also linear, so that by recursion the same result holds for any square of the graph (the proof being the same for a path going through the other sides of the square). contour 2
is linear: D
i ?1
of the optimal path end- ing in r
;i ?1
Suppose that the cost D
(contour 2) contour 1 i−1 r i s i−1 r i s s i−1 r i s (contour 1) r Figure 3: A linear path portion.
Continuous dynamic programming proceeds in a similar way, except that instead of associating a nu- merical cost value to each node, the cost of the opti- mum path ending in any point between two nodes is given by a parametric expression. The previous point on the path is similarly remembered as a paramet- ric expression. An initial description of this scheme has been described in [9], however with the initial hy- pothesis that the local cost function is quadratic. In the present case, the local cost function (Eq. 4) is not quadratic, but we will recursively exhibit a linear ex- pression for the total cost function.
( r
After the last column in the graph has been explored, nd the node in that column with minimal cost, then backtrack in the graph to get the complete path.
The graph is explored from left to right in order to nd the best path ending in each node. For each node in the graph, a simple reasoning yields the best path: for each possible node in the previous column, add the cost of the path ending in that node and the local cost from that node to the current node, and retain the one yielding a minimum cost.
Discrete dynamic programming proceeds as follows: A null cost is assigned to all points in the rst column of the graph, allowing the rst contour point to match any point on the second contour.
gramming , which allows an optimal continuous path to be determined.
Dynamic programming [4] [1] is a method of choice to nd an optimal path in a graph: it is guaranteed to nd the global optimum; it is a non-iterative method; it is computationally e cient. However, classical dynamic programming algorithms are restricted to nding a graph going through the nodes of a graph, whereas in our case the path can cross the lines and columnsof the graph anywhere. For this purpose, we have conceived a novel dynamic pro- gramming scheme, called Continuous Dynamic Pro-
The optimal match is the path with minimal total cost in the graph. It can be entirely de ned by its inter- sections with the lines and columns of the graph, so that only these points need to be determined.
contour 1 contour 2 point 1 2 3 2 3 point 1 Figure 2: Matching viewed as a path in a graph.
;i ?1
i ?1
i ?1
( D
) Substituting the value of D
i ?1
) + D
i ?1
( r
;i ?1
r i?1
) = r
) = min
i
( s
;i
D will be given by:
i
i ?1
) ds + , and the previous point on the path is given by dr = z ds .
We deal with this non-linearity by approximating the cost function by successive linear portions, with a xed maximal approximation error.
} 1 } } C C 3 C 2 Figure 4: Cost portions.
In order to analyze experimentally the precision and robustness to segmentation of the algorithm, a number of experiments have been carried out on two synthetic contour images of a sphere with subpixel edge localization.
4.1 Qualitative and quantitative analysis
In order to appreciate qualitatively the matching precision, a 3D reconstruction was done using the out- put of the algorithm. The result is presented in g- ure 5, and shows the precision as well as the smooth- ness of the matching process. For comparison, the result obtained with a discrete algorithm is presented.
A quantitative evaluation was then performed: since the exact position of the matches is known on these synthetic images, the mean error between each match and the actual position was calculated, yield- ing a value of 0.10 pixel. Under the same conditions, the discrete algorithm used for comparison produced a mean error of 0.9 pixel.
4.2 Robustness to segmentation
The robustness of the algorithm to segmentation was evaluated by using a number of polygonal ap- proximations of the contours with di erent tolerances.
Figure 5: Subpixel 3D reconstruction (mean error: 0.10 pixel). vs. discrete algorithm (error: 0.88 pixel). For each approximation, the mean segmentation er- ror was computed, as well as the mean matching er- ror after running the algorithm. Figure 6 shows the graph of the matching error versus the segmentation error. This graph shows the smooth behaviour of the algorithm with respect to segmentation: the increase of the matching error is nearly linear with the seg- mentation error, which shows that the matching pro- cess is little a ected by the segmentation of contours and that the algorithm is insensitive to the uneven distribution of points along contours. The polygonal approximation process yields segments whose lengths vary greatly and which are not necessarily similarly distributed in both images, yet the correct correspon- dences are recovered.
4 Experimental analysis
4.3 Computational complexity
Fig. 7 shows a plot of the computation time for di erent values of the segmentation tolerance against the number of contour segments in one image. The computational complexity is nearly linear. One must keep in mind that in this experiment, since the size of the contours was xed while the number of seg- ments varied, the number of candidates per segment Figure 6: Analysis of the robustness to segmentation: matching error vs. approximation error. Bottom: ex- ample of reconstruction of approximated contours. Figure 9: Reconstruction after polygonal approxima- Figure 7: Complexity analysis : computation time vs. tion of contours (left). Reconstruction with a discrete number of contour segments to be matched. algorithm (right). increased with the number of segments (the neigh- to the one obtained with the subpixel algorithm, even bourhood size for candidate search being constant), when the matching is performed on a polygonal ap- so that a quadratic complexity could have been ex- proximation of contours. pected.
5.2 Complex scene
5 Real images
This next example demonstrates the high resolu- tion 3D reconstruction that can be achieved using the
5.1 Simple scene subpixel matching algorithm on complex real images.
Figure 10 shows a view of an indoor scene, the re- The algorithm has been applied to a simple stereo construction obtained with the subpixel matching al- pair of a Rubik's cube. Figure 8 shows a full reso- gorithm, and the reconstruction obtained with a dis- lution 3D reconstruction, whereas Figure 9 shows a crete algorithm [6]. As can be observed, the jagged reconstruction obtained by matching a polygonal ap- contours produced by the discrete algorithm are much proximation of the contours. As can be observed, the smoother with the subpixel algorithm. quality of the results are comparable, while the com- putation time was reduced from 207 seconds in the rst case to 16 seconds in the second case, a gain
6 Conclusion
of more than an order of magnitude. Figure 9 also shows the result obtained with a classical discrete al- In this paper, we have described a novel algorithm gorithm [6]: the 3D reconstruction is strongly a ected for the optimal matching of contours. Based on a by discretization noise and the resolution is inferior contour deformation measure, this scheme exploits to a full extent the geometric information contained in the contour. No additional constraints are needed, al- though any available constraint can be used to reduce the search space and hence the computational com- plexity. This characteristic yields a general-purpose algorithm, suitable for any task where elastic match- ing is needed. Furthermore, due to its continuous na- ture, the algorithm has the ability to produce subpixel results, and can run unchanged on contour segments or contour chains. By feeding the algorithm with a polygonal approximation of contours, the computa- tional complexity can be reduced by an order of mag- nitude while preserving the quality of the results.
An extensive experimental analysis has been car- Figure 8: 3D reconstruction of a Rubik's cube ried out, demonstrating the precision, the robustness Figure 10: O ce scene: left image (top), subpixel 3D reconstruction looking from above (middle), discrete
3D reconstruction (bottom) to segmentation, and the linear computational com- plexity of the algorithm. An application to the sub- pixel 3D reconstruction of stereo images has been pre- sented. Excellent results on complex real images are obtained, and comparisons with the results obtained on the same images with a discrete algorithm show the gains in resolution that can be achieved.
The MIT Press, 1983. [6] R. Ma and M. Thonnat. A robust and e cient contour-based stereo matching algorithm. Re- search Report 1860, INRIA, 1993. [7] Mehran Moshfeghi. Elastic matching of mul- timodality medical images.
Conference on Computer Vision and Pattern Recognition
In
, 14(4):449 470, 1985. [9] Bruno Serra and Marc Berthod. Subpixel contour matching using continuous dynamic programming.
Perception
PMF - a stereo correspondence algorithm using a disparity gradient limit.
, 53(3):271 282, may 1991. [8] S. B. Pollard, J. E. W. Mayhew, and J. P. Frisby.
CVGIP: Graphical Models and Image Processing
[5] E.C. Hildreth. The Measurement of Visual Motion .
Extension of the algorithm to the matching of 3D contours is now underway. The theoretical framework presented in this paper can be applied to the 3D case in a straightforward manner, and preliminary results relative to the elastic matching of 3D medical images have been very encouraging. References [1] Amir A. Amini, Terry E. Weymouth, and
culus of Variations . Academic Press, New-York, 1965.
1981. [4] S. Dreyfus. Dynamic Programming and the Cal-
, 46:1 21, 1989. [3] D.J. Burr. A dynamic model for image registra- tion. Computer Graphics and Image , 15:102 112,
Computer Vision, Graphics and Image
[2] Ruzena Bajcsy and Stane Kovacic. Multiresolution elastic matching.
IEEE Transactions on Pattern Analysis and Machine In- telligence , 12(9):855 867, sept 1990.
Ramesh C. Jain. Using dynamic programming for solving variational problems in vision.
, pages 202 207, Seattle, 1994.