Quality control and performance measures

1.3 Quality control and performance measures

How do we evaluate the quality of image orientation from a structure from motion algorithm? Of course, each implementation of SFM has its own quality optimization criteria as well as different parameters that control the various stages from feature extraction, to feature matching, to image orientation. Changing the values of control parameters will change the output of the process and to some extent its quality as well; how is the output quality to be evaluated? In principle we can think of two sorts of rankings: from a user standpoint an absolute measure e.g. a sort of score of the orientation goodness would perhaps be the best, since no other term of comparison another sequence oriented with a different method is available; in a scientific and technical context, when comparing the results of different SFM strategies applied to the same image sequence, relative scores e.g. collinearity residuals, point redundancy, frame coverage, etc. might be acceptable. In both cases, practically every quality measures depend to large extent on the data set network geometry, image quality, object shape, calibration data, etc.. This means that both “absolute” and “relative” measures in fact apply to that specific dataset. The goal of this paper is to provide some tentative proposal on this topic, that we believe should get more attention within the photogrammetric community. In section 2 a brief description of the main blocks of our SFM strategy are presented; some details about the current software implementation are also given. In section 3 some proposals are put forward on how to evaluate the quality of automatic image orientation. In section 4, using datasets provided by the workshop organizers, the results of the automatic orientation and the checks performed on three image sequences of cultural heritage sites where control data are available are presented; besides, results and some statistics are presented on the orientation of other four image sequences also retrieved from the organizer’s website.

2. OUR IMPLEMENTATION OF SFM