MOGIE Limits of Precision

OGC 13-080r3 Copyright © 2013 Open Geospatial Consortium Page 14 experience.

6.6.6 MOGIE Limits of Precision

The Institute of Electrical and Electronics Engineers IEEE standard for Floating-Point Arithmetic IEEE 754 is a technical standard for floating-point computation that is widely implemented and supported in computer systems and software today. The current version, IEEE 754-2008 is published in August 2008 and defines: ฀ Arithmetic formats: sets of binary and decimal floating-point data, which consist of finite numbers including signed zeros and subnormal numbers, infinities, and special not a number values ฀ Interchange formats: encodings bit strings that may be used to exchange floating-point data in an efficient and compact form ฀ Rounding rules: properties to be satisfied when rounding numbers during arithmetic and conversions ฀ Operations: arithmetic and other operations on arithmetic formats ฀ Exception handling: indications of exceptional conditions such as division by zero, overflow, etc. The standard also includes extensive recommendations for advanced exception handling, additional operations such as trigonometric functions, expression evaluation, and for achieving reproducible results. 6 In the IEEE 754 standard a single precision floating point number is stored in 32 bits and a double precision floating point number is stored in 64 bits. These bit-level storage requirements limit the preservation of floating point number precision. For example, when a string representing the MOGIE data value: 179.9999999999999 e.g., 13 decimal precision is parsed into a double and then printed back as a string it will yield an output of 179.9999999999999. However, when a string representing the value 179.99999999999999 e.g., 14 decimal precision is parsed into a double and then printed back as a string it will yield an output of 180.0. This rounding error is introduced as a result of the limitation of 64 bit storage and the numerical range of position data used in MOGIE. A limitation feature identified in MOGIE was that the binary structure defined by IEEE 754 for storing floating point numbers does not preserve trailing zeros. For example, IEEE 754 binary cannot distinguish between 179.990000 e.g., P6 and 179.9900000000000 e.g., P13. IEEE 754 interprets both the P6 and P13 floating point numbers as: 179.99. The net result of this feature is that in MOGIE P6 and P10 data all trailing “0”s were omitted by software e.g., Java, PostGIS, etc. supporting IEEE 754. Refer to Appendix E for examples. In response to these IEEE 754 limitations, the decision was made to prescribe precision level for MOGIE as P6, P10 and P13. In addition, trailing “0” were not replaced in the data because it did not affect the outcome of the experiment in any way. Finally, to give readers an idea of what 13 decimal precision latitude and longitude position means in practical terms, P13 latitude and longitude precision identifies a specific location to 111,11110 13 at the equator, or approximately 1 angstrom e.g., the size of a chlorine atom. Although this level of precision is impossible to discern on a computer monitor, it has the potential to make a difference in some DoD targeting and tracking systems. 6 https:en.wikipedia.orgwikiIEEE_floating_point OGC 13-080r3 Copyright © 2013 Open Geospatial Consortium Page 15

6.6.7 MIL-STD-2525D