A quick precision test

[Paul Dale]

(06-04-2014 01:25 PM)Claudio L. Wrote:  newRPL does not use decNumber but uses mpdecimal, also correctly rounded and we also use a few digits more but all intermediate calculations are rounded to the user precision so the results from newRPL and WP34S should match exactly, as both should produce 34 correctly rounded digits on each intermediate calculation.

By using extra guard digits you get a double rounding step which can result in incorrectly rounded results. A quick example...

If the true result of a computation is 123.456789 but our guard digit format only carries 4 digits, the guarded result will correctly be 123.5. If now, our returned result only has 3 digits, we'll get 124 which is incorrect. It should be 123.

Sure this is contrived but the same can happen regardless of the number of guard digits.

Double rounding is bad. Guard digits are good. Generally guard digits are more good than double rounding is bad, which is why they are used. However, getting a correctly rounded result is, in general, extremely difficult.


- Pauli