|Re: Rounding conventions|
Message #37 Posted by Rodger Rosenbaum on 4 July 2009, 6:48 p.m.,
in response to message #1 by Wayne Best
While this may be ok for accounting and finance, it introduces a bias when rounding data sets, and would appear to be unsuitable for many scientific applications especially those involving sets of data.
The HP71 and later Saturn based calcs round-to-even (unless you set the rounding mode in the HP71 to something else) during arithmetic calculations, so this problem only shows up in the display of final results. One way around it without doing any programming is to set your calculator to display all 12 digits in the result and do the rounding to a smaller number of significant digits yourself.
Or, if you have a large number of final results to print at some lower number of significant digits, you can cause that number of digits to be rounded-to-even using a technique I described on comp.sys.hp48 in 2004:
Imagine that you want to round a number on your calculator to a reduced number of digits, smaller than the calculator's full precision, with proper rounding (round-to-even, that is!). If the calculator does N digit arithmetic and you want a K digit rounded result, R, the formula:
R = value - value * 10^(N-K) + value * 10^(N-K)
R = value + value * 10^(N-K) - value * 10^(N-K)
gives the K digit properly rounded result where "value" is the original full-precision number you want to round to K digits. These formulas use Dekker's method, and only work correctly if the calculator does proper rounding during arithmetic.
For example, on the HP48, if we have the number 123450000001 in the display and want to round this to 4 digits, we compute:
123450000001 - 123450000001 * 1E8 + 123450000001 * 1E8
and we get: 123500000000
But if my starting value is 123450000000 and I compute:
123450000000 - 123450000000 * 1E8 + 123450000000 * 1E8
we get: 123400000000
The result is rounded to even. Now start with 123550000000 and compute:
123550000000 - 123550000000 * 1E8 + 123550000000 * 1E8
we get 12360000000, and we see that the result is rounded to even, as it should be because the discarded digits were 50000000..., that is, a 5 followed by only zeroes.
Try reversing the order of the addition and subtraction in the last example:
123550000000 + 123550000000 * 1E8 - 123550000000 * 1E8
we still get: 123600000000
Now try this with a TI86 calculator:
123550000000 - 123550000000 * 1E10 + 123550000000 * 1E10
(use 1E10 as the multiplier because the TI is a "14" digit machine)
we get 123500000000 (only 12 digits are shown)
But if we try:
123550000000 + 123550000000 * 1E10 - 123550000000 * 1E10
we get 123600000000 on the TI86. So Dekker's method fails on the TI.