A quick precision test

[Claudio L.]

(06-03-2014 12:36 AM)Paul Dale Wrote:  We always round to 16 or 34 digits when we return the result to the stack. These are the only precisions a user or a keystroke program can access. Unless you are coding numerics in C, this is all you get.

The 39 digits are used inside but only during the evaluation of a built in function. Once the result is known, it is rounded. The reason is simple: guard digits. I don't need to and generally cannot get all 39 digits correct but having the extra digits gives some headroom to get the rounded result correct or at least close.


- Pauli

Paul,

EDIT: Nevermind, I just read the post from G. Barbosa, showing that WP34S matches exactly the results of newRPL. I'm not deleting the information below just for the record, but my post is pretty much useless from here on... except to actually validate that WP34S and newRPL are on the same page.

I think he's onto something. Rounding should not kill so many digits.

I just took the newRPL demo (shameless advertising: download the demo from http://sf.net/p/newrpl ) and did:

Code:


34 SETPREC
1.0000001
<< 1 27 START DUP * NEXT >> EVAL

In human words: Set the precision to 34 digits to match the intermediate rounding you are using in WP34S. Then do the 27 squares by hand and the result is:

674530.4707410845593826891847277722 (if I didn't make a mistake copying by hand the numbers)

The "real" result at 34 digits is:
674530.4707410845593826891780297468

This is obtained from doing:

Code:


1.0000001 2 27 ^ ^

The difference is 6.698E-21, so there's 25 good digits out of the 34. Intermediate rounding only affected the last 9 digits.

So the original poster might have a good point. newRPL does not use decNumber but uses mpdecimal, also correctly rounded and we also use a few digits more but all intermediate calculations are rounded to the user precision so the results from newRPL and WP34S should match exactly, as both should produce 34 correctly rounded digits on each intermediate calculation.

I think his findings might be worth a second look.

Claudio