Post Reply 
HP41: accuracy of 13-digit routines
09-02-2015, 10:18 AM
Post: #4
RE: HP41: accuracy of 13-digit routines
(09-02-2015 04:45 AM)Thomas Klemm Wrote:  Thus the internal 13-digit result isn't rounded correctly but cut off.
From this only the first 10 digits are used.

(09-02-2015 06:11 AM)Joe Horn Wrote:  Thomas is correct.
This "truncate, then round" procedure was not only used by the old 10-digit models, but is still used by the newer 12-digit HP calculators, which calculate internally to 15 *truncated* digits, then round that result to 12 digits.

Thank you very much. So the 13-digit routines truncate after 13 digits. I assume this also applies to the transcendental functions (log, exp, trig etc.). Now if I set a WP34s to RM 4 (truncate) and apply a RSD 13 to every result, this should yield the same result as the 41's 13-digit routines?

The reason I'm asking this is testing a Lanczos Gamma approximation with an optimized coefficient set, cf. the recent thread in the HP41 software library.

Find all posts by this user
Quote this message in a reply
Post Reply 

Messages In This Thread
RE: HP41: accuracy of 13-digit routines - Dieter - 09-02-2015 10:18 AM

User(s) browsing this thread: 1 Guest(s)