Post Reply 
HP41: accuracy of 13-digit routines
09-25-2015, 10:46 PM
Post: #18
RE: HP41: accuracy of 13-digit routines
(09-02-2015 04:45 AM)Thomas Klemm Wrote:  As far as I remeber rounding is just adding 500 to the 13-digit mantissa.
Thus in the example from above:
Code:
10.49954995499
+          500
10.49954995999
From this only the first 10 digits are used.

Looking at the V41 MCode console output, it seems that rounding is done by doubling the last three (guard) digits:

Code:
   10.499 54995 499
=> 10.499 54995 998
=> 10.499 54995

Dieter
Find all posts by this user
Quote this message in a reply
Post Reply 


Messages In This Thread
RE: HP41: accuracy of 13-digit routines - Dieter - 09-25-2015 10:46 PM



User(s) browsing this thread: 1 Guest(s)