HP41: accuracy of 13digit routines

09022015, 04:45 AM
Post: #2




RE: HP41: accuracy of 13digit routines
(09012015 09:23 PM)Dieter Wrote: In other words: do they really provide correctly rounded 13digit results under all circumstances? Or may the results be truncated instead of rounded? Or maybe even the last digit may be slightly off? This depends a little on the definition of "correctly rounded". You may try 104985/9999 which is: 10.499549954995... Correctly rounded to 13 digits would give us: 10.49954995500 If we'd round this to 10 digits we'd get: 10.49954996 But that's not what we get. Instead we get: 10.49954995 Thus the internal 13digit result isn't rounded correctly but cut off. As far as I remeber rounding is just adding 500 to the 13digit mantissa. Thus in the example from above: Code: 10.49954995499 HTH Thomas 

« Next Oldest  Next Newest »

User(s) browsing this thread: 1 Guest(s)