Post Reply 
Accuracy and the power function
02-19-2014, 01:07 PM
Post: #11
RE: Accuracy and the power function
Truncation (and keeping track of lost non-zero digits with the Sticky Bit) in the 15-digit routines is not only common practice, it's a necessity if you want to be able to produce a correctly rounded 12-digit result for simple calculations.
Suppose the 15-digit routines rounded as well, then:
5.02000000121e15 + 4995 (exact result: 5.020000001214995e15) would be rounded to 15 digits as 5.02000000121500e15
5.02000000121500e15 rounded to even to 12 digits is 5.02000000122e15
yet the 16-digit number rounded to 12 digits is 5.02000000121e15

Find all posts by this user
Quote this message in a reply
Post Reply 

Messages In This Thread
Accuracy and the power function - Dieter - 02-16-2014, 02:43 PM
RE: Accuracy and the power function - Werner - 02-19-2014 01:07 PM

User(s) browsing this thread: 1 Guest(s)