Post Reply 
Accuracy and the power function
02-19-2014, 12:35 PM
Post: #10
RE: Accuracy and the power function
(02-19-2014 09:54 AM)Werner Wrote:  There are two things at work here.
First, the 15-digit routines indeed are not exact to the last digit. In your example, %%LN(6.28318530718) is off by one ULP.
Secondly, the 15-digit routines (%%*, %%+, %%-, %%/) do not round, they truncate.
Ah, thank you very much, that's exactly what I have been looking for. I tried to simulate the error on a 34s in 16-digit mode with a similar calculation. So the key here is the logarithm that's off by one ULP.

Truncation of the last (guard) digit(s) seems to be a common practice. IIRC the TI58/59 did so as well, and today you still can see truncation at work, for instance on the 35s. Try the tangent function with very small angles:
0,00001   [tan]  →  1,74532920000 E-7
0,0000001 [tan]  →  1,74532000000 E-9

0,0005  [tan]  →  5,00000041660 E-4
0,0001  [tan]  →  1,00000000330 E-4
0,00001 [tan]  →  1,00000000000 E-5
In degrees mode, arguments between 1E-3 and 1E-9 have approx. 11...6 valid digits, where the last is truncated or even 1 unit low.

Find all posts by this user
Quote this message in a reply
Post Reply 

Messages In This Thread
Accuracy and the power function - Dieter - 02-16-2014, 02:43 PM
RE: Accuracy and the power function - Dieter - 02-19-2014 12:35 PM

User(s) browsing this thread: 1 Guest(s)