Accuracy and the power function
|
02-19-2014, 12:35 PM
Post: #10
|
|||
|
|||
RE: Accuracy and the power function
(02-19-2014 09:54 AM)Werner Wrote: There are two things at work here.Ah, thank you very much, that's exactly what I have been looking for. I tried to simulate the error on a 34s in 16-digit mode with a similar calculation. So the key here is the logarithm that's off by one ULP. Truncation of the last (guard) digit(s) seems to be a common practice. IIRC the TI58/59 did so as well, and today you still can see truncation at work, for instance on the 35s. Try the tangent function with very small angles: Code: [MODE] DEG Dieter |
|||
« Next Oldest | Next Newest »
|
User(s) browsing this thread: 1 Guest(s)