Accuracy and the power function

02192014, 12:35 PM
Post: #10




RE: Accuracy and the power function
(02192014 09:54 AM)Werner Wrote: There are two things at work here.Ah, thank you very much, that's exactly what I have been looking for. I tried to simulate the error on a 34s in 16digit mode with a similar calculation. So the key here is the logarithm that's off by one ULP. Truncation of the last (guard) digit(s) seems to be a common practice. IIRC the TI58/59 did so as well, and today you still can see truncation at work, for instance on the 35s. Try the tangent function with very small angles: Code: [MODE] DEG Dieter 

« Next Oldest  Next Newest »

User(s) browsing this thread: 1 Guest(s)