HP-71B internal summation weakness in Math ROM
|
06-05-2020, 05:43 PM
Post: #16
|
|||
|
|||
RE: HP-71B internal summation weakness in Math ROM
I need to correct my previous statement. In fact, there is no rounding mechanism that can take a result rounded to 15 bits then correctly round that to 12 bits. The expected error may be better, but there are always cases where fewer than 12 bits will be correct.
This has been known since the earliest computations of trig tables. |
|||
« Next Oldest | Next Newest »
|
User(s) browsing this thread: 1 Guest(s)