Post Reply 
HP-71B internal summation weakness in Math ROM
06-05-2020, 05:43 PM
Post: #16
RE: HP-71B internal summation weakness in Math ROM
I need to correct my previous statement. In fact, there is no rounding mechanism that can take a result rounded to 15 bits then correctly round that to 12 bits. The expected error may be better, but there are always cases where fewer than 12 bits will be correct.

This has been known since the earliest computations of trig tables.
Find all posts by this user
Quote this message in a reply
Post Reply 


Messages In This Thread
RE: HP-71B internal summation weakness in Math ROM - ttw - 06-05-2020 05:43 PM



User(s) browsing this thread: 1 Guest(s)