Post Reply 
HP 42S Version Speed Variation
06-13-2014, 05:20 AM
Post: #49
RE: HP 42S Version Speed Variation
(06-13-2014 04:55 AM)Paul Dale Wrote:  
(06-13-2014 04:23 AM)Mike Morrow Wrote:  The amount that the end result varies from a perfect 2500.0000000 is a benchmark for accuracy.

Insert another discussion about correct rounding and how 2500 isn't the expected result on any finite precision floating point device Smile

In single precision mode, the 34S gets 2500.000000006559. This is most likely the true correct result for a sixteen digit decimal device.

heh heh! My brain hurts!

In a "perfect" world with perfect infinite precision calculators, little details like floating point arithmetic would vanish or be banished. For now...let's just use truncation and rounding to get that perfect (looking) result! :-)
Find all posts by this user
Quote this message in a reply
Post Reply 


Messages In This Thread
HP 42S Version Speed Variation - Gerald H - 06-09-2014, 09:25 AM
RE: HP 42S Version Speed Variation - Mike Morrow - 06-13-2014 05:20 AM



User(s) browsing this thread: 1 Guest(s)