HP 42S Version Speed Variation

[Paul Dale]

(06-13-2014 04:23 AM)Mike Morrow Wrote:  The amount that the end result varies from a perfect 2500.0000000 is a benchmark for accuracy.

Insert another discussion about correct rounding and how 2500 isn't the expected result on any finite precision floating point device

In single precision mode, the 34S gets 2500.000000006559. This is most likely the true correct result for a sixteen digit decimal device.


- Pauli