H->HMS conversion HP-15C vs. HP42S vs HP67

[Albert Chan]

(09-12-2018 02:08 PM)Albert Chan Wrote:  Correctly implemented strtod guaranteed maximum error 0.5 ULP
Scaling by constant also guaranteed maximum error of 0.5 ULP
So, scaled value can at most be 1 ULP below User Decimal Input (where 40 second bug might hit)

I made an error adding up the ULPs.
What should be added is relative errors, not ULPs

Let binary mantissa of X = m1, of scaled Y = 100*X = m2
So, 2^52 <= m1 < 2^53, and same for m2 (units in ULP)

Max relative (under-estimated) error = 0.5/m1 + 0.5/m2

Max absolute error (ULP) = Max relative error * m2
= 0.5 + 0.5 m2/m1
= 0.5 + 0.5 * max(m2/m1)
= 0.5 + 0.5 * (2^53-1) / 2^52
< 1.5 ULP

Technically +1 ULP still work, but it is cutting it very close.
(If Y < 0.5 ULP below an integer, it will round-up to integer)

100.00000000000001*X patch is safer:
Let m3 = binary mantissa of 100.0 = 0.78125 * 2^53

Max relative (under-estimated) error = 0.5/m1 + 0.5/m2 - 1/m3

Max absolute error (ULP) = Max relative error * m2
= 0.5 + 0.5 m2/m1 - m2/m3
= 0.5 + m2 * (1/(2 m1) - 1/m3)

Since (2 m1) > m3, the factor is negative, we need to minimize both terms

Max absolute (under-estimated) error
= 0.5 + 2^52 * (1/2^53 - 1/(0.78125*2^53))
= 0.36 ULP < 0.5 ULP

In other words, DD.MM * 100.00000000000001 >= DDMM (in binary float)