(42) Double precision summation and DOT

[Albert Chan]

(04-15-2020 06:50 AM)Werner Wrote:  If you can beat 0.44 ENTER 1/X 1/X -, let me know ;-)

Nice find
For decimal version, with P decimal precision, above returned (-10)^(-P)

Prove:

Let x = 0.44  (exactly)       // this implied P ≥ 2

y = round(1/x, P-decimals) = c/x, where \(c = 1 + 1.2/(-10)^P\)

\(x-1/y = x(1-1/c) = x(c-1)/c = \large {0.528 \over 1.2 + (-10)^P}\)

\(\text{ulp_error} = (x-1/y)×10^P = \large {0.528 \over 1.2/10^P + (-1)^P}\)

For P = 2, 3, 4, 5, 6, ..., ulp_error ≈ +0.5217, -0.5286, +0.5279, -0.5280, +0.5280 ...

limit(|ulp_error|, P=∞) = |0.528/(0±1)| = 0.528 > 0.5

→ IROUND(ulp_error) = (-1)^P

→ round(x-1/y, P-decimals) = (-1)^P ÷ 10^P = (-10)^(-P)

P.S. for round-to-nearest, halfway-to-even, above work even with P=1
x - 1/(1/x) → 0.4 - 0.5 = -0.1 = (-10)^(-1)