(12C) Signum Function

[Albert Chan]

(03-03-2019 11:27 PM)Thomas Klemm Wrote:  If we just look at the mantissa it appears that this only happens for values between \(\sqrt{10} \approx 3.16228\) and \(5\).

This range can be explained by squaring errors: (x·(1+ε))² = x²·(1+2ε+ε²) ≈ x²·(1+2ε)
Assuming numbers in scientific notation, with mantissa = [1, 10), ulp = 1e-9

Max rel error of √(x²) ≈ ½ Max rel error of x² = ½ (½ ulp / mantissa(x²))

Max abs error of √(x²) ≈ ¼ ulp (mantissa(x) / mantissa(x²))

Round-trip errors occurs if √(x²) abs error ≥ ½ ulp
-> mantissa(x)    between √10 and 5.0
-> Max abs error between 0.79 and 0.50 ulp

Edit: this also explained why (√x)² may have errors upto 3 ULP
Example: (√9.11)² = 9.110000003