PROBLEM WITH PI π

[Albert Chan]

(12-29-2023 03:19 PM)bxparks Wrote:  My reasoning was that 1/3 = 0.0101_0101 (base-2, to 8 bits in this example). Multiply that by 11 (base-2) to get 0.1111_1111 (base-2). Subtract 1 to get -0.0000_0001 (base-2), which cannot equal 0.

No, intermediate calculation crossed 2^n boundary, size of ULP changed.

0.01010101      // 1/3, 7 bits precision
0.11111111      // 1/3*3
1.000000        // 7 bits precision (half-way rounded-to-even)

0.0101011       // 1/3, 6 bits precision
1.0000001       // 1/3*3
1.00000         // 6 bits precision (rounded-down)

This matched previous analysis

(12-29-2023 02:39 PM)Albert Chan Wrote:  Base 2, 1/3*3 - 1 = 0

float(1/3) = 1/3 ± ULP(1/3)/3      // sign depends on parity of bits
float(1/3)*3 = 1 ± ULP(1/3) = 1 ± ULP(1)/4 --> rounded back to 1