[HP-12C] Angular coefficent from linear regression off by 1

[Dieter]

(05-01-2016 07:12 PM)bshoring Wrote:  My HP-38C, predecessor to the HP-12C (12C is modeled after the 38C) also gives 11.09933165. However my iPhone emulator of the 38C, which carries 15 or 16 digit accuracy yields 11.09933164.

Hmm... in this case all variables of the equation for calculating the regression slope (i.e. the various sums accumulated by Σ+) have at most five significant digits. So the result can be calculated exactly even with merely 10 digits, while both the 38C and the 12C should use 13 digits internally.

Using the formulas in the 15C manual, the exact result for the slope of the given data is 48,16 : 4,339 which any HP should be able to calculate exactly. ;-) So something else must be going on here.

Dieter