|Re: HP 35s Standard Deviation|
Message #9 Posted by Karl Schneider on 21 Sept 2007, 2:40 a.m.,
in response to message #8 by bill platt
John Smitherman posted,
Enter 1,000,000 , 1,000,001 and 1,000,002 and find the sample standard deviation. The correct answer should be 1 but as has been discussed before the 33s provided an error message.
Bill Platt replied,
This is beyond the range of the machine. It doesn't work on a 32sii nor on a 15c either. It is not a flaw.
Bill is right; the "STAT ERROR" or "Error 2" message is attributable to roundoff errors in the statistical summation. What's ironic is that the failure to produce a result was made possible by extended-precision calculation, albeit using standard-precision inputs.
Here are the tabulations:
Xi (Xi)^2 (exact) (Xi)^2 (result retaining 10 or 12 significant digits)
1,000,000 1,000,000,000,000 1,000,000,000,000
1,000,001 1,000,002,000,001 1,000,002,000,000
1,000,002 1,000,004,000,004 1,000,004,000,000
S(X) S(X^2) (exact) S(X^2) (using 10 or 12-digit results)
3,000,003 3,000,006,000,005 3,000,006,000,000
([S(X)]^2) (exact result obtained using 13- or 15-digit extended precision)
([S(X)]^2)/3 (exact result obtained using 13- or 15-digit extended precision)
The exact calculation
sqrt ([S(X^2) - ([S(X)]^2)/n] / [n-1]) of sample standard deviation is
sqrt [(3,000,006,000,005 - 3,000,006,000,003) / (3-1)] = 1
The "erroneous" calculation is based on 13- or 15-digit internal computaion that utilizes the S(X^2) summations that were rounded off after each datum entry:
sqrt [(3,000,006,000,000 - 3,000,006,000,003) / (3-1)] = ERROR
(The HP-15C's extended-precision calculations for statistical summations are utilized within an example in the HP-15C Advanced Functions Handbook listed on pages 208-211.)
If extended precision had not been employed for the problem discussed here, both terms would have been rounded to 3,000,006,000,000 -- yielding the incorrect result of zero, instead of an error.
Edited: 26 Sept 2007, 2:09 a.m. after one or more responses were posted