Re: HP 35s Standard Deviation Message #9 Posted by Karl Schneider on 21 Sept 2007, 2:40 a.m., in response to message #8 by bill platt
John Smitherman posted,
Quote:
Enter 1,000,000 , 1,000,001 and 1,000,002 and find the sample standard deviation. The correct answer should be 1 but as has been discussed before the 33s provided an error message.
Bill Platt replied,
Quote:
This is beyond the range of the machine. It doesn't work on a 32sii nor on a 15c either. It is not a flaw.
Bill is right; the "STAT ERROR" or "Error 2" message is attributable to roundoff errors in the statistical summation. What's ironic is that the failure to produce a result was made possible by extendedprecision calculation, albeit using standardprecision inputs.
Here are the tabulations:
Xi (Xi)^2 (exact) (Xi)^2 (result retaining 10 or 12 significant digits)
1,000,000 1,000,000,000,000 1,000,000,000,000
1,000,001 1,000,002,000,001 1,000,002,000,000
1,000,002 1,000,004,000,004 1,000,004,000,000
Summations:
S(X) S(X^2) (exact) S(X^2) (using 10 or 12digit results)
3,000,003 3,000,006,000,005 3,000,006,000,000
([S(X)]^2) (exact result obtained using 13 or 15digit extended precision)
9,000,018,000,009
([S(X)]^2)/3 (exact result obtained using 13 or 15digit extended precision)
3,000,006,000,003
The exact calculation
sqrt ([S(X^2)  ([S(X)]^2)/n] / [n1]) of sample standard deviation is
sqrt [(3,000,006,000,005  3,000,006,000,003) / (31)] = 1
The "erroneous" calculation is based on 13 or 15digit internal computaion that utilizes the S(X^2) summations that were rounded off after each datum entry:
sqrt [(3,000,006,000,000  3,000,006,000,003) / (31)] = ERROR
(The HP15C's extendedprecision calculations for statistical summations are utilized within an example in the HP15C Advanced Functions Handbook listed on pages 208211.)
If extended precision had not been employed for the problem discussed here, both terms would have been rounded to 3,000,006,000,000  yielding the incorrect result of zero, instead of an error.
 KS
Edited: 26 Sept 2007, 2:09 a.m. after one or more responses were posted
