10-20-2021, 09:52 PM

I developed a program on HP50G to calculate the Theorical Normal Earth Gravity.

Then I checked some of my program key values with the ones given by Charles Karney, who suggested me to verify my figures myself by own calculations with the program Maxima.

And look what I found with Maxima:

bigfloat (1/298257223563) gives

3.352810664747480719845528618520555955732636178009763846626114916752568643973138090416416353127372856... × 10^-12

but, executing,

bigfloat (1/298.257223563) gives

3.3528106647474804902220313351790537126362323760986328125... × 10^-3,

having previously done

fpprec : 100

Surprising result for a dummy like me.

How do you tackle such an issue?

Thanks in advance for your help and insight on the topic.

Gil Campart

Then I checked some of my program key values with the ones given by Charles Karney, who suggested me to verify my figures myself by own calculations with the program Maxima.

And look what I found with Maxima:

bigfloat (1/298257223563) gives

3.352810664747480719845528618520555955732636178009763846626114916752568643973138090416416353127372856... × 10^-12

but, executing,

bigfloat (1/298.257223563) gives

3.3528106647474804902220313351790537126362323760986328125... × 10^-3,

having previously done

fpprec : 100

Surprising result for a dummy like me.

How do you tackle such an issue?

Thanks in advance for your help and insight on the topic.

Gil Campart