FORTRAN floating point accuracy problems

[Dieter]

(04-11-2016 11:41 AM)Werner Wrote:  But it's your example a few posts up :

There was one zero missing. #-)

The example is supposed to have the 1 in the last significant (i.e. 15th) digit. So that should be 1/5,0000 00000 00001. Or 1E14/(5E14+1). That's why I said you should replace the "E13" parts by "E14". OTOH the decimal value 0,19999 99999 99999 6 was correct.

In this case Excel indeed returns 20 for INT(100*dms).

(04-11-2016 11:41 AM)Werner Wrote:  Well, never mind. If you do want to use d.ms format, which can't be represented exactly in binary floating point, you'll need rounding.

That's what I said. And that's why I definitely prefer BCD arithmetics.

BCD is the standard on pocket calculators where this d.ms notation is common, and here it works well. On the other hand most programming languages use binary arithmetics, so here a different approach should be preferred. For instance with a record, a struct or whatever it's called in your preferred language.

Dieter