FORTRAN floating point accuracy problems

[Dieter]

(04-03-2016 03:49 PM)HP67 Wrote:  Still having problems with specific input values. I tried my own version before I looked at yours and I tried yours also. Unsurprisingly yours worked a little better. But we are still having problems with specific values.

The problem seems to be the integer values mmss, mm and ss which are truncated so that a true result of mmss in case 3 may actually be 29.999999... instead of 30, which is then truncated to 29 minutes and 99.999... seconds. In VBA this problem does not occur as an assignment to an integer variable automatically rounds a real up or down to the next integer. Adding some explicit rounding should fix that:

You could replace line 0006
MMSS = (DMS - DDD) * 1000000000
with
MMSS = (DMS - DDD) * 1000000000 + 0.5D0
or even better
MMSS = IDINT((DMS - DDD) * 1000000000 + 0.5D0)

And once again the "expected" values are wrong. For instance

Code:

 TEST          DMS       EXPECT       RESULT         DIFF
   ...
    4   71.0030000   71.0083330   71.0083333    0.0000003
    5   42.2453000   42.4147220   42.4147222    0.0000002
    6   38.4225000   38.7069440   38.7069444    0.0000004
    7   29.3030000   29.5083330   29.5083333    0.0000003
    8    0.4949000    0.8302770    0.8302778    0.0000008
   ...

...should actually read...

Code:

 TEST          DMS       EXPECT       RESULT         DIFF
   ...
    4   71.0030000   71.0083333   71.0083333    0.0000000
    5   42.2453000   42.4147222   42.4147222    0.0000000
    6   38.4225000   38.7069444   38.7069444    0.0000000
    7   29.3030000   29.5083333   29.5083333    0.0000000
    8    0.4949000    0.8302778    0.8302778    0.0000000
   ...

All these results are dead on. Actually only case 2, 3, 9 and 14 are off, and I think this can be fixed by adding the suggested integer rounding.

The second program version has essentially the same issues:
Line 0005
IDMS = DMS * 1.0D7
should better be rounded to
IDMS = DMS * 1.0D7 + 0.5D0
or even better
IDMS = IDINT(DMS * 1.0D7 + 0.5D0)

(04-03-2016 03:49 PM)HP67 Wrote:  I want to figure out how to code a double precision (8 byte) truncation function as I mentioned earlier. Your rounding routine is really great. If my implementation wasn't hampered by the existing 4 byte truncation function maybe all this would work a lot better?

Most probably, yes.

Dieter