FORTRAN floating point accuracy problems

[Dieter]

(03-31-2016 12:51 PM)HP67 Wrote:  FORTRAN IV's IDINT returns a 4 byte integer. There is no 8 byte INT() function in FORTRAN IV.

That's the problem.

(03-31-2016 12:51 PM)HP67 Wrote:  I see three possible approaches:

I see a fourth one: simply do it all in integer arithmetics.

Right at the beginning the dms input is converted to an integer. If the largest 4-bit integer is 2147483647, the largest dms input is 214,7483647. So we just multiply the input by 10.000.000 and use that as an integer.

I tried this in Excel VBA and it works well:

Code:

Function dms2dd(dms as double)
Dim x, ddd, mmss, mm, ss As Long

x = dms * 10000000
ddd = x \ 10000000
mmss = x Mod 10000000
mm = mmss \ 100000
ss = mmss Mod 100000

dms2dd = ddd + mm / 60# + ss / 3600000#

End Function

Note: the "\" operator means integer division. The "#" suffix marks a variable or value as double precision. I think you would use 60.0D0 and 3.6D6.

The major limitation of this method is the domain of the dms argument. The largest angle is 214 degrees and the precision of dms is 0.001 seconds. In other words, everything beyond the 7th decimal is neglected. This way the result has 5-6 valid decimals (0,001/3600 =1,7E–6).

You may check if Fortran truncates or rounds x when it is converted in the first line (x=dms*10000000). Maybe you will have to code this as X = IDINT(DMS*1.0D7+0.5D0).

What do you think?

Dieter