FORTRAN floating point accuracy problems

[Dieter]

(03-31-2016 07:30 PM)Dieter Wrote:  Also, possibly two more digits could be obtained if the integer and fractional part of dms would be handled separately.

This could be a possible solution then:

Code:

Function dms2dd(dms As Double)
Dim mmss, mm, ss As Long, ddd as Double

ddd = Int(dms)
mmss = (dms - ddd) * 1000000000
mm = mmss \ 10000000
ss = mmss Mod 10000000

dms2dd = ddd + mm / 60# + ss / 360000000#

End Function

Now nine decimals of dms are used for the conversion, i.e. the resolution is 0.00001 s. There is (almost) no limit for dms (max. ~2 billion hours).
The result should have eight valid digits after the decimal point.

Please note that VBA automatically rounds the result of "mmss = (dms - ddd) * 1000000000" to the nearest integer.
You may have to code this as MMSS = IDINT((DMS–DDD)*1.0D9 + 0.5D0).

Dieter