Post Reply 
FORTRAN floating point accuracy problems
03-27-2016, 03:02 PM
Post: #1
FORTRAN floating point accuracy problems
I'm getting some unacceptable results from rounding in old FORTRAN. I know there are some very talented mathematicians on the board here so I wonder if you guys would be willing to help me. To say I'm bad at math would be an understatement! I have no idea where things are going wrong.

I'm trying to accomplish something very simple in a language which has very little in the way of helpful functions to do what I want. Given a floating point input value for degrees, minutes, seconds in the form dd.mmsst... I want to produce the value in decimal degrees. This is no problem of math even for me. The problem is extracting the digits with only a truncation function and nothing else helpful in the repetoire as far as I can see.

Here is the piece of code:

Code:

      DOUBLE PRECISION FUNCTION DMS2DD (DMS)
      REAL*8 DMS, DD, MM, SS
      SS = (DMS - IDINT(DMS)) * 100.0D0
      SS = ((SS - IDINT(SS))  * 100.0D0) / 60.0D0
      MM = (IDINT((DMS - IDINT(DMS)) * 100.0D0) + SS) / 60.0D0
      DMS2DD = IDINT(DMS) + MM
      RETURN
      END

and here are the results of some tests:

Code:

  TEST    DMS         EXPECTED    RESULT       DIFFERENCE

     1    89.11150    89.18750    89.18750     0.00000
     2    12.15000    12.25000    12.25000     0.00000
     3    33.30000    33.50000    33.51111     0.01111
     4    71.00300    71.00830    71.00833     0.00003
     5    42.24530    42.41470    42.41472     0.00002
     6    38.42250    38.70690    38.70694     0.00004
     7    29.30300    29.50830    29.50833     0.00003
     8     0.49490     0.83030     0.83028     0.00002
     9    75.15000    75.25000    75.25000     0.00000
    10    43.22300    43.37500    43.37500     0.00000
    11     9.33450     9.56250     9.56250     0.00000
    12    33.57522    33.96450    33.96450     0.00000
    13    13.07244    13.12345    13.12345     0.00000
    14    21.30000    21.50000    21.50000     0.00000
    15    59.47211    59.78920    59.78920     0.00000
    16    65.11010    65.18360    65.18360     0.00000

You can see in test case #3 above there is a big problem extracting the minutes. The code winds up with 29 instead of 30 because of the truncation function.

I find this kind of problem is coming up a lot in various calculations I'm doing and I don't know how to code around it.

Can anybody please explain how I can rearrange the computations or do something differently to avoid this problem?

Thanks.

It ain't OVER 'till it's 2 PICK
Find all posts by this user
Quote this message in a reply
Post Reply 


Messages In This Thread
FORTRAN floating point accuracy problems - HP67 - 03-27-2016 03:02 PM



User(s) browsing this thread: 1 Guest(s)