Post Reply 
FORTRAN floating point accuracy problems
03-28-2016, 09:33 AM
Post: #7
RE: FORTRAN floating point accuracy problems
Hi and thanks for helping with this.

(03-27-2016 10:47 PM)Paul Dale Wrote:  For this kind of problem, I find first dealing with the sign and then getting into an integer domain as fast as possible and then applying corrections there is usually the best approach.

That would be ideal but here the truncation and loss of precision occurs during extracting the minutes and seconds. And once in integer values the results of division also produce truncated values. So, I can't see how to do this.

(03-27-2016 10:47 PM)Paul Dale Wrote:  You've got integer degrees already, multiply by 60 and get integer minutes, then again by sixty to get seconds (which will have to stay real). Then sanity check everything to deal with the corner cases.

If I multiply by 60 to move the minutes to the left of the decimal there is already some loss of precision for certain values like 30 minutes for example. I don't know how to extract the minutes or seconds. This is exactly the problem.

It ain't OVER 'till it's 2 PICK
Find all posts by this user
Quote this message in a reply
Post Reply 


Messages In This Thread
RE: FORTRAN floating point accuracy problems - HP67 - 03-28-2016 09:33 AM



User(s) browsing this thread: 2 Guest(s)