Post Reply 
FORTRAN floating point accuracy problems
03-30-2016, 08:32 PM
Post: #14
RE: FORTRAN floating point accuracy problems
(03-30-2016 06:38 PM)Dieter Wrote:  32 bits mean 9,3 decimal digits, which matches the results. Since 2 digits are used for the integer portion, rounding the mmss part to 7 decimals shoud be OK.

Dieter - what is the meaning of "9,3 decimal digits" ? I get that due to internal representation inaccuracies you can only get somewhere between 9 and 10 reliable decimal digits, but what is the meaning or interpretation of the ",3" ?

These threads are interesting to follow, I'm always learning something new, but I've not seen this usage before. Or maybe have simply forgotten it in the last 30+ years since doing this stuff on a daily basis.

--Bob Prosperi
Find all posts by this user
Quote this message in a reply
Post Reply 


Messages In This Thread
RE: FORTRAN floating point accuracy problems - rprosperi - 03-30-2016 08:32 PM



User(s) browsing this thread: 1 Guest(s)