Post Reply 
FORTRAN floating point accuracy problems
04-11-2016, 12:35 PM
Post: #41
RE: FORTRAN floating point accuracy problems
(04-11-2016 12:03 PM)Dieter Wrote:  In this case Excel indeed returns 20 for INT(100*dms).

Indeed it does.. but it's an example of Excel 'fiddling' - it adjusted the results to match its decimal audience ;-)

Free42 Binary returns 20 only with 1e16 - because then 5e16 + 1 equals 5e16.
Excel really should use decimal arithmetic..

Cheers, Werner

41CV†,42S,48GX,49G,DM42,DM41X,17BII,15CE,DM15L,12C,16CE
Find all posts by this user
Quote this message in a reply
Post Reply 


Messages In This Thread
RE: FORTRAN floating point accuracy problems - Werner - 04-11-2016 12:35 PM



User(s) browsing this thread: 1 Guest(s)