Post Reply 
WP 31S/34S: H.MS+/H.MS- anomaly?
07-24-2015, 09:47 AM
Post: #10
RE: WP 31S/34S: H.MS+/H.MS- anomaly?
(07-20-2015 07:16 AM)Paul Dale Wrote:  The conversion of 18 minutes came out as 0.3000....0001 in the internal reals. I didn't check where the error came from but I suspect the multiplication by 1/60 is the culprit.

Since 1/60 is always rounded, the multiplication result cannot be exact under all circumstances. Compared to a division by 60, is a slightly (IMHO negligibly) faster execution really worth the trouble?

On the other hand it is always possible that an H.MS operation yields a result with a "60 seconds" part, so this has to be handled anyway.

Dieter
Find all posts by this user
Quote this message in a reply
Post Reply 


Messages In This Thread
WP 31S/34S: H.MS+/H.MS- anomaly? - 2far - 07-17-2015, 10:50 AM
RE: WP 31S/34S: H.MS+/H.MS- anomaly? - Dieter - 07-24-2015 09:47 AM



User(s) browsing this thread: 1 Guest(s)