WP 31S/34S: H.MS+/H.MS- anomaly?
|
07-24-2015, 09:47 AM
Post: #10
|
|||
|
|||
RE: WP 31S/34S: H.MS+/H.MS- anomaly?
(07-20-2015 07:16 AM)Paul Dale Wrote: The conversion of 18 minutes came out as 0.3000....0001 in the internal reals. I didn't check where the error came from but I suspect the multiplication by 1/60 is the culprit. Since 1/60 is always rounded, the multiplication result cannot be exact under all circumstances. Compared to a division by 60, is a slightly (IMHO negligibly) faster execution really worth the trouble? On the other hand it is always possible that an H.MS operation yields a result with a "60 seconds" part, so this has to be handled anyway. Dieter |
|||
« Next Oldest | Next Newest »
|
User(s) browsing this thread: 1 Guest(s)