Post Reply 
newRPL: [UPDATED April 27-2017] Firmware for testing available for download
10-22-2015, 09:22 PM
Post: #80
RE: newRPL: [UPDATED Oct-19-2015] Firmware for testing available for download
(10-22-2015 06:16 PM)Werner Wrote:  But 527.67 x (5/9) does produce 293.15 ...
on a 12-digit machine:

527.67 x 0.555555555556
= 293.15000000023452

I see you multiply by 0.555555555555, that is not the correctly rounded 5/9
Alternatively you may multiply by 5 and then divide by 9 - only the latter will produce a rounding error.

Cheers, Werner

I never said there's a rounding error, the number is computed properly. But 5/9 is not exact, and it's therefore marked as approximated. The "19.9:" was a normalization bug, the number was not normalized and confused the text conversion routine.
When I show the number 0.55555555... I always add the ellipsis because there's a variable number of digits, up to 2000 digits but you are right, the last one is a 6.
Anyway, the bug is fixed, and the calculation returns exactly 20. (or should I say exactly-approximatedly 20. :-)
Find all posts by this user
Quote this message in a reply
Post Reply 


Messages In This Thread
RE: newRPL: [UPDATED Oct-19-2015] Firmware for testing available for download - Claudio L. - 10-22-2015 09:22 PM



User(s) browsing this thread: 3 Guest(s)