Post Reply 
Programming the INT and FRAC functions
01-05-2016, 07:37 PM
Post: #16
RE: Programming the INT and FRAC functions
(01-05-2016 12:42 PM)Pekis Wrote:  While the number 15.9999999999 (kept in internal memory) is displayed as 16, should the INT function have to display 15 (applying INT directly) or 16 (Rounding to 10 digits then applying INT) ?

Of course any function, including INT, should work on the exact value and not on the one rounded to display precision. Just imagine what would happen in FIX 0 mode. So the result of course has to be 15.

If you want to do it the other way round, simply have the number explicitely rounded before the function is called.

It's the programmer's job to take care of results that may come out as 15,9999999999 although the expected value is 16. This applies to any function. Even a simple x=y? test may go wrong. Luckily most calculators use BCD arithimetics – in binary things may be much worse.

Dieter
Find all posts by this user
Quote this message in a reply
Post Reply 


Messages In This Thread
RE: Programming the INT and FRAC functions - Dieter - 01-05-2016 07:37 PM



User(s) browsing this thread: 1 Guest(s)