Post Reply 
Programming the INT and FRAC functions
01-05-2016, 12:42 PM
Post: #15
RE: Programming the INT and FRAC functions
Hello,

Beware of the internal precision before applying INT ...

e.g. with a 12-digit mantissa and 10 digits displayed:

While the number 15.9999999999 (kept in internal memory) is displayed as 16, should the INT function have to display 15 (applying INT directly) or 16 (Rounding to 10 digits then applying INT) ?
Find all posts by this user
Quote this message in a reply
Post Reply 


Messages In This Thread
RE: Programming the INT and FRAC functions - Pekis - 01-05-2016 12:42 PM



User(s) browsing this thread: 1 Guest(s)