Post Reply 
Floating point bug in Canon F-789SGA
12-06-2014, 06:37 AM (This post was last modified: 12-06-2014 06:43 AM by Gerald H.)
Post: #6
RE: Floating point bug in Canon F-789SGA
(12-05-2014 09:39 PM)Peter Van Roy Wrote:  
(12-05-2014 02:20 PM)Gerald H Wrote:  i'm not sure it's a bug.

Handbook says:

"Precision +/- 1 at the 10th digit for a single calculation
+/- 1 at the least significant for exponential display" (page 10)

which I don't really understand, but could imply you are requiring accuracy above specifications.

The specification says clearly that internal precision is 18 digits (see page 11 of F-789SGA Scientific Calculator User Instructions). It is quite common for calculators to have higher internal precision than what they display. My observations show that floating point addition has a bug at the 17th and 18th digits.

Can't find the referenced text in "User Instructions" E-IM-2725 page 11 nor in f789sga.pdf E-IE-455 page 11.
Find all posts by this user
Quote this message in a reply
Post Reply 

Messages In This Thread
RE: Floating point bug in Canon F-789SGA - Gerald H - 12-06-2014 06:37 AM

User(s) browsing this thread: 1 Guest(s)