Floating point bug in Canon F789SGA

12052014, 02:20 PM
Post: #4




RE: Floating point bug in Canon F789SGA
(11302014 07:32 PM)Peter Van Roy Wrote: I recently bought a Canon F789SGA calculator because its specification intriguingly mentions that it has 18 digit precision. Tests show that it actually has 20 digit internal precision, and transcendental functions actually achieve this precision. That is the nice part. i'm not sure it's a bug. Handbook says: "Precision +/ 1 at the 10th digit for a single calculation +/ 1 at the least significant for exponential display" (page 10) which I don't really understand, but could imply you are requiring accuracy above specifications. Specs are certainly exceeded by the PFact function, where the greatest factor is declared "unfactored" whether prime or composite  in fact, the greatest factor found is prime for all values < 1009^2. Overall a pleasant calculator & accurate & integration is annoyingly slow. 

« Next Oldest  Next Newest »

User(s) browsing this thread: 1 Guest(s)