Question for Parisse about CAS floating point

04062015, 09:02 PM
(This post was last modified: 04062015 09:07 PM by jgoizueta.)
Post: #5




RE: Question for Parisse about CAS floating point
(04062015 01:44 PM)compsystems Wrote: hpprime calculator The 15.3434343434 result of the Prime uses 10 decimal places but has 12 digits of precision (counting the integer part). Actually, the 48 internal bits of the approximate CAS numbers can hold 15 decimal digits (and preserve their value when converted back to decimal) The actual CAS value 1519.0/99.0 could be presented as 15.3434343434343 with 15 digit precision, but the Prime refuses to do so. (you can check the actual internal value with Joe Horn's hex program and see that all those digits are there) I don't care about the precision of the approximate numbers in the CAS (it is larger than the 12 digits of HOME). What worries me is that each result is truncated (after being computed with the accuracy of a IEEE double), so numeric computations in the CAS may not be as well behaved as in home... Edited to mention the hex program Edit: oh, now I realize that the CAS values are shown to 12 decimal digits for consistency with the HOME world. 

« Next Oldest  Next Newest »

Messages In This Thread 
Question for Parisse about CAS floating point  jgoizueta  04052015, 06:01 PM
RE: Question for Parisse about CAS floating point  parisse  04062015, 06:43 AM
RE: Question for Parisse about CAS floating point  jgoizueta  04062015, 09:25 AM
RE: Question for Parisse about CAS floating point  compsystems  04062015, 01:44 PM
RE: Question for Parisse about CAS floating point  jgoizueta  04062015 09:02 PM
RE: Question for Parisse about CAS floating point  Joe Horn  04072015, 02:44 AM
RE: Question for Parisse about CAS floating point  jte  04102015, 07:58 PM
RE: Question for Parisse about CAS floating point  Joe Horn  04122015, 12:36 PM
RE: Question for Parisse about CAS floating point  jte  04122015, 08:11 PM
RE: Question for Parisse about CAS floating point  parisse  04122015, 05:36 PM
RE: Question for Parisse about CAS floating point  parisse  04132015, 06:40 AM

User(s) browsing this thread: 1 Guest(s)