Post Reply 
Round off error calculation example
07-23-2017, 10:16 AM
Post: #1
Round off error calculation example
I came across this example from youtube about the round off error on the older computer start from the earlier Pentium CPU which is about the floating point units. Here is the sample: ((2^54)+1)-(2^54) = 0 or 1 ?

I try this on two calculators Casio fx-300ES result is 0 and HP Prime result is 0
and try on the Google Search Calculator result is 0 and on the HP OmniGo 100 result is 0
Only from the WolfarmAlpha website gave result to 1
Find all posts by this user
Quote this message in a reply
Post Reply 


Messages In This Thread
Round off error calculation example - Gamo - 07-23-2017 10:16 AM



User(s) browsing this thread: 1 Guest(s)