A quick precision test
|
06-03-2014, 12:36 AM
Post: #6
|
|||
|
|||
RE: A quick precison test
We always round to 16 or 34 digits when we return the result to the stack. These are the only precisions a user or a keystroke program can access. Unless you are coding numerics in C, this is all you get.
The 39 digits are used inside but only during the evaluation of a built in function. Once the result is known, it is rounded. The reason is simple: guard digits. I don't need to and generally cannot get all 39 digits correct but having the extra digits gives some headroom to get the rounded result correct or at least close. - Pauli |
|||
« Next Oldest | Next Newest »
|
User(s) browsing this thread: 2 Guest(s)