Post Reply 
Angle being re-writed on entering it
05-30-2016, 08:26 PM
Post: #2
RE: Angle being re-writed on entering it
(05-30-2016 05:08 AM)eried Wrote:  Hello, somebody in youtube asked me why if you enter: 1080°00'37'' you get 1080°00′37.00001″ ?

I guess is probably how the calculator works with angles internally... but, is there any workaround to avoid these extra decimals?

If you highlight the result and press the DMS key again, you'll see the reason. When you key 1080°00'37'' and press Enter, the value that actually gets entered is 1080.01027778, which is the closest possible 12-digit approximation. THAT value expressed in DMS notation is 1080° + 37.000008". This is rounded in the display to 1080°00'37.00001" which is closer to the actual value than 1080°00'37".

The problem is that the user THINKS that he entered exactly 1080°00'37'' into the calculator, whereas in fact he entered the decimal value 1080.01027778.

This is yet another example of HP following the HP philosophy of "What You See Is What You Have, warts and all" as opposed to other manufacturers' philosophy of "What You See Is Not Really What You Have Because We Fudged It To Look Pretty Because That's What You Probably Wanted To See".

<0|ɸ|0>
-Joe-
Visit this user's website Find all posts by this user
Quote this message in a reply
Post Reply 


Messages In This Thread
RE: Angle being re-writed on entering it - Joe Horn - 05-30-2016 08:26 PM



User(s) browsing this thread: 1 Guest(s)