The Museum of HP Calculators

HP Forum Archive 19

 Rounding conventionsMessage #1 Posted by Wayne Best on 4 July 2009, 9:16 a.m. Dear All, I'm sure this must have been discussed before but I couldn't find it so apologies if it's a repeat. When trained as a scientist many years ago I was taught that when rounding numbers the convention was to round 5's to the nearest even number, thereby averaging out any introduced error when working with data. Everything I've seen and done in science in the last 30 years has reinforced this convention. I've only just found out that rounding away from zero (i.e., rounding up for positive numbers) is also an acceptable convention according to IEEE Standard Arithmetic. While this may be ok for accounting and finance, it introduces a bias when rounding data sets, and would appear to be unsuitable for many scientific applications especially those involving sets of data. I've used numerous calculators (nearly all HP) and they all round away from zero even calculators clearly designed for scientists and engineers like the 41C and 15C. I've also looked through a few of my manuals and I can't find any real discussion on the subject and there doesn't seem to be any setting or flag to allow the user to choose between the rounding methods. This is very inconvenient because it means that the number displayed is not the true rounded number (according the the scientific round to even method). So, my question is, why do all these calculators, even the scientific ones, use this round away from zero method, and is there any way around the problem (other than witting a program)? Are there any calculators on the market that round "properly" for a scientist? Thanks, Wayne

 Re: Rounding conventionsMessage #2 Posted by Gerson W. Barbosa on 4 July 2009, 11:53 a.m.,in response to message #1 by Wayne Best On the HP-71B you can choose one out of four rounding methods: ```OPTION ROUND NEAR (default) OPTION ROUND ZERO rounds towards zero OPTION ROUND POS rounds up OPTION ROUNT NEG rounds down ``` These can also be selected by clearing or setting the appropriate flags (-11 and -12). See HP-71B Owner's Manual, pages 56 and 198. For more on HP rounding philosophy refer to and Regards, Gerson. Edited: 4 July 2009, 11:54 a.m.

 Re: Rounding conventionsMessage #3 Posted by Wayne Best on 4 July 2009, 12:22 p.m.,in response to message #2 by Gerson W. Barbosa Thanks Gerson, I have a 71B and this still doesn't round the display "properly". The 71B (like other calcs) still appears to use "round away from zero" for the display when the last digit is 5. For example, enter 2.005 with a fixed 2 decimal place format and it will show 2.01 instead of what most scientists would want, 2.00. It seems to me from checking p56 of the manual that "OPTION ROUND NEAR" only dictates how the 71B rounds the final 12th digit, not how it rounds the currently viewed display (eg, fixed 2). Cheers Wayne

 Re: Rounding conventionsMessage #4 Posted by Quan on 4 July 2009, 1:03 p.m.,in response to message #3 by Wayne Best Hi Wayne, I am curious as to why you or most scientists would want to round down to 2.00 instead round up to 2.01? If there was a reason, I surely cannot recall as I can only remember to round up. Last year I helped my niece (3rd grade) with her homework, and the school also taught rounding up. Thanks. Quan

 Re: Rounding conventionsMessage #5 Posted by Gerson W. Barbosa on 4 July 2009, 1:45 p.m.,in response to message #4 by Quan Quote: I am curious as to why you or most scientists would want to round down to 2.00 instead round up to 2.01? Please follow the link below and take a look at Round-to-even method: Regards, Gerson.

 Re: Rounding conventionsMessage #6 Posted by Don Shepherd on 4 July 2009, 1:45 p.m.,in response to message #1 by Wayne Best Schools teach it this way: if the digit to the right is 0, 1, 2, 3, or 4, leave the rounding-to digit alone; if the digit to the right is 5, 6, 7, 8, or 9, add 1 to the rounding-to digit. So, five digits go one way, five digit go the other way. Sounds fair to me. I thought that is how all calculators do it too.

 Re: Rounding conventionsMessage #7 Posted by Thomas Radtke on 4 July 2009, 2:55 p.m.,in response to message #6 by Don Shepherd Don, 0.5 is the average of 0 and 1. If you always round it up, you'll introduce a bias. If your rounding depends on whether the INT is even or odd, you get rid of the bias in most cases. That's how I understand it. I might be wrong though :^).

 Re: Rounding conventionsMessage #8 Posted by Walter B on 4 July 2009, 5:45 p.m.,in response to message #7 by Thomas Radtke That's what I like at this forum: There's a good chance I learn something I didn't know before. I admit I had no idea about round-to-even and the reasons for it (or have forgotten that completely), though having a Ph.D. in physics and having worked with large data sets (way back). IMHO, in real life the error introduced by rounding 5 up (as I did it so far all the time) seems neglectable as long as you discard more than one digit in rounding.

 Re: Rounding conventionsMessage #9 Posted by Don Shepherd on 4 July 2009, 6:12 p.m.,in response to message #7 by Thomas Radtke So, 92.5 would be rounded to 92 (nearest even number). OK, one of you engineers will have to come to my math class and explain to my students why a 92.5 average results in a B now, rather than an A, on their report cards (standard range for A is 93-100). Jeesh, no wonder kids hate math! : )

 Re: Rounding conventionsMessage #10 Posted by Rodger Rosenbaum on 4 July 2009, 7:05 p.m.,in response to message #9 by Don Shepherd Quote: So, 92.5 would be rounded to 92 (nearest even number). OK, one of you engineers will have to come to my math class and explain to my students why a 92.5 average results in a B now, rather than an A, on their report cards (standard range for A is 93-100). Jeesh, no wonder kids hate math! : ) A 92.5 average should get a B, rather than an A, shouldn't it? They don't hate averaging, do they? Rounding is easier than averaging. You can't round to 2 digits before you do the comparison "IF score < 93 THEN grade = B.

 Re: Rounding conventionsMessage #11 Posted by Katie Wasserman on 4 July 2009, 3:27 p.m.,in response to message #1 by Wayne Best Actually, I'm not so sure that it's ok in finance either. I work on wall street and I've never personally seen any programs using the round-to-even method, but they call it "Banker's Rounding" here which must mean that some financial places do that. I can't think of a single hand-held calculator (not even the HP-50g) that has round-to-even as a possible rounding method. But some large math packages have this; Mathematica's "rint" function, for example. Edited: 6 July 2009, 2:25 p.m. after one or more responses were posted

 Re: Rounding conventionsMessage #12 Posted by Rodger Rosenbaum on 4 July 2009, 7:08 p.m.,in response to message #11 by Katie Wasserman Quote: Actually, I'm not sure sure that it's ok in finance either. I work on wall street and I've never personally seen any programs using the round-to-even method, but they call it "Banker's Rounding" here which must mean that some financial places do that. I can't think of a single hand-held calculator (not even the HP-50g) that has round-to-even as a possible rounding method. But some large math packages have this; Mathematica's "rint" function, for example. All the Saturn based calcs do round-to-even arithmetic (except the HP71 can do something else if selected); it's just the display rounding that isn't round-to-even

 Re: Rounding conventionsMessage #13 Posted by Katie Wasserman on 5 July 2009, 12:25 a.m.,in response to message #12 by Rodger Rosenbaum I thought that we're talking about display rounding in this thread. The original post said: Quote: While this may be ok for accounting and finance, it introduces a bias when rounding data sets, and would appear to be unsuitable for many scientific applications especially those involving sets of data. The "data sets" imply that the data was from an experiment or whatever and input into the calculator. So I thought that the discussion has been about display rounding on input not rounding of the internal functions to their maximum accuracy. Edited: 5 July 2009, 12:31 a.m.

 Re: Rounding conventionsMessage #14 Posted by Rodger Rosenbaum on 5 July 2009, 3:51 a.m.,in response to message #13 by Katie Wasserman But display rounding on input has no effect on the data that ends up in a data set in the calc (to be processed further, presumably). The data could have been captured from an HPIL device, such an HP3468 voltmeter. In that case, clearly display rounding has no effect on the captured data. Even if the data were typed in, the fact that the typed-in data appears different in the display doesn't change what is stored in a variable. For example, if the calc is in FIX 2 mode and you type in 2.005, it doesn't matter that the display shows 2.01; the number upon which arithmetic will be performed is still 2.005 The display rounding only matters when the human operator is looking at the final result.

 Re: Rounding conventionsMessage #15 Posted by Katie Wasserman on 5 July 2009, 12:28 p.m.,in response to message #14 by Rodger Rosenbaum How about if assume that the user enters the data via a program (keystroke entry or HP-IL device) and that the program uses the Round Function in the calculator (usually marked as RND on HP calculators) to force the data to N decimal places as set by the display format. I think that this would be a likely scenario for entering data into a calculator and display rounding is important since RND works based on display rounding.

 Re: Rounding conventionsMessage #16 Posted by Wayne Best on 6 July 2009, 8:05 a.m.,in response to message #14 by Rodger Rosenbaum Just to put this into the context of my real life situation. I was using my calculator to calculate the purity of a number of samples of a chemical. There are a number measurements (weight, volume, detector response, etc) for each purity calculation which I manually enter into the calculator to produce the answer. The number of significant figures I need (based on errors involved in the measurements) is 4, ie xx.xx% So I set my calculator to FIX 2, and just read off the (rounded) answers. I report these individual purity measurements separately. At this point all the answers have been rounded up which is probably no real issue but then I want to average these purities and this is when the bias introduced by rounding up (rather than to even) becomes an issue because the average purity will be slightly higher than it really should because all the 5's have been rounded up. I realise the error will be small but it could (and should) be removed entirely by simply rounding to even. I hope this makes sense and explains why this is a real problem. You may well ask why didn't I just use a spreadsheet for the whole process but this is another story .......

 Re: Rounding conventionsMessage #17 Posted by Garth Wilson on 6 July 2009, 3:08 p.m.,in response to message #16 by Wayne Best Makes sense; but even if you don't want the calculator to keep all the values simultaneously and then average them at the end, you could still have the calc keep a running average and avoid the problem. One register keeps the total of the results, and another one keeps the number of results added together to get that total, so at any time you can divide one by the other to get the average so far, to as many digits as you want, even if individual purities were dispalyed in only FIX 2.

 Re: Rounding conventionsMessage #18 Posted by Wayne Best on 7 July 2009, 10:42 a.m.,in response to message #17 by Garth Wilson Thanks Garth That would certainly solve the problem in many cases. But there are a couple of issues that still remain. I should also point out that as well as needing the average, I also needed the standard deviation because I had to multiply this by a factor (based on the number of samples) to give an error to a 95% confidence limit. Firstly, I was using a HP21 which, with only 1 memory register and no Stats, wasn't capable of doing what you suggest. Ok, I could have reached into my calculator cupboard and pulled out any number of calculators which could have handled this easily so it's not really much of an excuse. Secondly, and this is my main concern, I have collaborations with biologists, pharmacologists, computational chemists, and other disciplines. We routinely exchange data sets back and forth for various types of analysis. Naturally, we only send/receive the data with the appropriate number of significant figures, ie rounded data. So now we have (or sent) a biased data set which then gets used in further analysis. I accept that any errors will be small and in most circumstances will not have a material effect on the results. But I think the point is that, if the rounding were done "properly" then there would be no issue at all. Thirdly, and this is the actual event which highlighted the issue to me. I had to report the data and the average result to the client. So the option was to report the average of the rounded data (even though this was slightly higher than the real average) or report the real average based on the unrounded data even though this wasn't exactly the average of the reported results. It sounds rather pedantic but the report was to be presented in court in relation to a patent dispute. If the report had even a minor inconsistency like this, it could impact on our credibility as an expert witness. In the end I just manually rounded the data using the round to even method and that was fine. But it did strike me as odd that scientific calculators couldn't round "properly" and hence my post on this forum. I was hoping that there might be some undocumented trick such as setting a flag but it seems not. I'll be the first in line to buy the first calculator than can round to even! It's true that most of our "number crunching" these days is done in spreadsheets but there's nothing quite like running the numbers through a calculator yourself to really have faith in the answer.

 Re: Rounding conventionsMessage #20 Posted by Jeff Kearns on 7 July 2009, 1:50 p.m.,in response to message #18 by Wayne Best I too have been following this thread with interest and I found it odd that the data sets are not analyzed in Excel or even better, a proper statistical analysis package like Minitab or Statistica. On the one hand, there is the issue of expert witness credibility and on the other hand the desire (for whatever reason, albeit not fully explained) to use a time-consuming handheld calculator, with no 'paper trail' to show exactly how the data were processed. For your sake, I hope that there are not more a handful of data pairs... Would you be willing to share examples of the data with the forum in such a fashion that we are unable to determine the units or purpose - just for discussion purposes? Jeff Kearns

 Re: Rounding conventionsMessage #21 Posted by Jeff Kearns on 7 July 2009, 1:59 p.m.,in response to message #20 by Jeff Kearns My curiosity is really piqued now... I have analyzed some rather interesting data sets in my day (LeanSigma practitioner) and have always been impressed with the rapidity and reliability of a HP 49G+ or a 50G to calculate summary statistics, confidence intervals and conduct hypothesis testing on relatively small data sets (>20 pairs). Just how many of your data points end in 5's anyway? What is the worst case difference (statistically speaking) in mean values between the two rounding methods for your data sets? It seems to me that the difference would be negligible. I need to see the data!!! Jeff Edited: 7 July 2009, 2:03 p.m.

 Re: Rounding conventionsMessage #22 Posted by Wayne Best on 8 July 2009, 12:14 p.m.,in response to message #21 by Jeff Kearns Dear Jeff and others Thanks for your comments. I was trying to keep my description of the problem and how it arose simple so as not to confuse or bore people but here is a bit more background for those that are interested in the origin. Unfortunately, we don't have any nice stat packages and we probably couldn't justify the cost as most of our number crunching is pretty basic stuff. And no, I'm neither a troglodyte nor masochist, I do most of this sort of work in a spreadsheet, not a calculator :) I use OpenOffice not MS Excel but I don't think there's much difference. OpenOffice, like the calculators, does not round to even but rather rounds all the 5's up. In my "real life" example The actual data to be analysed was relatively small, 6 standards (in triplicate) to calculate a standard response (by linear regression) and then 8 samples (in triplicate) which were fitted to the regression to calculate concentration from which purity was finally calculated. There was several stages involved in the calculations with the early stages being manually recorded in a lab book before being transferred to a spreadsheet for further calculations and linear regression. The final spreadsheet was then brought to me for checking. I like to check results ab initio by an independent process rather than just check what has been done. As the data set was relatively small I thought I'd just check a few selected preliminary calculations manually by calculator (HP21) and then check the regression and line fitting for which I pulled out my 48GX (I wasn't aware that it could calculate confidence limits so thanks for the tip, I'll look into that). This is the point at which the saga began to unfold because my 8 manually calculated purity results did not agree with the spreadsheet. It took some time to find the source of the discrepancy between the two methods which had nothing to do with rounding up versus rounding to even but an inappropriate manual rounding of the slope and intercept in the spreadsheet was part of the problem. However, in tracking down the problem I started to take a closer look at potential sources of error and the rounding issue arose. I think the maximum error is only 1 in the final significant digit. So if you have 4 significant digits the error should only be between 1 in 1,000 to 1 in 9,999 but if you are rounding to only 2 significant figures it could in theory be as high as 1 in 10 (it's just gone midnight here and my mind is getting a little fuzzy so you might want to check that :)) From this simple data set (2.05, 2.15, 2.25, 2.35, 2.45, 2.55) you can see that the error is 0.1 in 2.3 or just over 4%, not insignificant at all. Ok, it does contain a lot of 5s. You ask "how many numbers end in 5 anyway" and this really does depend on where the numbers come from and how many significant digits you drop off a t a time. I guess from normal calculated numbers one would expect only 10% of numbers would end in five but for some visual measurements 5s are very common, for example melting points are normally given to the nearest 1/2 degree so every second measurement might end in 5. I need to look into this a bit more and see what the typical error might be in a more typical data set and significant figure reduction. I suspect it only really becomes significant if one reduces to 1 (not very usual) or 2 (not uncommon) significant figures. Ironically, I think the problem is worse when you only drop one figure because 1 in 10 are 5s whereas if you drop 2 at a time the number of 50's is only 1 in 100. I hope that makes sense. Also, I think I have found one example of round to even....the HP-95LX has a RND function which seems to do this but I need to look at this in more detail as I don't have a manual for this calc. Thanks all

 Re: Rounding conventionsMessage #23 Posted by Katie Wasserman on 8 July 2009, 1:27 p.m.,in response to message #22 by Wayne Best I tried the RND function on my 95LX and 200LX and I don't think that it works any differently than the RND function on all the other HP calculators that I've tried. Perhaps I'm doing something wrong, is there a particular reason why you think it does a round-to-even on the 95LX? -Katie

 Re: Rounding conventionsMessage #24 Posted by Wayne Best on 9 July 2009, 8:52 a.m.,in response to message #23 by Katie Wasserman Hi Katie I'm pretty sure my 95LX is using "round to even" . I am using the calculator feature of the device and have the format set to FIX 2. If I input 2.005 and presse ENTER the display shows 2.01 as all the other calculators do (of course the number is still 2.005). If I then press the RND function (F2) the display changes to 2.00, ie it has actually rounded down to even. The only downside of this is that it actually changes the number and not just the way it's displayed. It would be nice to keep the original number but just changed the display rounding. When I try the same procedure using the RND feature of the 41CV or 15C the result is 2.01 not 2.00 so it seems that the 95LX is the odd one out here. I wonder why this is? Perhaps it has something to do with it running on a DOS system? Unfortunately, I don't have a manual for the 95LX. Give it a try and let me know if it works for you, perhaps there is an option on the 95LX for this? Regards Wayne

 Re: Rounding conventionsMessage #25 Posted by Thomas Radtke on 9 July 2009, 3:02 p.m.,in response to message #24 by Wayne Best It appears to be the default method of the 95LX calculator application. Interesting. Edit: I almost forgot I have the manual. The RND description is wrong. It states that the number is rounded to correspond with the number displayed. Edit 2: Serial ABD3128A03721 Edited: 11 July 2009, 3:44 a.m.

 Re: Rounding conventionsMessage #26 Posted by Katie Wasserman on 10 July 2009, 9:31 p.m.,in response to message #24 by Wayne Best Wayne, I just tried your example on my 95LX and it doesn't work as yours does. Setting format Fix 2 and entering 2.005 will show 2.01 certainly. But when I press RND (F2 on the MATH menu) the X register stays at 2.01. I get the same results in ALG mode as I do in RPN mode I don't think that there are any other options that can be set in the calculator that would effect rounding. Could your 95LX and mine be different ROM versions with a changed Calc program? My 95LX is the last release based on the information found here. The serial numbering prefix found on this same page does not agree with the number of my machine, it's SG30300278 which indicates that it was made early in 1993 I think. -Katie

 Re: Rounding conventionsMessage #27 Posted by Wayne Best on 11 July 2009, 7:15 a.m.,in response to message #26 by Katie Wasserman Hi Katie Gee, that's really odd. Mine is a USA made model S/N ABA3139A00067 and is the early 512k version. I understand (from the excllent link in your post) that the calculator applet is based on the 19BII which also rounds up not even. I wonder if mine is an earlier version which came with round to even which was "fixed" in later models. Or perhaps mine is broken or I'm doing something wrong? I'd be interested to hear from other 95lx owners about the behaviour of theirs. Regards Wayne

 Re: Rounding conventionsMessage #28 Posted by Rodger Rosenbaum on 7 July 2009, 3:31 p.m.,in response to message #18 by Wayne Best Quote: It's true that most of our "number crunching" these days is done in spreadsheets but there's nothing quite like running the numbers through a calculator yourself to really have faith in the answer. Does Excel round-to-even? The answer to your original question is that HP calcs don't do what you want when display rounding is in effect. However, if you want a properly rounded-to-even result with a fixed 2 digits after the decimal point that will also display properly when the calc is in FIX 2 mode, 6 keystrokes will do it on a Saturn based calc: EEX 9 + EEX 9 - You could assign this as a tiny program to a used defined key, if the calc has that ability. Otherwise, just remember it. And, of course, if you want a different number of digits after the decimal point, adjust the constant 9.

 Re: Rounding conventionsMessage #29 Posted by Katie Wasserman on 10 July 2009, 9:42 p.m.,in response to message #28 by Rodger Rosenbaum Quote: Does Excel round-to-even? No. Microsoft calls in banker's rounding. Read this quite extensive writeup about rounding in VBA, Excel and in general. Edited: 10 July 2009, 9:44 p.m.

 Re: Rounding conventionsMessage #30 Posted by Rodger Rosenbaum on 11 July 2009, 12:47 a.m.,in response to message #29 by Katie Wasserman What I see on that page is: ```Product Implementation ---------------------------------------------------------------------- Visual Basic for Applications 6.0 Banker's Rounding Excel Worksheet Symmetric Arithmetic Rounding SQL Server Either Symmetric Arithmetic Rounding or Symmetric Round Down (Fix) depending on arguments Java Math library Asymmetric Arithmetic Rounding ``` It looks to me like you would only get Banker's rounding in Excel within a VBA application. Do you read it differently?

 Re: Rounding conventionsMessage #31 Posted by Katie Wasserman on 11 July 2009, 2:38 a.m.,in response to message #30 by Rodger Rosenbaum Exactly, you'd have to write in VBA to get banker's rounding in Excel. There's no function within Excel itself for that, although it does have ROUNDUP, ROUNDDOWN and regular ROUND.

 Re: Rounding conventionsMessage #32 Posted by Werner on 13 July 2009, 7:28 a.m.,in response to message #28 by Rodger Rosenbaum or, as I use it: ```2 E9 + LASTX - ``` Cheers, Werner

 Re: Rounding conventionsMessage #33 Posted by Rodger Rosenbaum on 14 July 2009, 1:14 p.m.,in response to message #32 by Werner Why 2 E9 rather than 1 E9?

 Re: Rounding conventionsMessage #34 Posted by Werner on 15 July 2009, 1:33 p.m.,in response to message #33 by Rodger Rosenbaum so that it works for negative numbers as well, eg: ``` -55.451 + 1e9 - 1e9 => -55.451 but -55.451 + 2e9 - 2e9 => -55.45 ``` Cheers, Werner

 Re: Rounding conventionsMessage #35 Posted by Rodger Rosenbaum on 16 July 2009, 2:02 a.m.,in response to message #34 by Werner Of course. I assumed without good reason that he would be rounding positive data. I should have mentioned that for negative numbers, he should use the sequence EEX 9 - EEX 9 +. I do all my calculator programming on an HP50 these days, and to enter 2E9 takes 1 more keystroke than to enter 1E9; that is, 2 EEX 9 versus EEX 9. And, the HP50 has no LASTX key; LASTARG isn't available as a key on the non-user-shifted keyboard. I happened to look at an old thread on rounding from 2003, and I think it might be fun to revisit it. I'll start a new thread.

 Re: Rounding conventionsMessage #36 Posted by Wayne Best on 4 July 2009, 8:43 p.m.,in response to message #11 by Katie Wasserman Thanks Katie I'ver never heard it called Banker's rounding before but now I have a new respect for the financial sector. I thought the banks would always want to claim that extra half cent for themselves :) Regards, Wayne

 Re: Rounding conventionsMessage #37 Posted by Rodger Rosenbaum on 4 July 2009, 6:48 p.m.,in response to message #1 by Wayne Best You said: Quote:While this may be ok for accounting and finance, it introduces a bias when rounding data sets, and would appear to be unsuitable for many scientific applications especially those involving sets of data. The HP71 and later Saturn based calcs round-to-even (unless you set the rounding mode in the HP71 to something else) during arithmetic calculations, so this problem only shows up in the display of final results. One way around it without doing any programming is to set your calculator to display all 12 digits in the result and do the rounding to a smaller number of significant digits yourself. Or, if you have a large number of final results to print at some lower number of significant digits, you can cause that number of digits to be rounded-to-even using a technique I described on comp.sys.hp48 in 2004: ------------------------------------------------------------ Imagine that you want to round a number on your calculator to a reduced number of digits, smaller than the calculator's full precision, with proper rounding (round-to-even, that is!). If the calculator does N digit arithmetic and you want a K digit rounded result, R, the formula: R = value - value * 10^(N-K) + value * 10^(N-K) or R = value + value * 10^(N-K) - value * 10^(N-K) gives the K digit properly rounded result where "value" is the original full-precision number you want to round to K digits. These formulas use Dekker's method, and only work correctly if the calculator does proper rounding during arithmetic. For example, on the HP48, if we have the number 123450000001 in the display and want to round this to 4 digits, we compute: 123450000001 - 123450000001 * 1E8 + 123450000001 * 1E8 and we get: 123500000000 But if my starting value is 123450000000 and I compute: 123450000000 - 123450000000 * 1E8 + 123450000000 * 1E8 we get: 123400000000 The result is rounded to even. Now start with 123550000000 and compute: 123550000000 - 123550000000 * 1E8 + 123550000000 * 1E8 we get 12360000000, and we see that the result is rounded to even, as it should be because the discarded digits were 50000000..., that is, a 5 followed by only zeroes. Try reversing the order of the addition and subtraction in the last example: 123550000000 + 123550000000 * 1E8 - 123550000000 * 1E8 we still get: 123600000000 Now try this with a TI86 calculator: 123550000000 - 123550000000 * 1E10 + 123550000000 * 1E10 (use 1E10 as the multiplier because the TI is a "14" digit machine) we get 123500000000 (only 12 digits are shown) But if we try: 123550000000 + 123550000000 * 1E10 - 123550000000 * 1E10 we get 123600000000 on the TI86. So Dekker's method fails on the TI.

 Re: Rounding conventionsMessage #38 Posted by Wayne Best on 4 July 2009, 9:17 p.m.,in response to message #37 by Rodger Rosenbaum Thanks Rodger Yes, you're right, it's all about the rounding on the display. As a user, that's all I see. I take it for granted (perhaps naively) that the precision of the calculations are fine and I cannot recall a single real life example where I've needed all the precision of a modern calculator. So I guess most of us set the display for the number of significant figures we need (for me that's rarely more than 4 decimal places). Once I set the display, I expect (would like) that when I average data that the display shows the "properly" rounded average but this is not so. Sure, I can set the calculator to display all the digits and do the rounding myself but this introduces human errors and, to be honest, I expect the calculator to do this for me and not present me with the "wrong" number. Ok, it's not really wrong but it's not really right either. Regards, Wayne

 Re: Rounding conventionsMessage #39 Posted by Rodger Rosenbaum on 5 July 2009, 4:39 a.m.,in response to message #38 by Wayne Best You asked in the first post "... is there any way around the problem (other than writing a program)?". If you don't want to look at all 12 digits and do the rounding-to-even yourself, then the answer is no. Presumably, you've been doing some arithmetic on a data set. This probably was the result of a program. It doesn't take much extra programming to use Dekker's method to round the final result(s) to K digits. Then with the display set to show K digits, you'll see the result rounded-to-even.

Go back to the main exhibit hall