The Museum of HP Calculators

HP Forum Archive 21

[ Return to Index | Top of Index ]

Good (simple?) calculation for benchmarking?
Message #1 Posted by Jedidiah Smith on 26 Feb 2013, 2:56 a.m.

Hi, this is probably way elementary for all of you, but I was wondering if anyone has a good suggestion for a *simple* yet good calculation I could type in to HP 48 / 49 series calculators to do a general benchmark comparison? I have 3 calcs at the moment that I would like to do this with, just for fun to see the differences in speed: HP 48GX, HP 49G, & HP 49G+ (w/50G ROM).

Thanks for your time!

      
Re: Good (simple?) calculation for benchmarking?
Message #2 Posted by Frank Boehm (Germany) on 26 Feb 2013, 10:42 a.m.,
in response to message #1 by Jedidiah Smith

I doubt that this will work as a benchmark. All of the mentioned calculators are probably too fast to even press start/stop on your stop watch.
So you would need a loop of calculations. Then you would have to factor different things into it: int performance, float performance, how well are functions implemented, gfx performance and so on.
No wonder a PC benchmark suite is quite complex :)
Easiest thing would be a +1 loop, like 10000 times. That should be hand-stoppable, this is more or less raw CPU speed dependent.

            
Re: Good (simple?) calculation for benchmarking?
Message #3 Posted by Marcel Samek on 26 Feb 2013, 11:01 a.m.,
in response to message #2 by Frank Boehm (Germany)

And, remember to benchmark an empty loop, so you understand the overhead of the loop, and subract that from benchmarks of other operations.

                  
Link to Article on benchmark
Message #4 Posted by gene wright on 26 Feb 2013, 12:09 p.m.,
in response to message #3 by Marcel Samek

Benchmark for lots of calculators

                  
Re: Good (simple?) calculation for benchmarking?
Message #5 Posted by Jedidiah Smith on 26 Feb 2013, 12:10 p.m.,
in response to message #3 by Marcel Samek

I see it is not so simple. :-)

So would a better way to do it be to write a small program to say, find pi to the 1000th digit or something like that, and then just time each one?

I've (obviously) never done any programming before, so I just thought it might be a fun exercise, but sounds like it's more complex than I thought!

**EDIT** thanks for the link above, it came in while I was typing. :-) I will check that out and see if I can reproduce it on these calcs.

Edited: 26 Feb 2013, 12:12 p.m.

      
Re: Good (simple?) calculation for benchmarking?
Message #6 Posted by Egan Ford on 26 Feb 2013, 2:08 p.m.,
in response to message #1 by Jedidiah Smith

Use this. It does useful work and solves a real problem and all the code for everything you want to test is here:

http://www.hpmuseum.org/cgi-sys/cgiwrap/hpmuseum/articles.cgi?read=700

Another benchmark (used for super computers (http://top500.org)) would be to solve a dense system of liner equations. Also a real problem. I've use this compare my 71b, 48GX, and 50g. IIRC, the random number generator in all 3 are the same and thus generated the exact same problem to solve. Memory limits the size of the matrix.

As you mention above Pi to 1000 digits. I like to use that to compare some calculators, but mostly 8-bit processors.

For all three you need to use the same algorithm if you want to reduce the number of variables. E.g.:

  1. See [url:http://sense.net/~egan/hpgcc/#Example: π Shootout] for an example of how different Pi computations on the same calculator vary widely with speed.
  2. The Top500 rules are very clear that the algorithm used must conform to the standard operation count for LU factorization with partial pivoting. In particular, the operation count for the algorithm must be (2/3)n3 + O(n2) floating point operations.
  3. For all my 8-bit Pi benchmarks I always use assembly (compilers and interpreters introduce too much variable) and I use the same Machin Arctan O(n2) algorithm.
      
Re: Good (simple?) calculation for benchmarking?
Message #7 Posted by Egan Ford on 26 Feb 2013, 2:19 p.m.,
in response to message #1 by Jedidiah Smith

Quote:
*simple* yet good
Go with the largest dense matrix and vector you can fit into memory/2 (48/49/50 matrix operations use a nondestructive copy doubling the amount of RAM required), then solve it. Use random numbers between -0.5 and 0.5. You can measure the time in seconds or use the internal 48/49/50 timer (ticks) to get higher resolution times.
      
Re: Good (simple?) calculation for benchmarking?
Message #8 Posted by Mike Morrow on 26 Feb 2013, 3:05 p.m.,
in response to message #1 by Jedidiah Smith

An iterative calculation of some sort is the only usable type of benchmark, preferably one that uses the results of the previous iteration in the next to produce a final result that is useful to gauge the propagation of truncation and accuracy errors. For scientific applications that likely make up a large portion of the applications for our HPs, it is logical to include various standard mathematical functions in the loop, in preference to some other benchmarks that spend most of the execution time moving data to and fro, and operating on them with basic arithmetic. However, that type of benchmark would provide better results for applications that use those capabilities, such as matrix manipulation intensive applications.

For several decades one of the benchmarks often utilized has been
the Savage Benchmark.  It was useful in the early personal computing
era when desktops were much slower than many handheld devices are
today.

One common version of this benchmark in RPL follows that is based on the exact series of operations that were easily programmed using the typical syntax of BASIC on early personal computers for the statement 10 A=TAN(ATN(EXP(LOG(SQR(A*A)))))+1 . Note: "sqrt" below is the square root symbol. << RAD 1. 1. 2499. START SQ sqrt LN EXP ATAN TAN 1. + NEXT >> The HP 50G (version 2.15) completes the above in about 65 seconds with this result: 2499.99948647 .

I prefer a slight modification that produces more meaningful results that takes advantage of the versatility of RPL syntax by: (1) Arranging for exactly 2500 iterations (first version does 2499), (2) Taking the square root, followed by the square (first version reverses this order and thus operates only on integer values), (3) Add steps to automatically indicate the execution time in seconds, after iterations are complete, (4) Add step to return calculator to DEG mode, and (5) Add 800 Hz two-second beep at end. << RAD TICKS 0. 1. 2500. START 1. + sqrt SQ LN EXP ATAN TAN NEXT SWAP TICKS SWAP - B->R 8192. / DEG 800. 2. BEEP >> The HP 50G (version 2.15) completes the above in about 65 seconds with this result: 2499.99946106 (and 64.8814697266 seconds for this run).

Note: For the 49G+ and 50G, one will not want the calculations done using the binary integer types that exist on those machines. That's the reason for the decimal point appearing with each value in the programs above. Remove those decimal points, and execution times increase very significantly.

Very simple benchmarks, such as a loop that just increments a value by 1 each iteration for a specified interval, produce very inaccurate estimates of the practical or typical speeds of two machines relative to each other.

For example, Free42 completes the Savage Benchmark in about 1.5 seconds on my Android smartphone, 400 times faster than a real HP 42S. But just counting the iterations of a X=X+1 type of loop for 60 seconds on these two machines indicates that Free42 on my Android is a whopping 7800 times faster than a real HP 42S. That's a factor of almost 20 in the comparative results caused simply by the choice of benchmark method!

There is no "universal" benchmark that is meaningful for every application.

Edited: 26 Feb 2013, 4:40 p.m.

            
Re: Good (simple?) calculation for benchmarking?
Message #9 Posted by Jedidiah Smith on 27 Feb 2013, 1:47 a.m.,
in response to message #8 by Mike Morrow

Oh man, thanks guys. We are already so far over my head with that last post. :-) It will take me a while to decipher that and figure out how to do it. I get it that the simple calculation won't work (like I first thought), so I will try to figure out how to do the more complex one as I have some time.

                  
Re: Good (simple?) calculation for benchmarking?
Message #10 Posted by Mike Morrow on 27 Feb 2013, 2:22 a.m.,
in response to message #9 by Jedidiah Smith

To cut to the essential, the Savage Benchmark may be quickly entered into any of your three calculators by storing the program object

<< RAD TICKS 0. 1. 2500. START 1. + sqrt SQ LN EXP ATAN TAN NEXT SWAP TICKS SWAP - B->R 8192. / DEG 800. 2. BEEP >>

("sqrt" stands for the "square root symbol")

and storing that as, for example, SBM.

Executing SBM will result in a display of the results of execution and the execution time. A result of 2500.00000000 would be perfect, but all of your machines will produce the same result (2499.99946106), at an execution time of about 65 seconds (for the 49G+) and greater (for the other two machines).

                        
Re: Good (simple?) calculation for benchmarking?
Message #11 Posted by Ronald Williams on 27 Feb 2013, 5:13 a.m.,
in response to message #10 by Mike Morrow

Does the TICKS command produce more accurate results then the TIME command? If not then it is simpler to use the TIME command (no conversions required).

                              
Re: Good (simple?) calculation for benchmarking?
Message #12 Posted by Mike Morrow on 27 Feb 2013, 12:31 p.m.,
in response to message #11 by Ronald Williams

All of the elapsed time display overhead takes place after the looping has been completed, after the begin and end TICKS values have been stored. There's no substantive advantage to using TIME, HMS-, and HMS-> compared to the TICKS process, and one situation (albeit unlikely) where TIME will fail.

<< RAD TICKS 0. 1. 2500. START 1. + sqrt SQ LN EXP ATAN TAN NEXT SWAP TICKS SWAP - B->R 8192. / DEG 800. 2. BEEP >>

versus

<< RAD TIME 0. 1. 2500. START 1. + sqrt SQ LN EXP ATAN TAN NEXT SWAP TIME SWAP HMS- HMS-> 3600. * DEG 800. 2. BEEP >>

The TICKS method does not develop a nonsense result such as the TIME method does if the loop begins on one side of 00.00 (midnight) and ends on the other side. :-)

                        
Re: Good (simple?) calculation for benchmarking?
Message #13 Posted by Gerson W. Barbosa on 27 Feb 2013, 11:41 a.m.,
in response to message #10 by Mike Morrow

Quote:
all of your machines will produce the same result (2499.99946106),

which happens to be exactly the same result produced by the HP-71B, where they inherited lots of math algorithms and Saturn code from:

>RADIANS
>1 T=TIME@A=0@FORI=1TO2500@A=TAN(ATAN(EXP(LN(SQR(A+1)*SQR(A+1)))))@NEXTI@DISPA;TIME-T
>RUN
 2499.99946106  387.53
                              
Re: Good (simple?) calculation for benchmarking?
Message #14 Posted by Mike Morrow on 27 Feb 2013, 12:20 p.m.,
in response to message #13 by Gerson W. Barbosa

Yes, that result is produced for all machines that are Saturn or Saturn-like...for example: The HP 42S, HP 32SII, HP 20S (lowly old algebraic Pioneer), and the recent non-Saturn HP 30b.

The HP 30b is the speed champion among all HPs that I've tried. It completes the 2500-iteration Savage Benchmark in 6.5 seconds. That's ten times faster than the HP 50G, and eight times faster than the HP 15C LE (which is not Saturn-like and produces a result with considerably less precision and accuracy).

I have yet to program my HP39GII for this. Unfortunately it has no real time clock implementation to make timing easier.

                                    
Re: Good (simple?) calculation for benchmarking?
Message #15 Posted by Gerson W. Barbosa on 27 Feb 2013, 9:37 p.m.,
in response to message #14 by Mike Morrow

Quote:
The HP 30b is the speed champion among all HPs that I've tried. It completes the 2500-iteration Savage Benchmark in 6.5 seconds.

It takes quite longer on my wp34s, but the results are outstanding:

001 LBL A
002 TICKS
003 # 250
004 SDL 001
005 0
006 INC X
007 SQRT
008 x^2
009 LN
010 e^X
011 ATAN
012 TAN
013 DSE Y
014 BACK 008
015 TICKS
016 RCL- T
017 SDR 001
018 x<> Y
019 END

DBLON XEQ A --> 2500. (2.500 000 000 000 000 000 000 000 000 053 091 E 0003) x<>y --> 319.3 (5m 19.3s)

DBLOFF XEQ A --> 2500.00000001 (2.500 000 000 006 559 E 003) x<>y --> 287.1 (4m 47.1s)

-----------------------------------------

P.S.: With the latest firmware, 3.2r3371:

V3.2 is slightly faster:

    306.0              (5m 06.0s)

275.9 (4m 35.9s)

Edited: 3 Mar 2013, 1:14 a.m. after one or more responses were posted

                                          
Re: Good (simple?) calculation for benchmarking?
Message #16 Posted by Paul Dale on 27 Feb 2013, 10:25 p.m.,
in response to message #15 by Gerson W. Barbosa

The logarithm will be the killer here I suspect.

- Pauli

                                                
Re: Good (simple?) calculation for benchmarking?
Message #17 Posted by Gerson W. Barbosa on 28 Feb 2013, 9:37 a.m.,
in response to message #16 by Paul Dale

Or TAN, according to Table 3 in this paper. The next ones in order would be ATAN, LN and EXP. The table applies to minimax polynomials, but might apply to Taylor series as well.

I was expecting faster results, but I had forgotten about this post by W. Bruce Maguire II last year:

WP-34s: Speed benchmark

Gerson.

                                                      
Re: Good (simple?) calculation for benchmarking?
Message #18 Posted by Paul Dale on 28 Feb 2013, 4:04 p.m.,
in response to message #17 by Gerson W. Barbosa

I'm almost certain it is the logarithm. The 34S uses taylor expansion for the trig functions and this is quite fast. For exponential and logarithm it uses other series and I know the logarithm is very slow.

- Pauli

                                                            
Re: Good (simple?) calculation for benchmarking?
Message #19 Posted by Gerson W. Barbosa on 28 Feb 2013, 4:48 p.m.,
in response to message #18 by Paul Dale

You are right! LN and EXP are indeed slower than TAN and ATAN:

...
009 ATAN
010 TAN             
011 ATAN
012 TAN             
...                -->  215.7 s  (2500.000000006559) 

... 009 LN 010 e^x 011 LN 012 e^X ... --> 358.2 s (2499.999999999975)

Gerson.
                                          
Re: Good (simple?) calculation for benchmarking?
Message #20 Posted by Mike Morrow on 28 Feb 2013, 10:39 a.m.,
in response to message #15 by Gerson W. Barbosa

Very impressive!!!

                                    
Re: Good (simple?) calculation for benchmarking?
Message #21 Posted by Tim Wessman on 28 Feb 2013, 11:18 a.m.,
in response to message #14 by Mike Morrow

6.5 seconds? Wow, how slow... ;-)

EXPORT SAVAGE()
BEGIN
A:=0
  FOR I FROM 1 TO 2500 DO
   A:=TAN(ATAN(EXP(LN(sqrt(A+1))))); 
  END:
END;

Run on the home screen with SAVAGE().

Res: 2499.99946106

Run on the home screen with Time(SAVAGE())

Res: 3.6_s

Use the Time(<argument>) function to time execution speed. There is an internal clock. Internally, it keeps time as ms since boot. If i remember correctly, it will roll over at about 47 days of running. No issues with midnight though! :-)

TW

Edited: 28 Feb 2013, 11:22 a.m.

                                          
Re: Good (simple?) calculation for benchmarking?
Message #22 Posted by Mike Morrow on 28 Feb 2013, 12:17 p.m.,
in response to message #21 by Tim Wessman

Thanks much, Tim. I haven't had as much time as I'd like to explore the capabilities of the HP 39GII, so I find your information very interesting. I had little doubt that the remarkable HP 39GII would be faster than the already impressive HP 30b. Now I know that it's twice as fast, for this benchmark. And 20 times faster than the HP 50G. The HP 39GII is the new HP Savage Benchmark speed champion.

The HP 30b and HP 39GII show all the good things that come from not having an emulation layer that the calculator hardware must muddle through.

                              
Re: Good (simple?) calculation for benchmarking?
Message #23 Posted by Harald on 28 Feb 2013, 6:22 p.m.,
in response to message #13 by Gerson W. Barbosa

Got a different result at first, but then realised you are doing this in radians. No I get the same result and a very similar time (387.87).

                                    
Re: Good (simple?) calculation for benchmarking?
Message #24 Posted by Gerson W. Barbosa on 28 Feb 2013, 8:05 p.m.,
in response to message #23 by Harald

Quote:
I get the same result and a very similar time (387.87).

This means the CPU clock frequency of your HP-71B is 621200 Hz, or very close to (per HP-71B CPU frequency?, a recent thread).
Notice SQR(A+1) is computed twice every iteration. The following saves about 22.5 seconds:

>RADIANS
>1 T=TIME @ A=0 @ FOR I=1 TO 2500 @ B=SQR(A+1) @ A=TAN(ATAN(EXP(LN(B*B)))) @ NEXT I
>2 DISP A;TIME-T
>RUN
 2499.99946106  365.09
                                          
Re: Good (simple?) calculation for benchmarking?
Message #25 Posted by Harald on 1 Mar 2013, 7:26 a.m.,
in response to message #24 by Gerson W. Barbosa

Quote:
Notice SQR(A+1) is computed twice every iteration. The following saves about 22.5 seconds:

Yes, I did notice that. I tried to substitute "SQRT(A+1) * SQRT(A+1)" with "SQRT(A+1)^2". To my surprise this was even slower. I appreciate that squaring takes longer than multiplying. But I didn't think it would take longer than calculating the extra square root.

                                                
Re: Good (simple?) calculation for benchmarking?
Message #26 Posted by Gerson W. Barbosa on 1 Mar 2013, 1:14 p.m.,
in response to message #25 by Harald

A user-defined function takes longer (about 460 seconds). Using a one-element vector instead of a variable and the DOT function (available in the MATH ROM) might work, but this would probably be even slower.

                                                      
Re: Good (simple?) calculation for benchmarking?
Message #27 Posted by Harald on 1 Mar 2013, 3:20 p.m.,
in response to message #26 by Gerson W. Barbosa

Yes, 460 seconds is roughly what I got. What do you mean by user-defined? There is nothing like x (square function) on the 71B, is there?

                                                            
Re: Good (simple?) calculation for benchmarking?
Message #28 Posted by Gerson W. Barbosa on 1 Mar 2013, 3:55 p.m.,
in response to message #27 by Harald

It appears SQ was not a standard BASIC function. My old 8-bit computer didn't have one either, as far as I can remember.

>10 DEF FNS(X)=X*X
>20 DISP FNS(5)
>RUN
 25
                                                                  
Re: Good (simple?) calculation for benchmarking?
Message #29 Posted by Harald on 1 Mar 2013, 5:13 p.m.,
in response to message #28 by Gerson W. Barbosa

I see, thanks!

I really need to play with the 71B a lot more. It is a fantastic machine and unfortunately I still know so little about it.

Cheers, Harald

Edited: 1 Mar 2013, 5:14 p.m.


[ Return to Index | Top of Index ]

Go back to the main exhibit hall