Yet another benchmark port on the wiki: Savage

The Museum of HP Calculators

HP Forum Archive 21

Yet another benchmark port on the wiki: Savage
Message #1 Posted by Pier Aiello on 13 Sept 2013, 6:43 a.m.

http://www.wiki4hp.com/doku.php?id=benchmarks:savage
I'm gathering the results from MoHPC and comp.sys.48 (from the entire web it is hard) thanks to google search. If anyone wants to help, he is welcome! The wiki is of all of us.
:)

Re: Yet another benchmark port on the wiki: Savage
Message #2 Posted by Paul Dale on 13 Sept 2013, 7:06 a.m.,
in response to message #1 by Pier Aiello

The 34S gives:

single precision 2500.000000001578 double precision 2499.999999999999999999999999999661
- Pauli

Re: Yet another benchmark port on the wiki: Savage
Message #3 Posted by Pier Aiello on 13 Sept 2013, 7:15 a.m.,
in response to message #2 by Paul Dale

Nooooooooooooooooooooo i want to know even the time! :(
Ok, just use the accuracy :(

Re: Yet another benchmark port on the wiki: Savage
Message #4 Posted by Paul Dale on 13 Sept 2013, 7:31 a.m.,
in response to message #3 by Pier Aiello

I obtained these results via the console mode emulator, so the timings aren't relevant. On real hardware, timing will vary depending on firmware version -- I went through three or four different implementations of LN e.g. The numeric results will be identical.
I believe that the 34S single precision results are correctly rounded. At least I think I checked this benchmark ages back.
As an aside, the 34S can be used to verify correct rounding for other calculators. Set double precision and after each operation, force a rounding step to the appropriate precision.
- Pauli

Re: Yet another benchmark port on the wiki: Savage
Message #5 Posted by Pier Aiello on 13 Sept 2013, 7:37 a.m.,
in response to message #4 by Paul Dale

Ok but i added another 34s result (check the page) that shows a different result. So other information (code, fw version, times, and so on) can help a lot.
Anyway, i'll add them only as accuracy :)

Re: Yet another benchmark port on the wiki: Savage
Message #6 Posted by Paul Dale on 13 Sept 2013, 8:03 a.m.,
in response to message #5 by Pier Aiello

I think your program is wrong :-) It is incrementing before the transcendentals rather than after. This means one more time through the loop I think -- the iteration that gives 2500 as its answer doesn't apply the six functions to the almost 2500 result.
The program in note 1 & 2 also has the square root and x² operations reversed, the x² should come first. This doesn't make a difference on the 34S but might on other devices.

LBL A TICKS 2 4 9 9 # 001 LBL 00 x[^2] [sqrt] LN e[^x] ATAN TAN INC X DSE Y GTO 00 TICKS RCL- T SDR 001 x[<->] Y END
- Pauli

Re: Yet another benchmark port on the wiki: Savage
Message #7 Posted by Pier Aiello on 13 Sept 2013, 8:12 a.m.,
in response to message #6 by Paul Dale

many thanks! So i get rid of that ;) .

Re: Yet another benchmark port on the wiki: Savage
Message #8 Posted by Tom Grydeland on 17 Sept 2013, 3:55 a.m.,
in response to message #6 by Paul Dale

Reusing Paul Dale's program for the HP-15CLE:

43,22,11 @ LBL A 2 4 9 9 44 1 @ STO 1 1 43,22, 0 @ LBL 0 43 11 @ x^2 11 @ SQRT 43 12 @ LN 12 @ e^x 43 25 @ ATAN 25 @ TAN 1 40 @ + 42, 5, 1 @ DSE 1 22 0 @ GTO 0 43 32 @ RTN
I get 2499.970322, the same result reported for the HP-41CX in Wlodek's article, according to message #3 in this thread. Note that this is _not_ the same result reported for HP-15CLE in the benchmark page on the wiki.
My runtime is 51.5 seconds, consistent over three runs.
--T

Re: Yet another benchmark port on the wiki: Savage
Message #9 Posted by Pier Aiello on 17 Sept 2013, 7:40 a.m.,
in response to message #8 by Tom Grydeland

As you probably saw, i reported it from the source cited in the wiki page.
I guess that someone has mixed the order of operations, like first x^2 then sqrt, or first sqrt then x^2 and so on. Anyway thanks i'll report your code asap.

Re: Yet another benchmark port on the wiki: Savage
Message #10 Posted by Pier Aiello on 13 Sept 2013, 8:10 a.m.,
in response to message #4 by Paul Dale

Oh i forgot that: thanks!

Re: Yet another benchmark port on the wiki: Savage
Message #11 Posted by Pier Aiello on 13 Sept 2013, 10:17 a.m.,
in response to message #4 by Paul Dale

I found two interesting remarks in my searches:
http://www.hpmuseum.org/cgi-sys/cgiwrap/hpmuseum/archv020.cgi?read=193816#194289
http://www.technicalc.org/tiplist/en/files/pdf/tips/tip6_50.pdf (page 5)
So, since accuracy change a lot given a computation, maybe it's better to use the savage benchmark as benchmark to see (in general) how fast some non trivial but common operations are executed (instead of nqueens, addloop and similar).
Edited: 13 Sept 2013, 10:31 a.m.

Re: Yet another benchmark port on the wiki: Savage
Message #12 Posted by Tom Grydeland on 16 Sept 2013, 4:34 a.m.,
in response to message #3 by Pier Aiello

Quote:
i want to know even the time!
On my WP-34S, rev. 3450, with crystal and fresh batteries:
Single prec: 283.2 sec, result-2500: 6.49e-9
Double prec: 312.0 sec, result-2500: 5.248e-26
(The code by P. Dale in message #6, below)

Re: Yet another benchmark port on the wiki: Savage
Message #13 Posted by bhtooefr on 13 Sept 2013, 6:17 p.m.,
in response to message #1 by Pier Aiello

Running it on my MK-61 (and using my 41CX to time it) using this code:

00 1 01 STO 0 02 STO 1 03 2499 07 STO 2 08 RCL 2 09 RCL 1 10 - 11 x!=0 29 13 RCL 0 14 x^2 15 SQRT 16 LN 17 e^x 18 ATAN 19 TAN 20 1 21 + 22 STO 0 23 RCL 1 24 1 25 + 26 STO 1 27 GTO 08 29 R/S
We'll see how bad this is. Looks like it'll be about a 6 hour run, based on the loop counter that I can see being flashed at me.

Re: Yet another benchmark port on the wiki: Savage
Message #14 Posted by bhtooefr on 14 Sept 2013, 7:40 a.m.,
in response to message #13 by bhtooefr

So, I fell asleep before the run completed. But, every extrapolation I made, including one at 5h15m completed, had it in the 6h15m-6h18m range, so we'll go with that as being the time.
And, I realized that I didn't actually RCL 0 at the end.
But, RCL 0 returns 2269.0879. That's... wildly off. (I don't doubt that it's really how much error the MK-61 has, this really is an 8 digit calculator, though.)

Re: Yet another benchmark port on the wiki: Savage
Message #15 Posted by Pier Aiello on 14 Sept 2013, 9:43 a.m.,
in response to message #14 by bhtooefr

i'll report it :)
Done :)
Edited: 14 Sept 2013, 9:53 a.m.

Re: Yet another benchmark port on the wiki: Savage
Message #16 Posted by Tom Grydeland on 17 Sept 2013, 6:14 p.m.,
in response to message #1 by Pier Aiello

Quote:
If anyone wants to help, he is welcome!
Adapting Paul Dale's program for the HP-30b (RPN mode). This is the version which starts at 1, and runs the transcendental functions 2499 times, adding 1 after each iteration.
Checksum: 27.154
00 - SH* @ assign to shift-multiply button 01 - 2 02 - 4 03 - 9 04 - 9 05 - STO 0 06 - 1 07 - LBL 00 08 - X^2 09 - [sqrt] 10 - Ln 11 - e^X 12 - Math 13 - Input 14 - Down 15 - Down 16 - Down 17 - Input @ ATAN 18 - Tan 19 - 1 20 - + 21 - DSE 0 22 - Gto 00
Result: 2499.99948647 (2500 - result = 5.135300e-4)
Timing (using external stopwatch): 7.5 seconds, consistent across three runs.
(wow, the 30b is FAST, though precision is not up to WP-34S standards)
--Tom

Re: Yet another benchmark port on the wiki: Savage
Message #17 Posted by Paul Dale on 17 Sept 2013, 6:19 p.m.,
in response to message #16 by Tom Grydeland

Quote:
(wow, the 30b is FAST, though precision is not up to WP-34S standards)
The 34S is carrying almost three times the number of digits internally, has a few more functions and wasn't coded for performance (rather, for accuracy then space).
- Pauli

Re: Yet another benchmark port on the wiki: Savage
Message #18 Posted by Tom Grydeland on 18 Sept 2013, 2:40 a.m.,
in response to message #17 by Paul Dale

Quote:
The 34S is carrying almost three times the number of digits internally, has a few more functions and wasn't coded for performance (rather, for accuracy then space).
I know, I know. I have been reading the discussions here. Most days, I'll choose the accuracy over the blazing speed, and for real number-crunching I have a desktop computer.

Quote:
The 34S [...] has a few more functions
That has got to be candidate for understatement of the year! :-)
--T

Re: Yet another benchmark port on the wiki: Savage
Message #19 Posted by Tom Grydeland on 18 Sept 2013, 3:57 a.m.,
in response to message #1 by Pier Aiello

Redoing the HP-32S, HP-42S and RPL version to match Dale's version exactly (i.e. loop up to 2499, square before square root, increment after transcendental functions) I also get identical numerical results between different saturn-based (or emulating) models:
Result: 2499.99948647; 2500-result = 5.13530e-4
Timings:
HP-42S (external stopwatch): 9 min, 56 sec, or 596 sec
HP-32S (external stopwatch): 6 min, 37 sec or 397 sec (best of 2)
HP-48G: 117.475 secs
HP-50g: 65.0018 secs
I'm somewhat surprised the 32S is that much faster than the 42S.
Listings:
RPL (HP-50g, HP-48G):

\<< RAD TICKS 1. 1. 2499. START SQ [sqrt] LN EXP ATAN TAN 1. + NEXT SWAP TICKS SWAP - B->R 8192. / 800. 2. BEEP \>>
HP-32S:

LBL S @ 015.5 bytes; CHKSUM=C573 RAD 2499 STO U 1 LBL U @ 016.5 bytes; CHKSUM=EC87 x^2 SQRT LN e^x ATAN TAN 1 + DSE U GTO U
HP-42S:

{32-Byte Prgm} LBL "SAVAGE" RAD 2499 1 LBL 00 X^2 SQRT LN EXP ATAN TAN 1 + DSE ST Y GTO 00 END
--T

Re: Yet another benchmark port on the wiki: Savage
Message #20 Posted by Pier Aiello on 18 Sept 2013, 4:38 a.m.,
in response to message #19 by Tom Grydeland

Thanks, i'll report them asap (both #16 and #19)

Re: Yet another benchmark port on the wiki: Savage
Message #21 Posted by Pier Aiello on 18 Sept 2013, 11:57 a.m.,
in response to message #1 by Pier Aiello

An interesting update: the nspire CX seems to have a lot of problems with "standard" trig/exp functions:
http://tiplanet.org/forum/viewtopic.php?p=148292#p148292

Savage benchmark: Results for HP-33s and HP-41CX
Message #22 Posted by Tom Grydeland on 19 Sept 2013, 7:27 a.m.,
in response to message #1 by Pier Aiello

The benchmark again in the format by Dale: 2499 times through the loop, increment after.
Timing:
HP-33s (external stopwatch): 7 min 52 sec, or 472 sec
HP-41CX (internal, cross-checked with external stopwatch): 48 min, 52.8 sec or 2932.8 sec (consistent within 2 secs on 2 runs)
Results:
HP-33s: 2499.99942403, 2500-result = 5.7597e-4
HP-41CX: 2499.970322, 2500-result = 2.9678e-2
Listing HP-33S is identical to the '32S reported previously
Listing HP-41CX

LBL 'SAVAGE TIME RAD 2499 1 LBL 00 X^2 SQRT LN E^X ATAN TAN 1 + DSE Y GTO 00 TIME R^ HMS- BEEP
--T

Savage benchmark: extended
Message #23 Posted by Pier Aiello on 19 Sept 2013, 9:09 a.m.,
in response to message #22 by Tom Grydeland

added, thanks!
Two notes:
- I don't report your result if they match with the score of the same calculator already present in the list (even if i could report it as confirmation :/ ). If you don't like it, feel free to modify the page, it's the wiki of hp calculators users!
- Since you seems enjoyed by this benchmark, could you do also the savage extended? (careful, it takes a lot of time)
http://www.wiki4hp.com/doku.php?id=benchmarks:savage-extended

Edited: 19 Sept 2013, 9:11 a.m.

Re: Savage benchmark: extended
Message #24 Posted by Tom Grydeland on 23 Sept 2013, 3:25 p.m.,
in response to message #23 by Pier Aiello

Quote:
I don't report your result if they match with the score of the same calculator already present in the list (even if i could report it as confirmation :/ )
Fine. Since there are many almost-equal results from almost-equal versions of the benchmark I thought it useful to have them all for exactly equivalent programs.
I've just got a Prime, and it is too fast for me to time externally: less than a second on the 2499-iteration Savage. Result: 2499.99948647.
code:
EXPORT SAVAGE() BEGIN A:=1; FOR I FROM 1 TO 2499 DO A:=TAN(ATAN(e^(LN(\|(A*A)))))+1 END;
END;


Re: Savage benchmark: extended
Message #25 Posted by Pier Aiello on 25 Sept 2013, 4:59 a.m.,
in response to message #24 by Tom Grydeland

Added, thanks. About the speed of prime, you can do the savage extended, that should be harder.

Re: Savage benchmark: extended
Message #26 Posted by Tom Grydeland on 26 Sept 2013, 6:09 a.m.,
in response to message #25 by Pier Aiello

Quote:
About the speed of prime, you can do the savage extended, that should be harder
Sorry, but I don't like the savage extended.
A benchmark, IMO, should give some information in addition to the timing. Savage does that (some sort of measure of numerical accuracy for transcendental functions). The extended savage, since it requires you starting over and repeating the N-1 iterations from the previous savage every time through the outer loop, means arriving at the answer in a silly and extremely roundabout way.
--T

Re: Savage benchmark: extended
Message #27 Posted by Pier Aiello on 26 Sept 2013, 12:30 p.m.,
in response to message #26 by Tom Grydeland

Quote:
A benchmark, IMO, should give some information in addition to the timing. Savage does that (some sort of measure of numerical accuracy for transcendental functions)
I agree with you! indeed the savage-extended wants to return more information about the accuracy of the calculator. Just read the part about "the idea" on the wiki page or the analysis done here http://www.technicalc.org/tiplist/en/files/pdf/tips/tip6_50.pdf. To be clear we should do a batch of accuracy tests over an interval of values (like the work done in the article above), to plot how the accuracy changes for each value.
Since it's a quick benchmark, we can't do comparisons between plots, so at least we measure the sum of absolute errors for each value.
I don't get why the Savage extended is silly in terms of information about accuracy, if it is silly, the Savage standard is definitely useless in comparison.

Re: Savage benchmark: extended
Message #28 Posted by Tom Grydeland on 27 Sept 2013, 5:57 p.m.,
in response to message #27 by Pier Aiello

Quote:
I don't get why the Savage extended is silly in terms of information about accuracy
That's not why it's silly.
Every iteration in the outer loop of the extended savage benchmark could be computed by a single iteration from the last result, but instead it starts from one every time.

Re: Savage benchmark: extended
Message #29 Posted by Pier Aiello on 28 Sept 2013, 3:31 a.m.,
in response to message #28 by Tom Grydeland

You are right! I didn't think about that! (i just thought "nest the savage benchmark in a loop that tests the accuracy for several values". That is inefficient (1)). So it is quite easy to fix, just delete the outer loop and move the sum of absolute errors to add the error after each iteration.

input: n: upper bound value to test the savage benchmark ---
globalSum = 0 sum = 1 for k=1 to (n-1) { //note the "n-1" sum = sum + tan ( atan ( exp ( log ( sqrt ( sum*sum) ) ) ) ) //CAREFUL here! //for the original savage benchmark many have done sqrt first and square then, //obtaining different results. //The algorithm must be the same in terms of main operations: //Square first //then square root //then log //then exp //then atan //then tan globalSum = globalSum + abs((k+1)-sum) } print globalSum //the global sum obtained
(1) I have even done personal tests and plots to check how the accuracy changes with a similar, inefficient, code! Poor me, i "wasted" a lot of energy.
Edited: 28 Sept 2013, 3:31 a.m.

Re: Savage benchmark: extended
Message #30 Posted by Paul Dale on 25 Sept 2013, 5:08 a.m.,
in response to message #24 by Tom Grydeland

Fun the benchmark a couple of dozen times and divide:

EXPORT SAVAGE() BEGIN A:=1; FOR J FROM 1 TO 50 DO FOR I FROM 1 TO 2499 DO A:=TAN(ATAN(e^(LN(\|(A*A)))))+1 END;
- Pauli

Re: Savage benchmark: extended
Message #31 Posted by Tom Grydeland on 26 Sept 2013, 5:54 a.m.,
in response to message #30 by Paul Dale

Quote:
Fun the benchmark a couple of dozen times and divide:
Okay, about 24 seconds for 50 times, should mean 0.48 seconds per 2499-iteration Savage.
There is a TICKS function in the Prime, but I haven't been able to get useful results from it in programs (it works fine in the Home screen).
--T

Re: Savage benchmark: extended
Message #32 Posted by Tim Wessman on 27 Sept 2013, 6:22 p.m.,
in response to message #31 by Tom Grydeland

TIME(SAVAGE())
There you go.
TW

Re: Savage benchmark: extended
Message #33 Posted by Joe Horn on 27 Sept 2013, 9:37 p.m.,
in response to message #32 by Tim Wessman

Quote:
TIME(SAVAGE())
TIME(whatever) returns the elapsed time with the "_s" unit attached. If you don't want the unit attached to the result, type "time" in lowercase:
time(SAVAGE())
This returns the number of elapsed seconds as a real, not a unit.
-Joe-

Re: Savage benchmark: extended
Message #34 Posted by Tom Grydeland on 28 Sept 2013, 3:42 a.m.,
in response to message #32 by Tim Wessman

Indeed. Thanks.
I'm getting timings between 0.498 and 0.513 seconds for SAVAGE.
--T

Re: Savage benchmark: Results for Improved Version
Message #35 Posted by Mike Morrow on 25 Sept 2013, 7:15 p.m.,
in response to message #22 by Tom Grydeland

Quote:

LBL 'SAVAGE TIME RAD 2499 1 LBL 00 X^2 SQRT LN E^X ATAN TAN 1 + DSE Y GTO 00 TIME R^ HMS- BEEP

I really really dislike that version of the Savage Benchmark:
1. It only iterates 2499 times to produce the near 2500 result. It would be more elegant for it to do a full 2500 iterations to produce that near 2500 result!
2. It first squares the number, then takes the square root. That's a little cheesy, because if every other result were perfect the "square, then square root" sequence would be operating only on whole numbers. Better to take square root first, then do square.
Those two imperfections stem from the common implementation of the Savage Benchmark on early PCs using some version of the BASIC language. In BASIC it's slightly more complex to do the job right, i.e., without the objectionable aspects I listed above. In common calculator syntax, it's just as easy to do the job right as not.
I use the Savage Benchmark version that corrects those issues.
Example of RPN version on HP 42S: (Put 2500 in X and execute SB)
LBL "SB" RAD 0 LBL 01 1 + SQRT X^2 LN E^X ATAN TAN DSE ST Y GTO 01 BEEP DEG END
Results: 2499.99946106 in 605 seconds
Example of RPL version on HP 50G: (Execute SB)
<< RAD TICKS 0. 1. 2500. START 1. + sqrt SQ LN EXP ATAN TAN NEXT SWAP TICKS SWAP - B->R 8192. / DEG 800. 2. BEEP >> 'SB' STO
Results: 2499.99946106 in 64.9 seconds
Example of Algebraic version on HP 20S: (Put 2500 in display and execute B)
LBL B RAD STO 0 0 LBL 1 + 1 = sqrt X^2 LN E^X ATAN TAN STO 1 1 STO-0 RCL 0 X=0? GTO 2 RCL 1 GTO 1 LBL 2 STO 1 DEG RTN
Results: 2499.99946106 in 350 seconds
One of the fastest HP machines is the original HP 30b, programmed in RPN. Its results are 2499.99946106 in 6.5 seconds.
The more sophisticated algorithms found the the WP 34S cause its results from the same hardware to be very accurate, but also rather slow even with external DC power: 2500.00000001 (actually, 2500.000000006559) in about 300 seconds.
Performing this on the WP 34S without external power will give variously longer run times. With fresh CR 2032 batteries I get about 450 seconds as the WP 34S automatically slows its speed when low battery voltage is sensed, even with new batteries, during the later portion of the run.
Edited: 25 Sept 2013, 7:18 p.m.

Re: Savage benchmark: Results for Improved Version
Message #36 Posted by Pier Aiello on 26 Sept 2013, 3:22 a.m.,
in response to message #35 by Mike Morrow

Quote:
It only iterates 2499 times to produce the near 2500 result. It would be more elegant for it to do a full 2500 iterations to produce that near 2500 result!
It's due to the first value, anyway is just aesthetic. Put 2500 on the stack, and then subtract from it (in the program) one. As you don't compute the single iterations involving tan, atan, log, exp, roots and squares, you can even leave the calculator to compute 2499 for you.

Quote:
It first squares the number, then takes the square root. That's a little cheesy, because if every other result were perfect the "square, then square root" sequence would be operating only on whole numbers. Better to take square root first, then do square
I disagree. Because if you do 2*2 and the square root, you have a chance to get 2. If you do square root of 2 first you have no chances to get 2, due to the fact that sqrt(2) is irrational and the calculators has a limited memory. But even you can't get 2 back if you do computations numerically, and not symbolically, on a paper. To get the perfect sqrt(2)*sqrt(2)=2 handling sqrt(2) as 1.414.... you need infinite time just to compute sqrt(2). So, imo, it's easy to get a false value if you do square root first. Then, accordingly to what i said, is better the version with square first for me.
Moreover i found some remarks about "accuracy by chance" here on MoHPC (i hope to find more discussion about that topic on comp.sys.48 when i will have time) and i reported them on the wiki.
In few words they say: the accuracy evaluation done on only a specific value of the test (in this case, 2500) has little meaning. The best is: do a batch of computations over consecutive values and check how the accuracy changes.
Edited: 26 Sept 2013, 3:29 a.m.

[ Return to Index | Top of Index ]

Go back to the main exhibit hall