Mailing List Archive

[ANNOUNCE] Hello World Benchmarks, updated
Hey,

The latest Hello World benchmarks at available at:
http://www.chamas.com/bench/hello.tar.gz

To reproduce the BELOW results on your platform, for
whatever tests are available, run:
./bench.pl -test
./bench.pl -version -time=60

--Josh


DISCLAIMER: these benchmarks test only what they test for,
and I try to make no assertions about the fitness of any system
for your needs. They are open source, feel free to critique
the tests as you like, and provide constructive feedback.

CHANGES: New HTML::Template 2000 test. My system is newly
built with apache 1.3.20, mod_perl 1.25.

NOTES:
mod_perl
The mod_perl environments are starting to look faster than PHP.
Maybe something in the default mod_perl 1.25 gave a performance boost?

mod_caucho
used to look a lot faster, but my testing methodology changed.
I used to take the results of the second benchmark run, and
publish those, but this time only ran the -test for minor
caching after starting resin ( & tomcat ). So, I'm guessing
that mod_caucho compiles aggresively in the beginning, killing
performance for a dry run ( even 60 seconds! ). To improve
the numbers for mod_caucho using this methodology might require
and longer test cycle than 60 seconds.

HTML::Embperl
2.0 series is much faster ( than PHP & ASP too), but 1.3.0 is stable,
& installable from CPAN, so I run that.

HTML::Mason
< 1.0 version used to be much faster on 2000 benchmark

mod_jserv
Why is the hello bench slower than the 2000 bench? I don't know.
It may be a funny caching strategy. It may be the JVMs were
cleaning up from the 2000 test during the hello test. Speculation
only, as I don't know much about java.

HTML static
for the first time, looks slower on my system than mod_perl.
It may be that the default 1.3.20 apache install/config does
more work on the headers. Also this is a dual proc system
which has slowed down the HTML static test before, relative
to single CPU systems.

Test Name Test File Hits/sec Total Hits Total Time sec/Hits Bytes/Hit
------------ ---------- ---------- ---------- ---------- ---------- ----------
Apache::ASP v2.19 2000 h2000.asp 226.2 13578 hits 60.02 sec 0.004421 28998 byte
Apache::Registry v2.01 2000 mod_per h2000.reg 339.5 20376 hits 60.02 sec 0.002945 28179 byte
HTML::Embperl v1.3.0 2000 h2000.epl 111.3 6677 hits 60.00 sec 0.008987 28841 byte
HTML::Mason v1.03 2000 h2000.mas 83.5 5014 hits 60.02 sec 0.011969 28799 byte
HTML::Template v2.3 2000 h2000.htmp 98.2 5892 hits 60.00 sec 0.010183 29152 byte
mod_caucho JSP 2000 h2000.jsp 76.6 4595 hits 60.01 sec 0.013060 28965 byte
mod_jserv JSP 2000 h2000.jsp 150.0 9002 hits 60.02 sec 0.006667 29408 byte
mod_php PHP 2000 h2000.php 228.4 13705 hits 60.01 sec 0.004379 28866 byte
Template v2.04 Toolkit 2000 h2000.tt 52.9 3176 hits 60.02 sec 0.018900 28889 byte
Apache::ASP v2.19 hello.asp 378.3 22706 hits 60.02 sec 0.002643 242 bytes
Apache::Dispatch v0.09 handler hello/worl 587.6 35259 hits 60.01 sec 0.001702 197 bytes
Apache::ePerl hello.eper 345.5 20742 hits 60.03 sec 0.002894 218 bytes
Apache::Registry v2.01 CGI Raw hello_raw. 669.9 40196 hits 60.00 sec 0.001493 52 bytes
Apache::Registry v2.01 CGI.pm hello.cgi 448.6 26924 hits 60.02 sec 0.002229 217 bytes
Apache::SSI v2.16 hello.shtm 533.8 32029 hits 60.00 sec 0.001873 200 bytes
HTML static hello.html 768.2 46120 hits 60.04 sec 0.001302 312 bytes
HTML::Embperl v1.3.0 hello.epl 459.8 27595 hits 60.01 sec 0.002175 221 bytes
HTML::Mason v1.03 hello.mas 373.3 22406 hits 60.02 sec 0.002679 198 bytes
HTML::Template v2.3 hello.htmp 535.9 32165 hits 60.02 sec 0.001866 199 bytes
mod_caucho JSP hello.jsp 88.1 5321 hits 60.37 sec 0.011345 231 bytes
mod_cgi CGI Raw hello_raw. 153.5 9210 hits 60.00 sec 0.006515 197 bytes
mod_cgi CGI.pm hello.cgi 11.0 662 hits 60.02 sec 0.090660 217 bytes
mod_include SSI hello.shtm 223.2 13396 hits 60.03 sec 0.004481 199 bytes
mod_jserv JSP hello.jsp 53.5 3223 hits 60.29 sec 0.018706 358 bytes
mod_perl handler hello.benc 808.4 48522 hits 60.02 sec 0.001237 197 bytes
mod_php PHP hello.php 665.9 39954 hits 60.00 sec 0.001502 226 bytes
mod_speedycgi hello.cgi 187.0 11222 hits 60.02 sec 0.005349 217 bytes
Template v2.04 Toolkit hello.tt 468.3 28103 hits 60.01 sec 0.002135 199 bytes
Apache::ASP v2.19 XSLT Hello hxslt.xml 222.3 13337 hits 60.00 sec 0.004499 280 bytes
AxKit v1.4 XSLT Hello hxslt.xml 297.2 17840 hits 60.03 sec 0.003365 397 bytes

Apache Server Header Tokens
---------------------------
(Unix)
Apache/1.3.20
AxKit/1.4
OpenSSL/0.9.6a
PHP/4.0.3pl1
Resin/1.2.1
mod_perl/1.25
mod_ssl/2.8.4
tomcat/1.0

PERL Version: 5.00503

JAVA Version: java version "1.3.0"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.0)
Classic VM (build 1.3.0, J2RE 1.3.0 IBM build cx130-20001124 (JIT enabled: jitc))

Operating System: Linux 2.2.14-9.0 (root@gate) (gcc egcs-2.91.66) #1 2CPU [gate.]

---------------------------------------------------------------------
To unsubscribe, e-mail: asp-unsubscribe@perl.apache.org
For additional commands, e-mail: asp-help@perl.apache.org
Re: [ANNOUNCE] Hello World Benchmarks, updated [ In reply to ]
Joshua Chamas wrote:
>
> mod_caucho
> used to look a lot faster, but my testing methodology changed.
> I used to take the results of the second benchmark run, and
> publish those, but this time only ran the -test for minor
> caching after starting resin ( & tomcat ). So, I'm guessing
> that mod_caucho compiles aggresively in the beginning, killing
> performance for a dry run ( even 60 seconds! ). To improve
> the numbers for mod_caucho using this methodology might require
> and longer test cycle than 60 seconds.
>

Perrin convinced me that getting "steady-state" numbers
was important enough to ignore the compile penalties
for such apps as mod_caucho which does runtime java
compilation.

To this end, I have added a -prime switch to the bench.pl
which will run all the tests for a couple seconds, before
doing the real benchmark where the results are scored.
This allows any system caching to be done before the
numbers start counting.

However, despite the new -prime setting, I was still not
getting reproducable resin/mod_caucho results, varying
from 80 hits/sec one run to 280 hits/sec the next for
the Hello World 2000 JSP benchmark.

What I found is that in order for the mod_caucho results
to be reproducable to 5% variation from one run to the
next, I need to run the benchmark for 10 minutes! At 3-5
minutes, there was a 10-15% variation, which seems too high.
Benchmarks < 120 seconds, which were my benchmarks from before
seem next to meaningless now.

Here's some #s to show what I mean:

Test Name Test File Hits/sec # of Hits Time(sec) secs/Hit Bytes/Hit
--------- --------- --------- --------- --------- --------- ---------
mod_caucho JSP 2000 h2000.jsp 75.4 9069 120.33 0.013269 28965
mod_caucho JSP 2000 h2000.jsp 249.4 2505 10.05 0.004010 28965
mod_caucho JSP 2000 h2000.jsp 274.5 4118 15.00 0.003643 28965
mod_caucho JSP 2000 h2000.jsp 191.9 5760 30.01 0.005211 28965
mod_caucho JSP 2000 h2000.jsp 133.7 8022 60.01 0.007481 28965
mod_caucho JSP 2000 h2000.jsp 104.8 12591 120.10 0.009539 28965
mod_caucho JSP 2000 h2000.jsp 96.0 11581 120.66 0.010419 28965
mod_caucho JSP 2000 h2000.jsp 113.6 19937 175.51 0.008803 28965
mod_caucho JSP 2000 h2000.jsp 103.9 18283 175.98 0.009625 28965
mod_caucho JSP 2000 h2000.jsp 91.4 27146 297.16 0.010947 28965
mod_caucho JSP 2000 h2000.jsp 105.0 31335 298.57 0.009528 28965
mod_caucho JSP 2000 h2000.jsp 91.4 27210 297.76 0.010943 28965
mod_caucho JSP 2000 h2000.jsp 88.8 52440 590.56 0.011262 28965
mod_caucho JSP 2000 h2000.jsp 93.5 55129 589.64 0.010696 28965

The numbers seem to indicate that the resin/mod_caucho java
environment gets slow the longer tests are run for, but also
that the results stabilize the longer tests are run for.

As a result, when I post benchmarks, it will be with
10 minute tests run in the future. The idea here is
stable, repeatable benchmarks we can use.

Here were the 60 second #s I had posted before just to show
how far off the mod_caucho #s were.

> Test Name Test File Hits/sec Total Hits Total Time sec/Hits Bytes/Hit
> ------------ ---------- ---------- ---------- ---------- ---------- ----------
> Apache::ASP v2.19 2000 h2000.asp 226.2 13578 hits 60.02 sec 0.004421 28998 byte
> Apache::Registry v2.01 2000 mod_per h2000.reg 339.5 20376 hits 60.02 sec 0.002945 28179 byte
> HTML::Embperl v1.3.0 2000 h2000.epl 111.3 6677 hits 60.00 sec 0.008987 28841 byte
> HTML::Mason v1.03 2000 h2000.mas 83.5 5014 hits 60.02 sec 0.011969 28799 byte
> HTML::Template v2.3 2000 h2000.htmp 98.2 5892 hits 60.00 sec 0.010183 29152 byte
> mod_caucho JSP 2000 h2000.jsp 76.6 4595 hits 60.01 sec 0.013060 28965 byte


--Josh

---------------------------------------------------------------------
To unsubscribe, e-mail: asp-unsubscribe@perl.apache.org
For additional commands, e-mail: asp-help@perl.apache.org