UPDATE: This is a followup – please read the roogiional – and remember, these tests are NOT Microsoft official and NOT endoursed by my Employer.
Last week I posted some comparative performance result that I got from comparing the relative performance of PHP on PHP 5.2 to PHP 5.3, PHP on Linux to PHP on Windows and PHP to ASP.NET.
First, I want to thank the PHP community for their responses. With only a few exceptions the PHP community proved that they are not a bunch of teenage jealous (anymore 
? ) but rather a group of professionals, confident in their work and ready to engage in respectful dialog.
A couple of my friends in the PHP community (See: to Andi Gutmans & Brandon Savage) took exception to the perf results – and for fair reason.
In my tests I declined to install PHP op-code caching for my tests because its not installed on some of the hosts I use and is not built in to PHP (yet – though I understand it will be in a future version and is part of Zend’s free Community Edition Server)
Andi, Brandon, and other pointed out that in business application deployment, most PHP shops use op code caching, so I installed it on Linux and Windows and re-ran most of the tests.
Note that the PHP 5.3 with APC tests were run on a clean Debian 5 setup on the same hardware and running the same code. I moved to Debian because building on Ubutu was giving me dependency fits. You’ll note that the performance delta between Ubuntu and Debian was negligible.
You can click on the image below to see the whole updated times table.
[ Click here for the whole table. ]
Some items showed a small improvement in performance, some stuff up to 25% faster, but overall it was far less than I expected.
Some things ran slower with APC but I attribute that to simple machine variance. (Note, the numbers in the table are NOT the first page run. I loaded the page, refreshed twice to make sure to hit the cache and then took the numeric results.)
Some folks suggested that these weren’t “fair” tests.
I disagree.
An empty loop test or an empty function call, for example, is very important because the results indicate language or platform OVERHEAD. That overhead is addition latency in page delivery and this, if it exists, is an important consideration.
A few of my PHP friends concede the accuracy of the results but made very insightful comments.
I’ll paraphrase them here.
- The amount by which ASP.NET is faster than PHP doesn’t matter to me. PHP is simply my preference and it’s fast enough in my applications.
- Sure, ASP.NET might be faster in raw execution but in my web applications I can easily buy that performance back with good page and JavaScript practices.
- I build on Drupal and I know PHP best so what I might need to spend on more hardware I make up for in developer productivity.
These are all excellent points !
Though I think the tests that I’ve run so faw are fair, they are incomplete and their incompleteness prevents us from seeing the whole story.
Contrary to the couple of inevitable allegations that I’m “just” a Microsoft shill. I’m doing this because I want to KNOW.
My last tests revealed things like PHP file access and MySQL access on Windows has real performance issues and this information was received by the various teams involved here at Microsoft. They are following up and hopefully the data will directly result in improvements.
So – HERE IS YOUR CALL TO ACTION.
Use the contact form http://misfitgeek.com/contact/ and tell me what tests you would like me to run in order to grow my test suite in ways that you think are fair and meaningful.
Here are some of what I’m planning.
- Real Page tests – loops, calls, and object instantiation are one kind of test, but full page rendering is another meaningful benchmark.
- Load tests – can one environment or another handle more simultaneous requests.
- During Simultaneous requests, does one stack degrade performance more than another.
- In the above context – how much does 64 bit matter on each platform
I hope you’ll send me your suggestions.
RE: PHP Linux Windows ASP.NET Performance – Redux !
Pingback from Comparativo entre PHP e ASP.NET | André Baltieri
Ill be installing IIS7/PHP5.3/Win2k8 and compare it to apache on same box… since you wont! you can add it as a footnote about how no one who is sane (i am insane) should use apache (even as fastcgi) on windows (even though i do… in production! *dundundun*). BUT WE WILL SEE!
Very cool – let me know if I can help.
RE: PHP Linux Windows ASP.NET Performance – Redux !
Pingback from Linux | All Days Long
Joe, thanks for redoing these tests and posting more on your thoughts. I’ve responded: http://www.brandonsavage.net/…/of-lies-damned- Have a greaet Thursday!
RE: PHP Linux Windows ASP.NET Performance – Redux !
Pingback from PHP Linux Windows ASP.NET Performance ??? Redux ! : Misfit Geek | Webmaster Tools
Just one nit: in the php code many of the test have an echo statement that prints the results to screen inside the time measurement. I think the echoing should be outside of the measured part, as it’s not the IO system that is being measured here, but the plain execution speed of simple operations…
RE: PHP Linux Windows ASP.NET Performance – Redux !
Pingback from PHP Linux Windows ASP.NET Performance – Redux ! : Misfit Geek
I just wanted to know why you’re not running the Tests with ASP.NET v3.5 SP1 and W2K8? If I know it right the .NET Runtime is more optimized in the v3.5 SP1. And with NGENed assemblies?
RE: PHP Linux Windows ASP.NET Performance – Redux !
Pingback from PHP Linux Windows ASP.NET Performance ??? Redux ! : Misfit Geek
Why do ASP.NET stats have a different decimal precision than other tests? Did you use a different method to collect the execution time? That could be a source of error in your experiment.
Personally, I take exception with any "you didn’t turn this on" comments for either side. IMHO, the software ought to install out of the box in the most commonly used running configuration, and that’s what the tests should be based on. If that’s what you did, then it’s perfectly fair.
It’s a little like Michael Johnson claiming it wasn’t a fair race because he didn’t have his gold shoes on. Well, that may be, but that was his own responsibility, yes?
These days, I would expect an enterprise class system to do a little inspection and optimization on its own to save me time, thank you very much.
RE: PHP Linux Windows ASP.NET Performance – Redux !
Pingback from PHP Linux Windows ASP.NET Performance ??? Redux ! : Misfit Geek
RE: PHP Linux Windows ASP.NET Performance – Redux !
Pingback from PHP Linux Windows ASP.NET Performance ??? Redux ! : Misfit Geek
RE: PHP Linux Windows ASP.NET Performance – Redux !
Pingback from PHP Linux Windows ASP.NET Performance ??? Redux ! : Misfit Geek
RE: PHP Linux Windows ASP.NET Performance – Redux !
Pingback from PHP Linux Windows ASP.NET Performance ??? Redux ! : Misfit Geek | Tech Talk
Really cool stuff.
First, I think this "series" is great. Keep it up.
Second, since it’s early on in the comments (well before the "you sux" ones that will come later), I think the author’s been pretty clear about unbiasedness and at most, ultimately a subtle friendly challenge to the PHP community should the results tilt in ASP.NET’s favor.
If you look at the benchmarks, I mean my god, who is going to perform 2 gazillion encryption / object creations in less than a second. Maybe this is an area of actual concern for Google or MSN, but not for even well-trafficed sites like the WordPress.org community blogging site and such. That is, PHP and ASP.NET both share a common denominator of being proven, reliable frameworks for large scale web applications.
So before the "OS / LAMP vs. M$FT" bull-shisten starts hitting the fan with this Redux post, let’s just try to prove Joe wrong and THEN sling comment-mud at him
Third:
ASP.NET is simply awesome. Complete and unashamed bias here…
Tell me I am bias, But I am strongly for Asp.net. Its not just about these performance but also other eas of development that accompany asp.net. Also code reusability is a big stuff, we can use the same code in silverlight wpf or other windows applictaion(I am not saying all code but a lots of code in a medium size application).
"… and php died. But at least we will remember the php generation, and how it forced Microsoft to create the Microsoft Public License, and at least release some of their products for free."
Or not. But the fate of php is in the hands of the companies backing up php. The way they are doing it now, though, is making things worse. For example, Zend created the Zend Framework to counter-act RoR, but they didn’t through a monolitic php library that every day looks more like PEAR. Microsoft did the same, through ASP.NET MVC, but they did it through a dll library of (probably) C++ compiled code. Imagine if Microsoft created their MVC framework in C# code to run at runtime, crazy right? Well, that is what the php community is doing with all that bunch of php libraries they want to call frameworks (e.g.: CakePHP, Symphony, Zend, CodeIgniter, Kohana, etc). What someone has to do is a compiled C++ MVC dll for php.
Actually, even the creator of the php language is saying that php alone is not enough to run a big size. He strongly criticise the framework movement and suggests direct C++ extensions for php. I couldnt agree more.
Im also tired of the argument "but most websites never reach those limits, so php is good for them". Respectfully, that’s crap, I think that we have to think big, and my goal is to create the next google, the next yahoo, the next facebook, not just my little blog. Anyway, just me venting.
Joe, thank you for the tests.
ASP.NET is certainly generally faster(compiled), more scalable, more secure, more enterprise grade …
but PHP is lightweight, easier to code (for beginners and scripters), less resource hungry and pretty fast (for simple non transactional SQL reads on MySQL)
seems to me the old Java vs ASP (classic) debate … just as ASP/VB scripters very derided by Java people 10 years ago for their bad (spaghetti) non-OO code, now the PHP/Ruby scripters are "fashionable"
RE: PHP Linux Windows ASP.NET Performance – Redux !
Joe Stagner ha publicado recientemente una comparativa de rendimiento de PHP versus ASP.NET , en diferentes
RE: PHP Linux Windows ASP.NET Performance – Redux !
Pingback from Of Lies, Damned Lies, And Benchmarks ??? Redux! – Brandon Savage | rapid-DEV.net
@Luis Morales
ASP.NET mvc sources can be downloaded from http://www.microsoft.com/…/details.aspx
It is all c#, not "dll library of (probably) C++ compiled code" as you say.
RE: PHP Linux Windows ASP.NET Performance – Redux !
Pingback from PHP Linux Windows ASP.NET Performance ??? Redux ! | rapid-DEV.net
@Tomas S
You are right, it’s C# (and probably they have a VB code too), but it will still go compiled when you deploy it. I thought it would be C++, to be honest. But the argument remains the same. The php "frameworks" dont get compiled to byte code (except of course, the unpredictable [see Joe's tests] accelerators / caches). With ASP.NET MVC, you can make the changes to the source, if you want, compile it and load it to the server. What you use is a compiled framework. Actually, everything (even your application) gets compiled, that’s why asp.net is so much faster than php.
My point is that, we could do the same with php. Instead of creating the frameworks in php, create them in C++, compile it and then use the MVC framework as an extension to php. Of course, you could change the source code to your needs, but in C++. Isnt this obvious, or is it just me?
RE: PHP Linux Windows ASP.NET Performance – Redux !
Pingback from Dew Drop – August 14, 2009 | Alvin Ashcraft’s Morning Dew
Great work Joe…Thanks for the redux to answer the cache/byte code questions. Such a rabble-rouser! (^o^)
These are interesting benchmarks.
Sadly as I suspected APC did not help much in this benchmark as most of the time is spent in loops and not reloading PHP code (which is what the opcode cache is designed to do).
Would you be able to take things further and test how many pages a second can be served as this is a very useful metric. Try 1, 5, 10, 20, 50 parallel accesses over a given time frame.
I know this tests the server performance as much as the PHP/ASP.net execution speed, but would be a nice comparison, especially to see how the opcode cache of PHP responds (which I suspect will be very positive).
Cheers, Nick
@Craig I’m with you on this I don’t think Joe shouldn’t have to turn anything on or off to run these tests, straight out of the box should be the standard starting point for any bench mark when you tweek things one way or another in PHP or ASP.NET you have a customized version of either technolgy. I think the cry babies should cowboy up and let the chips fall where they may
Good work Joe thanks for stickin your neck out
RE: PHP Linux Windows ASP.NET Performance – Redux !
Thank you for submitting this cool story – Trackback from DotNetShoutout
There are lots of things to consider, besides benchmark results. My comparison includes:
1- Platforms supporting the technology:
- ASP.NET: Microsoft Server and Apache (Mono Castle)
- PHP: All known commercial and community platforms.
2- Corporations Backing or Contributing to the technology:
- ASP.NET: Microsoft
- PHP: Microsoft, Zend, IBM, Oracle, Sun, Google, etc.
3- How intuitive is the technology:
- ASP.NET: Good
- PHP: Excellent
4- How flexible is the technology:
- ASP.NET: Debatable.
- PHP: Flexible by nature, not debatable, almost an absolute fact.
5- Performance:
- ASP.NET: Excellent on Windows Server. The Mono / Castle project is still working on performance.
- PHP: Good on most platforms. Results may vary depending on a lot of settings out of the box or included by extensions. But given to an expert on PHP and Apache, can be as fast as ASP.NET on a Windows Server.
6- How expensive would be implementing the technology on a large scale:
- ASP.NET: ask MySpace and Monster (very expensive but works good out of the box = good for Fortune 500).
- PHP: ask Facebook, Flickr and others (considerably cheap and there are no license issues = good for startups.)
@Luis Morales
ad 1 Platforms
Agree
I think that Mono is very important for long and healthy .NET life until Microsoft lightens Windows server licencing garbage for web facing servers. It is still hard to find good Windows server hosting in Europe – especially VPS (small companies are probabaly facing some barriers). Mono is the solution.
take a look at http://news.netcraft.com/
ad 2 Corporations Backing or Contributing to the technology
Oracle and Sun are now one company and google contributes rather to java and python.Besides there is one strong point having big company behind product – consistency and completness. I would rather have 2-3 full featured frameworks then gazillions half baked stacks.Third thing is that MS is commited to have great stack , if not they will loose and fade away
ad 3 How intuitive is the technology
Agree here, php is very easy to grasp (asp.net mvc is quite simple too)
ad 4- How flexible is the technology
ASP.NET is very flexible, much more then php for me .
Frameworks – choose between Webforms/MVC/wcsf/monorails … or write one, Data layer – Nhibernate/EF/Linq/Subsonic … + commercial ofering or write one . Flexibility is determined by performance too. Faster the platform – more you can do with it.
ad 5 Performance
There are some comparisons flying over the web, but I will do my own tests between mono(2.4)/linux and win . As to performance, the more advanced application the more php lags behind .net .
Try to write layered, loosely coupled architecture with advanced security, validation, AOP features … – you will see. For simple, "short path" applications php is fast enough.
ad 6 How expensive
I don’t think that costs are important here. Use Windows only for web serving and FreeBSD/postgres for database. Blend technology and use what is best suited for the task/scenario .You see more applications written in php because it is much more popular. Google/Yahoo/Oracle won’t use .net from known reasons.
These benchmarks are not about PHP vs ASP.NET, Windows vs Linux, but rather .NET engine versus Zend Engine.
To make comparisons between ASP.NET and PHP, you should use Phalanger to compile PHP into .NET assemblies, if Phalanger was a mature project.
PHP is just a language. There are several compiler/executors that can be used to run PHP besides Zend Engine (Zend opcodes), like Phalanger (.NET assemblies), Quercus and Project Zero (Java bytecodes) and RoadSend PHP compiler (native machine code).
When you mix runtime environments you are comparing virtual machines, not languages.
Besides that PHP is a language. ASP.NET is not a language, but rather a framework. C# is a language. Java is a language. If C# and Java are faster in certain circumstances, you can run Java and C# code inside PHP code, but not the other way around.
Anyway, personally I hate these benchmarking comparisons because they always intend to persuade people to move to one side, often taking advantage of their ignorance of certain aspects they were not aware.
I wish Microsoft or their employees would refrain from publishing these benchmarks as they pass the idea that Microsoft is trying to promote a war to discourage people from using PHP, and use instead whatever products Microsoft wants to push.
My mother used to say, "you will never catch flies with vinegar", meaning you should avoid attempting to attracting people that prefer using other platforms by saying those platforms are a bad choice.
If Microsoft wants people to use their products, my advice is to stop publishing benchmarks that intend to make people to believe that PHP, Linux or any other alternative technologies are bad.
Steve Ballmer was very unfortunate when he said that Linux is a cancer. For Microsoft’s sake, do not repeat his mistake. Otherwise, you will only be promoting a stupid war that will ruin Microsoft credibility further.
This post couldn’t have come at a better time for me, as I was doing a lot of analysis of both technologies for my next project. I’m still undecided, and I’m actually thinking about throwing jsp / Spring into the equation.
I was watching a Seinfeld episode (the Parking Space, season 3), and George said about parking garages:"Parking garages are like going to prostitutes, why pay for them if can get it for free if you apply yourself"(or something to that effect.)
I find it funny and thought the same principle applies to ASP.NET and PHP; why pay for ASP.NET (hosting, ide, others) when you can get it for free (PHP) if you apply yourself? Well, analysing it further:
- the investment in PHP, like real (non-prostitute) relationships, you spend a lot of money and time on dates and stuff; but once you have a relationship you have a secure source of pleasure.
- the investment in ASP.NET is as frequent as you want pleasure, but you know for sure that whenever you want it you’ll have to pay for it.
I dont know, I find the analogy (of both parking spaces and web-technologies) funny.
Manual,
As the publisher of the / one of the most popular PHP websites, I understand that you would be displeased about a comparison that suggests ANYTHING is not better in PHP. Some stuff is better in PHP, some stuff is better in other development choices. This is true of all things.
Perhaps your love for PHP has driven your comments and your assumptions about my motivations.
My mother also had a saying. "Ignoring things don’t make them go away."
To say things like ASP.NET is not a language, or – you should have compared Phalanger is kind of silly to me.
Clearly, as other people seem to have understood, the idea was to see If, as a programmer, I make certain choices…. Where would things run faster.
As my posts clearly state, this is not a declaration that anything is "bad", to the contrary, I reiterated several reasons why one might choose to overlook these performance results.
It seems to me that your complaint here is not that ASP.NET (in my tests) is faster than PHP, but rather that your emotional investment in PHP makes it offensive for me to suggest that ANYTHING is not perfect in PHP. Your passion is comendable.
Whether you believe it or not, I love BOTH PHP and ASP.NET and I work in them both every day. (Yes, it’s possible, just as I love BOTH my children.)
I also want BOTH to be the best that they can be.
The CLR and IIS teams have both emailed me asking "where are we slower, and how can we make our stuff faster and better".
Isn’t THAT how developers should react to comparisons of their work?
In kind, some PHP folks have emailed me to suggest additional tests to see how PHP realy compares in more "real world" scenarios.
These suggestions come from folks in the PHP world that don’t want to complain about the results of my tests or my motivations but who really want to know the results. This is good for PHP and the community.
You also suggest that becuase I work for Microsoft I shouldn’t be allowed to write about anything that is not "offical Microsoft". Microsoft is my employer. I enjoy the people and the products I work with, but being a Microsoft employee does not void me of my own personality or my own opinions. If if it did, this blog wouldn’t exist.
By the way, you might be supprised to know that I’m currently being told the same thing by a group here at Microsoft who suggest that we "just don’t want to risk make anyone upset with us".
For me, if the results of a fair test make people angry at me or even sillier, my employer, well,,,,
Some people are going to hate Microsoft (and Linux, and Fords, etc) no matter what.
Personaly, I think the idea of hating a company is foolish, but some folks like to hate and other need to channel their anger about another entities success.
For me, I just really wanna know how things compare. I think it’s valuable and interesting.
An old frind used to tell me "Good information doesn’t displace OTHER good information."
In any event, thanks for your comments, See you at http://www.phpclasses.org/ which remains one fo my favorite web sites.
If you have suggestions about tests that DO matter to you, I hope you’ll send them.
Joe you are missing my points.
First let me clarify that I am all in favor of Microsoft approaching the PHP core developers to work together in making PHP run better, even if that effort only results in benefits for PHP running on Windows.
For me Microsoft is a company. I do not have feelings about companies. So I do not hate nor love Microsoft. I have feelings about people. Actually I do not even hate people. That is is a very heavy feeling for me to keep against any person. I may not sympathize with some people. But when that happens, I just move away. All this to say that if I did not sympathize with you, I would not even bother to comment your posts.
But back to to subject, I don’t know if you are aware, in 2008 Microsoft invited me and many other well known PHP developers to go to WebDev Summit in Redmond. I went there and even gave two talks: one about PHP community in Brazil and another about what Microsoft can do to make PHP better on Windows. So, I really wanted and still want to cooperate.
When I returned I published this post in PHPClasses blog:
http://www.phpclasses.org/…/85-What-is-Micr
It is a very long post, but to sum it up, in my opinion I see with good eyes this approach of Microsoft towards the world of PHP and other Open Source technologies.
Fortunately for Microsoft many people agree with me. Microsoft was mean to Free and Open Source projects in the past. But at the present things seem to have changed.
Microsoft no longer seemed to attack and say things that sound like "our products are good because free and Open source software suck in some way".
So what is good in this new approach of Microsoft to the PHP community? Microsoft sponsored several PHP events on which they were allowed to give keynote speeches and tell the PHP communities that their intentions are good. In exchange the events get some money to fund the coming of some speakers coming from far places or even other countries.
But in my vision, the greatest benefit that the PHP community may get from Microsoft is greater credibility in the enterprise world of companies that are Microsoft customers. Hopefully this may mean more a better paid jobs for PHP developers.
So what is wrong with your post? Basically you are going back to the old days, promoting confrontation between Microsoft products and Open Source alternatives. As I said in my post, that is neither good for Microsoft nor for the Open Source world.
Why does that bother me? Believe me, I am not a PHP die hard. What bothers me most is that posts like yours compromise the credibility of me and others that have been helping Microsoft to be a more trustworthy company among the PHP community.
Believe me, I do not take an hit when you say that PHP performance is slower than ASP.NET in equal conditions. If there is anybody that should take an hit for that is Zend and core developers that have commit karma to make things better. It is their work you are questioning, not mine.
What compromises the credibility of your posts, is that you did not make comparisons in equal conditions. Actually it is even impossible to make comparisons in equal conditions.
For instance, I suppose that ASP.NET assemblies go through an optimization stage and the resulting assemblies are stored in cache to skip those stages in the next executions.
In PHP, the official distribution does not come with any Zend opcode caching or optimization extensions. There are several Open Source opcode caching and optimizer extensions but Zend people always objected to ship them within the PHP official distribution (except for PHP 6 which is not ready), obviously to protect their business of selling their on "PHP accelerator" extensions.
Anyway, even if you could make technically fair compariso
Oops, your blog truncated my post. What happened? Did it hit some SQL server record size limit? Never mind, it just truncated the last paragraphs. I do not remember exactly what I wrote, but it was something like this:
Even if you could make technically fair comparisons between PHP and ASP.NET, my advice is that you publishing them this way.
I do not take an hit when you make comparisons between PHP and ASP.net. If is there anybody that may take an hit because of this, is Zend people or maybe other non-Zend PHP core developers, because they are responsible for developing the PHP run-time execution code.
I am sure PHP benchmarks are welcomed but the right place to post them is the php-internals mailing list. That is where the people in charge of PHP core development hang. They are the ones that may consider any points about cases on PHP with Zend Engine does not do so well.
Publishing PHP versus ASP.NET benchmarks in your blog or some other public place is a big no no to me. You will only foster controversy, promote confrontation between Microsoft and PHP communities, and give reason to all those that say that Microsoft should never be trusted no matter what you do.
I am all for peace between the PHP and Microsoft people. I hope you are too and now you see my points.
PS. So you clearly understand on which side I am, I suppose you are not aware that next week I will be at TechEd by invitation of Microsoft Brazil to participate in an "Ask the experts" session about interoperability.
Also, just a few days ago I posted this interview on the PHPClasses blog about PHP and Windows Azure.
http://www.phpclasses.org/…/100-PHP-on-Wind
I am all in favor of having PHP support on Windows Azure, even more because Google does not seem to care about PHP to make it run natively on Google App Engine, and other cloud solutions that may support PHP are not easy to implement.
asp rocks!
I am a beginner and I don’t really understand why people are saying that PHP is for beginners. I graduated about a year ago (undergrad), and I had an advance web development course that introduced both technologies. The course had an emphasis in PHP but we were free to utilize whatever was more convenient or rather easy. I tried PHP and I personally though it was very difficult. However, ASP.NET was so easy, code re-usability was amazing, and my overall understanding of the subject (computer Science) became much clear with the "oh so horrible Microsoft Empire" (quoted sarcasm). My projects at school were the best ones in each class because of ASP.NET, everyone else used PHP and were very upset. So the assumption that PHP is for beginners is wrong, lets just leave it at some people rather use one over the other for x reason
I’ve look into these wars sometimes from both sides to try to understand each side. This is actually the first one I see from and ASP.NET point of view, all other have been from a biased PHP view. Thanks!
RE: PHP Linux Windows ASP.NET Performance – Redux !
Pingback from Programming news: Windows 7 Code Pack release, an anti-IE 6 campaign | Programming and Development | TechRepublic.com
Joe I have a set of issues with your test. The first is what I mentioned in your previous post. Why does it take more to do less? Your ASP code with no method call (do loop) seems to be taking longer (when extrapolated) than the same do loop doing a method call.
How many tests are you averaging to obtain those results? Are you just publishing the results of a single run?
Here’s a little food for thought. I took a bit of Java code and ran it to see how long it would take on the same machine when running Windows and then Linux.
Code:
for(int j=0;j<100;j++) {
for(int i=0;i<20000000;i++) {
}
runTime=System.nanoTime();
System.out.println("Run:"+(runTime-nowTime));
totalTime+=runTime-nowTime;
nowTime=runTime;
}
Basically 100 iterations of a 20 million empty for loop.
Results:
Windows: average :31510736 (milliseconds) "jdk1.6.0_07"
Linux : average :214105 (milliseconds) "jdk1.6.0_13"
Interesting how the same jdk can produce such different results on two OSes. But it doesn’t end there. Here are a few samples of the times per iteration:
Windows:
Run:33954735
Run:29782346
Run:29490620
Run:28964086
Run:31581947
Run:29131915
Run:28100359
Linux:
Run:7264821
Run:5385321
Run:284534
Run:84997
Run:84647
Run:84229
Notice how on Linux the time drops quickly while it remains practically the same on Windows. If I take the first 5 runs out from the average then it drops from 214105 to 93133. Going from aprox 150 times faster to aprox 340 times faster.
So if I just took the first run on each OS and compared them, the conclusion would be "it runs 4.7 times faster on Linux (33954735/7264821). When I average this out over 100 runs, the performance is 150 times faster. If I remove the first 5 runs it runs 340 times faster. Now I can discard the first five because in a real life long running application these events would be negligible over the life of the application.
I’d appreciate it if you could do repeated runs of these tests and then average out the result. See how they perform.
@Manuel Lemos,
What tha heck? This is Joe’s personal blog, he can say whatever he wants. I didnt see anywhere in here that he was representing Microsoft in this blog. Manuel, we call it FREEDOM OF SPEECH, and we have paid a huge price for it.
I did my own benchmarking; asp.net was a little bit faster, not considerably, though.
Now I’m testing the development environments. I just finished ASP.NET 15 minute database (a tutorial from their site.) Was kind of smooth. I had never worked with the Entity Framework, so the modeling part is was a little bit shaky. Plus I like to see the getters and setters with my own eyes, so I didnt like the graphical interface so much. It was a good experience overall.
Now I’m ready to test the Zend Framework Quickstart from their website on NetBeans and using ZendTools from the Command Line. Will report afterward.
Well, here reporting back as promised in my last comment (see above). I just finished creating an application, more or less, similar to the one I did under ASP.NET with Visual Web Developer.
The ASP.NET version took me around 20 minutes. Took me a little longer than expected because I was not familiarized with the entity framework and didnt know the importance of consistency in the namespaces so you objects are strongly coupled (this is a very important subject to which I’ll come back.)
The Zend FM / NetBeans version took me around 1.5 hours. First to set up the project took me a long time setting up the php.exe, php.ini, zf.bat and zf.php correctly. Bottomline, the Quickstart "tutorial" is not clear on how to set this up correctly and I think assumes the person is a Unix user. I had to dig very deep in the Reference Guide to find out that all the files mentioned before have to be in the same folder, and that to create the project your command line has to be in that folder. I think they assumed all users were Unix because in Unix you can do something so you can call a sh file from wherever. I dont know if I can do the same in Windows, this is like the 5th time I use the command line in all my life.
Once I set up the project the base application run ok. So I started by creating my database. Which took me a long time because NetBeans Database explorer is a pain-in-the-ass so I had to go to my phpMyAdmin local version to set up the columns.
Once I had the DB ready I formally started creating almost a similar application from the one I did in ASP.NET following the Quickstart. In ASP.NET setting the model took me like 2 minutes. In Zend / NetBeans I had to hardcore everything, which is good because I followed the logic while I was typing.
But, here comes the strong point of Visual Web Developer, all the user interface tools help you in creating strongly typed objects; and by following the intellisense you know if a variable or an object exists. I can imagine that in a large development shop that feature prevents a large percent of future bugs. No tests are necessary to get that capabilities in the Visual Studio environment.
For PHP the IDE of my preference is NetBeans (I cant stand Eclipse [when it wishes to install in my Windows PC]). Although NetBeans has awesome capabilities for PHP, it doesnt guarantee that you are creating strongly typed objects. In my example, in the Model I didnt initially create the protected $_mapper variable. Once I had the view ready, I got an error saying that the property wasnt recognized. I had to create a test to see which property it was talking about, and then I found out it was the $_mapper variable. That whole process took me at least 40% of the time.
The end result products vary significantly in quality, as well. The ASP.NET version has edit, details and create links; which where created automatically, all I would have had to do was create the views, which only required one right click the corresponding controller action, yeah, just one click Im not kidding. The Zend / NetBeans version just has one plain list of the items.
@Joe,
Yes, me again. As you can see this post got me interested in the subject. I did some additional analysis, and I think I now understand why some experts (like Rasmus Lerdorf) believe PHP is not designed to be as fast as a compiled language. I mean it just wont, there is no way. Even with all cache and accelerators you try to use. Unless… well this is kind of absurd but I’ll throw it anyway; unless you can cache the whole application. There I said it, make fun of the newbie now. Can you please provide some feedback?
Luis, you also missed my point. I never forbidden anybody from publishing benchmarks of PHP against anything. Who am I to forbid anybody?
My advice is just for Microsofters stop doing things that upset PHP and other Open Source community members that they want to cooperate. Posting articles like this that use synthetic benchmarks to "prove" that Microsoft products are faster then Open Source products is one of such things that upset those communities.
Even if Joe says it was not his intention, it all sounds like an attack to put PHP down. It is counterproductive for Microsoft and only puts off for future cooperation.
Furthermore, let me clarify that PHP is a compiled language since PHP 4.0 launched in 2000. That is almost 10 years ago. PHP with Zend Engine compiles PHP code into opcodes, which are similar to Java bytecodes and .NET assemblies. The difference is that PHP Zend opcodes remain in memory by default rather than be saved and loaded from files.
Anyway, all of that have busy PHP sites, to maximize performance we compile PHP as static library of Apache 1.x . We also avoid using shared libraries because those impose a performance penalty of about 30% due to inefficiencies introduced by using relative addressing that shared libraries require.
ilia.ws/…/32-Apache-1-vs-
Most Linux distributions of PHP built it as shared libraries of Apache 2.x to make it easy to disable PHP or any of its extensions. This also seems to apply to Zend PHP server (at least for community edition).
So all these benchmarks that run PHP builds based on shared libraries are affected by this performance penalty.
Maybe neither you nor Joe did not know of this detail. In any case, the results of the benchmarks are unrelated with real world usage. That is one more reason why publishing them will only upset the PHP community.
Manuel,
Hi. Thanks for the info. Is there an article I can read to learn about php runtime process? I am reading several books right now, but they are about the syntax and usage of the language, not about the Zend Engine / PHP run. I guess not every body is interested in that subject.
http://www.php.net/…/internals2.php
Hi Manual,
I guess we have to agree to disagree. If the publication of data or opinion "upsets" a communuty then I’d suggest the problem lies with the community.
I think this discussion PROVES that many PHP developers are happy to participate in a meaningful discussion.
Perhaps the data lacks "real" meaning some (I think the information has value.)
The presentation of information shouldnever be considered offensive.
I was a member of the PHP "community" before I joined Microsoft. If my tests indicate that some things >NET are faster than PHP, or than some things Windows are fatser than Linux – then that data is simply DATA (not insult).
The HUNDRENDS of comments and emails I’ve recieved (minus the ASP.NET guys) have nearly ALL be suggestions (do this next, or would you try this).
PHPers offering the Microsoft guy suggestions – NOT on how to make PHP "look" better in the tests, but rather to help evolve this series to be more meaningful.
I would have published the resuls no matter what they were so that people that found the data actionalbe would have access to it.
I plan to do this as I do more tests no matter what the results.
Teams at Microsoft have come back and said "tell us where you think we should do better" and what will you test next"?
If PHPers don’t think these results matter, then they can ignore them.
I love that some folks have said, "I don’t choose PHP for high performance, I choose PHP becuase ….."
In any event, thanks for participating in the conversation.
Manuel,
I’ve been reading about ASP.NET ORM implementation: Entity Framework. I am amazed, this is the way web development should operate. Do you know if there is a similar implementation in PHP? I want to know who is pushing the evelope on this regard. I think this is way more important than the performance features.
You know, after a week to think about it, I’ve come to the conclusion that the real indicator on all of this (relative to TCO) is going to be cloud computing. Once the comparative numbers start showing up there, companies are going to start seeing the bills separated out a bit. They’ll look at them and say, "ASP.NET costs X amount to run, and PHP costs Y amount, on average." They will also look at development times and say, "ASP.NET averages X amount of man hours, and PHP averages Y amount." And of course, "ASP.NET development tools cost X, and PHP development tools cost Y." They will weigh that out in terms of dollars, and begin to make big choices. I’m interested in the early lead on those numbers, because that will be huge in the direction the industry goes. From my personal viewpoint, accurately predicting the industry direction (in addition to staying aprised of new technology) keeps me relevant and marketable.
IMHO, if the overall system of applications are correctly designed with reusable components, the development becomes more a matter of simply assembling those components, as opposed to developing from scratch. This greatly reduces any gap in development times regardless of the language used, making it a matter of how much it costs to run it in the cloud. The numbers should be out there, if we can just get hold of them. That’s your winner, if you ask me. And I think I already know the answer…
My prediction is that PHP will last a good long time, until the next big ASP.NET competitor comes along (new flavors seem to keep coming and going while ASP.NET stays.) But until you get fully native executable code, I’m convinced that ASP.NET will run faster and use less hardware, and become the logical choice for large corporations in the end. Maybe PHP.NET? Best of both worlds? Is that even possible? idunno…
Craig you touch an interesting point regarding cloud computing. While these numbers seem to show an advantage for ASP.NET. The sheer size required to power a Win2008 server outweighs the benefits. I can run PHP on a smalll 256MB RAM cloud server. I’m doubtful I could fit 2008 in it, much less ASP.NET.
An point that’s got my full attention is Microsoft’s handling of upgrade paths. The habit of big releases will clash with the "always on" nature of cloud computing. IMHO maintaining an application is way more expensive than developing it. Microsoft has a tendency to change more things than anyone else and it will be interesting to observe how this turns out.
Gerardo, my belief is that as go the large corporations, so go the big bucks. At my former employer, one of my assigned special side projects was performance analysis for an application that had 40,000+ users. The web services alone took 5 million hits per day, just for the one app. The database I built to track 6 months’ historical information was around 540GB in size.
The point being, large corporations have large applications that require multiple servers with lots of RAM, not simple servers with 256MB RAM. Maybe you can’t fit ASP.NET on your server, but the size of the core becomes a lot less relevant with more RAM. I mean, if you have a server with 3.75 GB free RAM compared to my 3.5 GB on the same hardware, but ASP.NET turns over client requests at even 2x the rate of your system, then I get much better usage than you do, and my overall hardware costs (software costs are a separate issue) are about half as much. We’re talking fully scalable multi-server systems here, not just a little box for Joe’s pizza shop. (Sorry, Joe.
Even twice the rate is HUGE in terms of large corporate use.
You make a good point regarding the "always on" nature of cloud computing, but that’s really a separate topic of proper design and planning. Every once in a while, a city has to shut down a major highway or bridge for improvements. It’s just a fact of life. But with a good up front design, you can minimize that impact. I’m not a proponent of "big" application designs; on the contrary, I believe that the web facilitates breaking things up into more manageable modules that can be quickly updated and swapped out. Modular design is always better, reducing SDLC times all around in a variety of ways.
Honestly, if someone were to come up with fully native web apps on a free system like Linux/Apache, where the development was on par with Visual Studio, I think it would take the whole pot in a heartbeat. Free server OS, cheap dev tools, and blindingly fast to boot. But to my knowledge, no one has done that yet (at least, not well.) Until they do, I think Microsoft has the edge with the big boys. Java was designed for cross-platform (hence no native exe), .NET was designed for a single platform. If you change the approach, and design something specifically for the free OS (like, Linux?) that will provide some real competition.
Here’s a question: Why doesn’t someone step up and write a compatible CLR for Linux? I do recall that being one of the original selling points for .NET, that it could be done… Then we could develop in Visual Studio and deploy on Linux…
Craig, I agree with you in regards to the big companies. And yes in that area ASP.NET has a huge advantage. But I feel Microsoft is shooting itself in the foot by doing such great monolithic systems that depends on Windows. This also touches on the CLR for Linux question you posed.
We know from Joe’s tests that ASP.NET runs faster on Windows than PHP. But we don’t know if ASP.NET runs faster on Linux than PHP because it can’t be tested. Now its obvious that ASP.NET has to be faster than PHP. Because I’m paying for it. Right? On top of that Microsoft controls the whole stack from OS to language. So obviously it needs to achieve some advantage. I’m not trying to trash Microsoft here, just state the obvious and expected. The fact that ASP.NET is faster than PHP should be a given. Not something to be questioned, but rather quantified and determined HOW much faster. Based on this boost in performance we can then do some math and come to conclusions like you rightfully attain with your 40k+ user a day application.
But there is another side to the story. A story that unfolds when you consider the entry costs and requirements of ASP.NET. I do some sites which have considerable traffic, but obviously not nearly as much as yours. One for example might have 100 to 200 thousand page views in one day. Mostly concentrated over 5 hours. It is served by a machine with 1 Gig of RAM and a Dual Core Athlon. Lets define this a one server unit.
Now it would be great to achieve three or four times better performance by using ASP.NET. Theoretically and based on Joe’s results I could use 1/4 of a server unit. Thus a 256 Meg cloud server will suffice when factoring in ASP.NET’s performance benefits. But Windows 2008 doesn’t fit there. I’m doubtful it would fit in a 1 Gig server. So I’d probably have to allocate a 2 Gig or 4 Gig server just to have ASP.NET. That’s two or four times more than I’m paying for right now. And factoring in performance boosts, it’s eight or sixteen times more than I’d need. Do you see the relevance of the 256Meg cloud server example now?
Now, could a CLR for Linux exist? Well ask Microsoft about it. They should be able to do one. Then there is Mono. Which is a move towards achieving that. Like a lot of things Microsoft does (or promises to do), the CLR for Linux is one of those things that never happened. Thus I stick to PHP and Java. Because they support a lot more platforms than ASP.NET. And because there are not so many "big company contracts" out there, it makes no sense to commit to a technology which has benefits in large to very large deployment scenarios. If ASP.NET ran on Linux and also turned out to be faster than PHP and fit on small as well as large footprint servers I’d use it. But it doesn’t.
Gerardo, of course I see the relevance to the small app. (See "Joe’s pizza shop" in my earlier post. Sorry again, Joe.) But I feel you’re ignoring scalability here. Suppose you design your system which adequately supports your customer’s needs. But then, the customer’s business takes off, and suddenly you find yourself in need of 100x the processing capacity you had before. It isn’t in their best interest to keep your system if it creates multiple times the hardware cost. Your model makes the assumption that your customer is never going to grow. They won’t happy with you if business is trying to boom, but the app is choking off their customers. Plan for the future, don’t just accomodate the present.
Plus, a little extra memory is dirt cheap these days; for about $15, you could add a gig. So the really important factor is not the larger core footprint of ASP.NET; it’s more about the footprint used by every client session of ASP.NET vs. PHP. That becomes very important in determining cost. If PHP fits 10 times as many simultaneous requests into memory as ASP.NET, but ASP.NET actually turns them over 20 times as fast as PHP, then ASP.NET is actually saving you some serious dough in large apps, right? You just have to plunk down an extra $15 in hardware costs to get it. You seem to be complaining about the extra $15 that Microsoft is causing you to pay, while they’re actually saving you the cost of extra servers. But you don’t care about that, because you dont need them, at least not right now. Bottom line, while the 256MB solution may make sense for the short term stop-gap solution, it doesn’t make much sense to me in planning for the potential.
As for Microsoft providing something for the money (without launching into a major fight over whether or not Microsoft is the great white Satan of computer companies) I think they earn their keep in many areas, *especially* in the realm of software development. I think that without many of their employees and the blogs they maintain, the software world would be a much less innovative place. I mean, give credit where credit is due. If you look at some of the stuff they’ve been doing lately, they’re actually starting to come alive, IMHO. But I also think that the free software arena is a good thing too, because it keeps the competition alive. It keeps them working hard.
Thanks for keeping up such a great site, Joe. And send me some pizza.
@Craig
There is CLR for linux and its name is MONO .
You can write asp.net aps in visual studio and deploy on Linux !
Problem with benchmarks is, as I remember, that you must obtain permition from MS to publish frameworks performance comparisons.
Craig, I’m not ignoring scalability. But scalability goes both ways: scale down and scale up. Your argument regarding growth is very valid, but it doesn’t tell the whole story.
While it is true that a less efficient platform like PHP will create added costs if the demand grows a lot. A heavy platform like ASP.NET will create added costs while the demand is small. During the initial startup and growth phase the client will have to pay a premium it really doesn’t need to. My proposal would be higher priced than the competition’s and we don’t even know if the needs a year from now will be the same.
So I risk putting forth a more expensive initial solution. The client ends up paying more from the get go in the hopes that down the road there will be a significant saving. And we don’t even know if today’s needs will persist into the future as the company grows. Then of course there’s the whole issue of projects that wont grow. Those systems that by their nature will remain small or relatively small regardless of the company size. And then there’s the list of projects that will be used for a certain amount of time and then discarded. And then there are companies that don’t grow. By their nature they will remain small, but require certain applications. For these the added unnecessary costs would seem unacceptable.
Please don’t think I’m trying to make up excuses to not scale or look forward. I just want to point out that not everything is a super high performance application. That some solutions in php (actually many) are not high load high performance. And the added bonus provided by ASP.NET does not outweigh the costs and minimum requirements for the OS alone.
@O.S. -
Thanks for clueing me in. I should have known about MONO, there’s just so much info out there these days, you have to pick and choose from headlines what to learn about. Since you can’t publish benchmarks, maybe we can produce an open source tool that produces benchmarks on ASP.NET in general. Should be easy, since you can write the same app and deploy it both places. We just can’t publish the results, but others can generate their own sets of results. That’s in compliance, right?
Let me guess… that’s already out there too? In any case, I’m definitely going to be looking into the MONO solution. Sounds promising. Time to drag out the spare CPU and reload Linux…
@Gerardo -
If O.S. is right, and MONO is actually stable, then our discussion becomes pointless without more information. Personally, I started out where you are before, and even without MONO I would probably still try to sell ASP.NET to my customer over PHP as a long term solution that would support limitless growth. Not every client chooses the cheapest solution, especially if they know your product supports their dream of having the next mega-company. Many companies automatically throw out both the lowest and highest bids, looking at the rest. (I’ve also hired contractors before, during my 3-yr IT Manager stint. Glad to be back, frankly.)
If you truly believed in ASP.NET as a better solution, you should be able to sell it pretty easily. However, since you don’t, you are probably better off selling them PHP. You definitely have to believe in what you’re selling. And if that’s the case, then there is no amount of statistics other than up front costs that are going to change your mind. ASP.NET could be 100x faster, and it wouldn’t matter, because Microsoft is a company that actually sells their product instead of giving it away for free. You’re better off forgetting that ASP.NET even exists until that changes, which will be like… well, never.
Plus, you can always drop multiple bids in to your clients, you know. A cheap option with the PHP, and the long term solution with ASP.NET. I think you’ll be surprised at their choices when you explain the difference in terms they can understand. Me, I like the ASP.NET hosting option for small clients (usually for my acquaintances with small companies.) It takes all the worry of maintaining systems off of their hands, and sets it all up as a monthly budget line item. That’s probably the single biggest reason large corporations are moving into the cloud as well. It gets rid of a LOT of internal headaches, both technical and administrative. It seems all the systems guys that I know are all moving to places like RackSpace these days… Shared hosting makes your ASP.NET solutions much cheaper and attractive in a "host" of ways.
Craig you pretty much described the reason I propose Java solutions. I’ve looked into Mono and it is mostly C# that works well. It doesn’t feel as polished as the real product from Microsoft. Now if Microsoft actually delivered a Linux solution it would be a totally different ball game.
With Java you get the real thing on Windows, Linux and OS X and other platforms. Not some look alike which is always under legal scrutiny. If my client has a Windows infrastructure it runs. If my client uses Linux it runs too.
Now let me set this straight. It isn’t only an issue of money. I pay for my Java development tools too. And I use open source frameworks too. Which I could do with ASP.NET as well. For example Hibernate (an ORM tool) exists for Java as Hibernate and for .NET as NHibernate. I can use Hibernate to make my application talk to MySQL or MSSQL server or Oracle. It isn’t about freeloading. It’s about flexibility. Java works well on multiple platforms, ASP.NET doesn’t. It limits me to Windows. And some clients might not be willing to have a Windows solution. So what should I do? Cover .NET development and Java development or specialize in one. That’s what I mean by Microsoft shooting itself in the foot. Their lock’in strategy is turning more into a lockout effect. So yes, ASP.NET could be 100x faster and I still wouldn’t seriously consider it because a) there are scenarios were I wouldn’t even be able to apply it b) I wouldn’t dare suggest Mono as an alternate without some serious support like the one provided by Microsoft to .NET c) 100x performance boost benefits doesn’t count always (small load solutions). Like I mentioned an ASP.NET solution on Linux (sparing me the unnecessary cost of a Windows License) for a small load solutions would be great. But not possible. Not because I don’t want to. But because Microsoft doesn’t want to. It is inflexible.
I got together with a group of friends who I consider experts on their fields and would be my partners when we launch our next web project. They include various fields, to wit: lawyer, software engineer, business MBA, artists (creative artist) and sociologist (yep, we have an agenda in mind!).
We analyzed this subject from various perspectives, including, return on investment, windows of opportunities, social impact, software performance and artistic appeal (among others.) We will not release our theory of operation, for the moment, since it includes business sensitive information. But I will share one word that, to our beliefs, is key the analysis:
CROSS PLATFORM.
Enough said; I will let you guys reach your own conclusions.
Joe, it seems that I still did not succeed in making my point.
What I am trying to tell you is that you are not going to make more people from the PHP world cooperate with Microsoft if you keep promoting confrontation.
Regardless of what you say, many people look at your benchmark and say "there is Microsoft still trying to put down competition with bogus benchmarks", but they don’t bother to even come here and tell you.
Just looking at the first tests everybody can see that you picked tests that do nothing. What real world programs use loops that do nothing? What does that prove? That ASP.NET is faster in situations that never happen in real world programs?
Other than that, I already mentioned above and you seem to have ignored that to make PHP or any program at full speed in Linux, you need to compile all manually with static libraries. Static compilation makes code run 30% faster because it uses CPU direct addressing instead of relative addressing, which is needed to access code and data when using shared libraries.
I am not even going to suggest that you use opcode optimizer because I do not use them myself. Those that exist, either are closed source and cannot be audited or do not provide a good test suite with nearly 100% code coverage that assures that the optimizations do not introduce bugs. But that is a personal criteria. Anyay, for top performance, PHP opcode optimizers should be considered. Do not confuse opcode optimizers with opcode caching.
Anyway, to make a real world test running PHP on Linux, you would have to compile PHP and Apache 1.3 as static modules. Then you would be able to make comparisons that are more fair than your results.
What if you did that and the results would show that PHP running on Linux would be faster than ASP.NET on Windows? Would you still publish the results in your blog? I suspect you wouldn’t because that would make the products of your employer look bad and than it would upset your employer.
So, if you do not want to upset the PHP community, why bother publishing benchmarks that make PHP look bad using a synthetic built only in conditions that favor Microsoft products?
There is no problem in the PHP community. They see your results, do not give them credit, and most the PHP developers simply would not give you any feedback because your benchmarks were not made with PHP running with real world setups, so they never can be fair.
So, why I am bothering to tell you that if I do not give credit to your benchmarks? The problem lies in the fact that your colleagues of Microsoft Open Source labs have been working hard in the latest years to get closer of PHP and other Open Source projects to get cooperation in making Microsoft products run better with PHP and other projects.
When you publish these synthetic benchmarks you are passing the idea that all these efforts of Microsoft OSS Labs are just a bait to get attention to Microsoft products that compete with PHP and convert PHP and Open Source users into users of Microsoft paid products.
If baiting was not your intention and you agree with me that is a bad image that Microsoft is passing, I suggest that you avoid promoting confrontation publishing synthetic benchmarks.
You can still do your tests and pass them directly to the PHP developers that care in the php-internals list. That was the right thing to do.
If you disagree with me, that is OK. In that case it seems that you are really not interested in feedback, but rather stay in Microsoft dark ages when "Linux is a cancer" was your motto to fight Open Source.
Luis, I cannot comment on that Entity Framework because really do not know it, as I never used it.
In the PHP world there many ORM solutions, probably too many that copy each other deficiencies.
I use Metastorage. That is an ORM tool that I started developing in 2002 and is very mature.
It implements a true ORM (Object Relational Mapping) because it departs from the object model and generates classes of objects that map the objects into data in relational database tables.
Many so called ORM tools are in reality ROM (Relational-Object Mapping) tools because they require that you first create a database with tables so they map data into table records, which does not seem a very productive approach because you end up having doing the object-relational mapping exercises in your mind, rather than making a reliable tool do that for you. More on that subject in this article:
http://www.phpclasses.org/…/82-PHP-ObjectRe
Anyway, Metastorage made me very productive since when I started developing it in 2002. It generates self contained code. This means that the generated code does not require special frameworks to perform the actual Object-Relational mapping.
The generated code is very compact, well formatted and even commented, just in case you want to peak and try to understand what it does. It is the code of the quality of the same I would write if I had to do it manually. The advantage is that Metastorage generates that code from a classes model definition, just in a few seconds. So, it makes me very, very productive.
It provides a nice Web interface so you do not have to wonder how to generate your ORM classes struggling with command line options.
It supports many relational databases like MySQL, PostgreSQL, Oracle, Microsoft SQL Server, SQLite, etc..
For more information, you may look here.
http://www.metastorage.net/
As for who is pushing the envelope, I don’t know for sure, but it seems in many aspects Microsoft is learning from the Open Source world, even if that takes them years to realize what makes developers more productive.
Well Manual, I guess we’ll just have to agree to disagree.
I, and hundreds of other folks understand why these tests are staticically significant.
If they are irrelevant to you, or you don’t understand their impact, that’s OK.
But they are what they are. Passion does not change fact – only opinion.
Joe could you or one of the hundreds of other folks that understand the statistical significance of these results explain to me what I asked in my post. Why does doing more take less? Why does a loop with a function call take less to execute than an empty loop?
How many times did you run a particular test to obtain each value in your table? Once? Twice? 100 times? How did you spread out these test so the particular load on the machine at any one given moment didn’t affect the results? If you did more than one test for each cell value, would you care to publish info like standard deviation? The tests I’ve run do have quite a bit of variance from one to the next. That is why I decided to average out 200 tests. And even then I get variations from running it at one time or another. The ideal would be to have maybe 10 or 20 two hundred test runs spread out randomly along five or ten minutes.
I also have to agree with Manuel (with an e) when he mentions "real life" applications. When you start compounding your results to build a "real life app" things begin to slip from the ideal. Take for example your String & Integer Test and your Database Access. Lets be generous and par the worst String/Integer Test from PHP vs the best in ASP.NET. That would be PHP 5.2 @ Ubuntu vs ASP.NET (1.641 vs 0.617). If we subtract and divide by 600000 (number of iterations) we obtain 0.000001707. This is the amount of time PHP slips behind ASP.NET on every iteration.
Now lets take PHP 5.2 with MySQL vs ASP.NET with MSSQL. Thats 0.251 for PHP and 0.295 for ASP.NET. This time ASP comes in a bit slower, but both cells are green so I’m still taking a good value for ASP’s performance. We subtract and divide by 1000 (0.295-0.251)/1000. We obtain 0.000044. This is the amount of time ASP.NET slips behind PHP on every iteration.
Now dividing 0.000044 by 0.000001707 we get 25.776 (aprox 26). What does this mean? Well it means that ASP.NET is faster doing String and Integer operations. But one single database access on its part sets it behind PHP 26 iterations. All the work done to get ahead is lost in one single database access. How many queries does your average page consume? Each one is setting ASP.NET behind PHP’s performance by 26 String/Integer tests.
This of course is a bit of a bogus statement on my part too. Because your test does a connect on every iterations. This usually happens only once in a page’s request and it is usually cached in bigger applications by means of a connection pool. But the example stands. On an application I’m working on. A great deal of performance boost was done by caching info from the database. Calculation times for a big data set were dropped from 10 minutes to 1 minute 30 seconds by first caching then calculating then writing results. Out of the 1 minute 30 seconds 980 milliseconds are actual calculation time and the remaining 1 minute 29 seconds are database access. According to your results. Switching to ASP.NET would give me a 70% boost. Bringing 980 milliseconds to about 370 milliseconds. Achieving an overall performance gain of 1.7% (610 ms). Hardly enough to justify porting the tool to another language. Much less one that works on only one OS.
Sure,
I can’t answer "rhetoric" questions, but …..
Each test was run 5 times consecutlvly.
The "real life" argument is simply emotions crap becuase a few folks are emotionally offended that PHP did not fair better.
The REAL LIFE value is that if creating a database connection, creating an object instance, calling a method, etc. has additional overhead (read IS SLOWER) than THAT MATTERS !
Complaining that "calling an empty loop is not REAL LIFE" is simply choosing to us an excuse to ignore the data.
In any event, if YOU don’t see the value, fine – but your lack of acceptance does not netage my results or invalidate MY OPINION about their meaning.
Rather than beating a deap horse a complaining about the results.
How about a more professional and "real life" responce ?
1.) Make PHP Core Execution Faster !
2.) Accept the results as "info" that doesn’t effect your choice of PHP
3.) Accept the tests but declaure that, while other may find them usefull, you do not (and stop repeating the same comments.)
4.) Hand engineer tests that make PHP appear faster so that you can be happy.
5.) Stop complaining that my tests don’t meet your undefined "real world" critera and write a clear and complete description of what is necessary to comply with your "real world requirements" and then write some benchamarks of your own to see which is faster under those conditions.
6.) Wait for me to do more testing and complain about THAT work.
7.) Read the rest of the comments to understand that these perf results need not effect your platform choice.
It’s just sour grapes to continue to complain about this prrof that ASP.NET (C#) core execution is faster – and the results ARE statistically significant.
This is only the first peice of performance profiling and it revealed usefull results for the product developers that wanted ty listed. (Several teams on the Microsoft side want our stuff to perform faster and are using this data to make changes.)
Nobody is gonna stop using WordPress because of these results.
The "perfoamce" difference between a Toyota and a Ferrari does matter very much when the speed limit is 55 !
@Luis, you said it all, my friend. Thanks. Microsoft (et al, self included) should take heed of that one statement.
In a day where we constantly talk about abstraction in our applications, we must remember to consider the issue of the application being abstracted from the OS itself.
What I would like to see in the future are two things: 1) Microsoft creating a free (or cheap) stripped down version of its server software that removes all the extra unnecessary stuff, just for running straight HTTP and .NET apps for the small shop. Remove the enterprise-class features. (Tall order, I know, but doable…) and 2) Better .NET portability. I think if Microsoft provided those two things, they would regain a big part of the market share, and fairly so. Plus, the performance comparison tests would have a bit more meaning.
But I digress. This page topic is about a performance comparison. I will refrain from further off-topic discussion.
Joe, please let me clarify that I do understand your tests and the relevance of empty loops and noop function calls. I also believe that ASP.NET is faster than PHP. After all it is a paid for product. I would expect it to be faster. The question that remains is how much faster.
That is why I’m so persistent on the question I put forward. Which you still haven’t answered by the way. Why does the loop containing the function call run faster than the same loop without the function call? It should run a bit slower. Just like the other two cases.
If the issue is to determine performance values then it is important that these values be consistent. That adds up to the value of your information. The issue of the loop that takes less to run needs to be addressed and a reason for that found. Otherwise it demerits the value of your information. It won’t make ASP.NET run slower than PHP. We all know it runs faster. But for the sake of your credibility they need to be consistent. Because if that is inconsistent it raises the question as to what else might also be so.
Dude,
What does the tfact that you don’t understand WHY result happens have to do with my credibility ??????
Since I don’t have the CLR source code on my desktop – I can’t tell you. Somce clver person on the CLR team figured out how to do an optimization based on the detectable criteria of the code.
This is VERY common in compiler implementation.
How does knowing specifically how a developers was able to code an internal optimization matter to the results.
I renew my contention that you are simply posting FUD becuase you don’t like that fact that PHP is slower.
Are you a core contributer PHP? Perhaps you can make PHP core execution faster.
On the other hand, since you keep saying the difference doesn’t matter, I’m confused at why you continue posting the same argumentsd here.
Well Joe, it seems we are finally moving towards finding an answer. My next question would then be. Why didn’t that clever person also include that optimization for while loops? It would be great to have that performance increase while being able to test for a condition prior to the block rather than after it.
I’m also interested in figuring out what that optimization is. Particularly since according to you it is very common. If that optimization consists of the compiler detecting that there is nothing to do and actually removes the call. Then there isn’t much of a test being done is there? As loop with empty function call falls back to just an empty loop.
For example in my case I decompiled the Java test code to see if the function call was still there and it was. So I’m sure it was being called. The difference in performance between Linux and Windows turned out to be the usage of the -server flag. Which uses a server JVM available in the JDK and that was not being used on Windows, but was being used on Linux. With that flag on, the performance under Windows was similar to Linux and not 140 times slower as it appeared initially.
Now had I just sat back and said Windows runs Java code 140 times slower than Linux I’d be posting FUD. But I didn’t. I went out and took the time to answer very basic discrepancies in my results.
I’d also like to add that I don’t care much if PHP is slower. I don’t use PHP a lot anymore. Aside from Drupal, WordPress and similar things. I believe we’re having a conversation here and everyone is exposing their point of view. I’ve continued to post the same arguments because they hadn’t been addressed until now. And to be honest we’re just beginning to scratch the surface of the matter.
If I’m persistent is because I believe something like PHP.NET could be attainable. After all it is the objective of CLR right? You could have PHP.NET and have php code converted into common runtime. But when you say I FUD and refuse to further research and explain discrepancies in your data. Then I become doubtful of the objective of this blog. When you don’t respond to my questions (which I believe are well based on your data) and you call me a FUDster. Then I’m more inclined to understand Manuel’s position that you’re trying to promote confrontation rather than reach understanding.
Let me put it dead simple. If you go up to a decision maker with this data. And he or she asks you the same questions I just did (which are pretty obvious by just scrolling down the column of YOUR data). I’d expect you to have an answer and not just call him or her a FUDster. To do what you just did would get you nowhere but out the door. Because in REAL LIFE people are going to ask these and tougher questions before putting REAL LIFE money on your solution.
LOL – please re-read my last reponse
Idem
@Joe -
Don’t bow to the pressure here. IMHO, Gerardo is riled up over nonsense.
@Gerardo -
If you want to understand the reasons that one thing is slower than another, you must understand the compiler techniques employed by both platforms. In this case, I’m guessing that the compiler is probably optimizing the function call better than the loop. But that’s really irrelevant. What’s relevant is the fact that it does what it does. Joe’s not to blame for that, nor does he likely *care* that the variance exists, nor can he personally fix that variance in the tested versions of the platforms. It’s a little like me telling you my dog runs faster than my cat, and you demanding that I tell you why my dog pulls her left knee higher than the right when she runs. How the heck should I know? It just is what it is, an unsolved mystery that has no significant bearing on the fact that my dog runs faster than my cat. If you think the answer will provide you with a significant return on your investment, consult a dog specialist, pay tons of money and go through the process to find out. In the end, you will have spent tons of time and money without getting anything truly useful in return.
There is very little value in pursuing why one technique is microscopically faster than another, unless the answer is readily available. The likely explanation has been given, and any further investigation on Joe’s part would not be fruitful. It’s obviously due to compiler optimization, be that reasonable or not. If it’s that big a deal to you, post it to the right forum to get the answer. Who knows, maybe it will save you millions. Um, yeah.
And in light of the fact that you went round about with me over the fact that the performance difference didn’t affect your decision anyway, why the HOOHAH do you care? You told me that it was all about the fact that Microsoft cost you more and you couldn’t land contracts with it. You’re just being beligerent. Make up your mind.
Peace out.
I am a .Net developer and I recognize that PHP is faster in many things against .Net, including database access, I know I can improve performace with less Server-side code and more Client-side/Ajax (but I will not do it, reminds me classic ASP).
But I love several little things about .net apps:
- easy to deploy, I have portals and enterprise aplications with less than 20MB and little number of files (I’ve seen Joomla, OS-Commerce and Moodle with 1000 files and more than 250MB);
- easy too maintain (less files, and compiled libraries);
- PHP remindes me Classic ASP (I hate classic ASP, done for many years).
Craig, I think you’re mixing too many things I said. First of all performance difference didn’t affect my decision as long as there isn’t a cross platform .NET solution supported by Microsoft. That doesn’t mean I don’t care about the performance difference. If tomorrow Microsoft were to launch a Linux ASP.NET or like I mentioned a PHP.NET. I’d be interested. Thus it is good to keep up with things like these. In Joe’s tests there is also information regarding performance of PHP under Windows and Linux. Information which is relevant to me right now.
I hope this clarifies things and makes it clear that it isn’t all about the fact that Microsoft costs me more. Sure, Microsoft would cost me more in some cases. For example small sites or low traffic solutions which are well served by equipment not fit to handle Windows 2008 or that don’t justify the license costs of the OS itself. On the other hand you have a progressive call system I’m working on now. My client runs the VoIP solution on Asterisk and Linux. After you factor in the Linux license, the hardware that is required to manage all those calls, the codecs for voice compression, expansion cards, telephones, etc. You come up with quite a price tag. License cost isn’t so much an issue here as their expertise on Linux. I can’t just come and say hey use Windows because ASP.NET only runs on it. I use a tool that runs on Linux. In this particular case Java.
Getting back to Joe’s results. I think we are both in agreement that what’s relevant is the fact that it does what it does. And what ASP.NET does is run faster. We know that. We expect that because its a paid for product. The importance of answering the questions I’ve made is not to conclude that ASP.NET runs slower than PHP nor to once again conclude that ASP.NET runs faster than PHP. That’s settled.
The importance is to validate the results and verify the methodology used to arrive at those results. ASP.NET isn’t going to run any faster or slower because we figure out what is happening. But Joe’s results will sure look a lot better if those doubts are cleared out. Instead of being brushed under the rug like you’re doing.
My interest is in the PHP on Linux and Windows results. Doubts in one piece of data cast doubt on the rest. I’m interested in the rest of the data. Thus I’m interested in the discrepancies I mentioned.
Personally I think you’re doing Joe a disservice in using words like "guessing", "probably" or that he just probably doesn’t care. Makes it look like you’re brushing the issue under the rug. I think Joe has gone through a lot of trouble to get this info. I questioning things that if answered would harden his information significantly and benefit him. But that makes me "the bad guy". How’s that?
Hi y’all! I thought within some days this discussion was going to die. It hasn’t; as a matter of fact, I thinks it’s defining an epoch. Who would have thought 15 years before that PHP had a chance to compete with a Microsoft baby?
Anyway, I will share with you part of the performance analysis my friends and I did. We replicated Joe’s tests with only one difference: We put all PHP code in functions! Performance was way closer than Joe’s tests, although ASP.NET remained just a tiny little bit fast. If you know a little bit about the Zend Engine, you will understand why this is true. If you don’t believe bit then I suggest you study the garbage collection mechanism of PHP. But to summarize it: PHP functions DO get compiled to opcode. So when a user sends a request all subsequent requests will be served from opcode. Do you want a more permanent form of opcode (i.e. – one that doesnt destroy at garbage collection time): there is a very simple and light solution, that I believe is underrated: CacheLite (part of the PEAR libraries). It’s so simple to use, that I… anyway.
@Gerardo -
1) "Your ASP code with no method call (do loop) seems to be taking longer than the same loop doing a method call."
Simple. When a compiler sees a loop that performs a straight call inside a loop, it would likely optimize to make that call x number of times, instead of performing all of the math and memory operations on the loop control variables. On the other hand, when it compiles an empty loop, it has no choice but to actually make the loop x number of times (to preserve the code), making index adjustments (x number of math operations and memory updates) and loop test comparisons all along the way. That’s the likely explanation for the discrepancy. I surmise that if you changed the function call to actually pass the value of the loop index, it would be slower than the plain loop, because it would prevent the compiler optimization, and force it to evaluate the loop control as well. If Joe is willing, perhaps he could add that one in for comparison. (I have already requested it at the link he gave us.)
2) "Interesting how the same jdk can produce such different results on two OSes."
Equally interesting is that the Java Virtual Machine for *both* OSes is not produced by Microsoft, and actual code execution of a for loop by the JVM will not have anything to do with the OS itself. More on this in a moment.
3) "How many tests are you averaging to obtain those results?… Notice how on Linux the time drops quickly while it remains practically the same on Windows"
Why didn’t the JVM optimize the code on the first run, and why is there no optimization on the Windows platform? That seems problematic to me. I don’t know about you, but in real world application, my users don’t run their reports, then run them again to see if they get them faster the second time. It’s the first time that counts, unless you’re writing a game or possibly some cryptography stuff with a lot of repetition. And again, note that Microsoft does not produce the JVM. There would seem to be a discrepancy between the JVM on each platform.
4) "I’d appreciate it if you could do repeated runs of these tests and then average out the result."
And what, exactly would this prove, other than restating the fact that there is most definitely a discrepancy between the JVM on each platform?
This is definitely not an OS issue. Java is compiled to byte code and then executed by the JVM on the processor. Binary instructions are not somehow passed through the OS first before it goes to the CPU, unless they are calls to functions offered by the OS itself (which would be insane for a simple for loop.) The JVM is a native executable that gets direct processor utilization just like every other native executable. Windows itself should not be a large factor, except for CPU contention with other processes. Did you happen to note the CPU utilization of each platform and application during your tests? Did each test reach 99% utilization?
If we kindly assume that the JVMs are correctly optimized for both platforms, one could easily assume that for that large of a variance on a simple loop, there must be a hardware issue. Could you post the hardware configuration information for both the Linux machine you used as well as the Windows machine? I’m assuming you did some sort of dual-boot configuration so that the hardware comparison is fair, as opposed to running one virtually. Also, this wouldn’t happen to be the 256MB machine you mentioned, would it? If so, please refer to previous discussion on OS footprint. There has to be sufficient physical RAM free so that both processes run in-memory vs. paging for a fair test.
Craig, the issue between Windows and Linux was solved and published on Aug 24. It was related to the -server flag. When using that on Windows performance is similar. Like I commented on that day’s post. I did check system loads and actually decompiled the code to see if there were some changes. There were none.
Maybe Joe could check the decompiled code to see if there is one or many calls like you mention. Then compare decompiled code for the for loop, while loop and do while loop to see if there are differences.
Regarding your comment on the reports. Reports are usually generated quite often on an application. The dataset will change, but the code that actually renders them doesn’t. So an optimization in the loop that renders table rows is overall beneficial regardless of the actual values of the row cells. Thus you don’t need to run the same report twice to see a benefit. Unless of course you only generate a report twice in the whole runtime of the application.