High-load Website (WordPress) Optimization : IlmuKomputer.com
Mr. Romi, founder of IlmuKomputer.com (IKC), yesterday asked me to help optimize this website. A bit about IlmuKomputer.com, it means “Computer Knowledge”, and contains a lot (and I mean it) free high quality computer tutorials.
As you can easily guess, the website is very popular. On peak hours, it’ll usually become overloaded, and will become unresponsive.
I’m only too happy if I can be of assistance to IKC’s team in their good cause. So I started working on it with help from one of my staff, Yopi.
Turned out that what we’ll be doing will be very different with what most others do. Anyway, IKC is a very popular website (and “slashdotted” daily, by leechers), so what works for most others doesn’t work for us.
The Bottlenecks
A bit of background - IKC uses WordPress as its CMS. It’s a very nice CMS, and makes your life easier. I’ve used WP myself since version 1.5.x. However, being database-based, there are a lot of points within its a WP-based infrastructure which can become a potential bottleneck. So if your website started to become popular with this CMS, you will need to start optimizing it.
After examining the situation for a while, it’s clear that MySQL was THE bottleneck. Output of top shows it using at least 8 times of CPU time than other service. Mr. Romi also told me how it kept on falling down on peak time.
Apache (and PHP, since it’s compiled as Apache module) is the next one; with each of its process using more than 10 MB of RAM. This may seem insignificant at first, but multiply that by (potentially) 150 processes - and you’ve got quite a memory hogger here.
Also CPU-usage wise; I’m quite surprised to see that each incoming request will cause the particular process’s CPU usage to spike to more than 50%.
Initial actions
I asked Mr. Romi to increase the size of MySQL’s internal cache size. He did, but the machine still fell down in daily basis.
He has also implemented caching on the app server (PHP) by way of wp-cache plugin. Still no joy too.
The Edge
I decided that we need to go straight to the “edge”, and stop the load there.
I proposed that I setup Squid in HTTP Acceleration mode. This way, most of the requests won’t even touch Apache, much less MySQL. Squid will bear most of the load, but since it’s very efficient, it should be able help a lot in making the website perform better.
Since I’ve got a few things to do myself, I asked Yopi to setup Squid in our test machine.
I just gave him pointers now and then, yet he managed to finish testing the setup and implement it in IKC’s server in just about 3.5 hours.
Then I showed him “tail -f /log/squid/access.log”, and we watched in amazement on how quickly the TCP_MISS lines are changing to TCP_HITs.
After about 12 hours, I increased the cache_mem size, and the TCP_HITs are slowly changing to TCP_MEM_HITs.
The result
Squid is working as we expected.
Average server load dropped from 30% plus to about 3%. While squid’s CPU usage increased from 0% to an average of only 2%. A very nice trade off.
After about a month, I checked the website’s logfiles, and saw some very nice numbers — traffic to IlmuKomputer.com has doubled ! Needless to say, Mr. Romi is very happy with it.
I also found that everyday there will be people downloading the contents using crawler software - such as Teleport Pro, wget, etc. I asked Mr. Romi if he’s got problem with it, and he says no. It is his mission to spread knowledge for free after all. So I let these leechers alone.
Come to think of it, it’s possible that these crawlers are the ones causing IKC server to fell down at peak hours. Example, Teleport Pro is able to download 10 links simultaneously at the same time. Then once any of it is finished, it will instantly start download the next one. When all 10 downloads access the database, and many crawlers at the same time, not many servers will be able to stand up to it. It’s like being machine gunned wearing just a simple leather cloth. If you have had the experience of having your website linked from Slashdot or Digg, you’ll understand what I’m talking about.
In this case, squid acted as a thick titanium armor, and taking most of the hits for your server. I suspect now the number of crawlers has increased than before, but it shouldn’t be a problem.
MySQL is a bit strange though. Sometimes its CPU usage can be as high as 160%. Thankfully this is very rare, so it’s probably just some internal clean-up routine.
One day, after happily watching the low load on the server for a while, suddenly everything froze. Even my SSH connection. Attempts to reconnect to the server failed.
After a while, I was finally able to connect again. Looking around, I noticed there’s some sort of bandwidth limiter daemon running on the server. After consulting with Mr. Romi, I killed it. The problem stopped.
Happy ending ?
I’m still monitoring the server as we speak for glitches. For example, squid seem to hang from time to time. This can be caused by anything from bad memory to problem with specific hardware configuration; so for now I’ve setup a cronjob which will restart it in certain intervals.
It seems to help, so I can troubleshoot the problem in peace.
Anyway, I’m sure that with the increased availability, even more people will visit the website (Ed: confirmed!). Then at some time in the future, we may find the server overloaded again.
In that case, there are still many things which we can do to keep IKC up & running in just one server :
- Coral-ize internal links : Coral is a global cache with servers all over the world. It has proven to help people with overloaded servers to lighten their load (when slashdotted, digged, etc). With the Coralize plugin, all of your internal links will point to its Coral cache.
Actually, for most people, this may be the easiest and the best step they can do. I can setup Squid because IKC has its own dedicated server. Not everyone does, I personally also own a (shared) webhosting account. Coral CDN (Content Distribution Network) is a very nice & easy solution to us. It’s rarely mentioned though, so here you go.
If you’re not using WordPress, you can still utilize Coral CDN easily ! Just append .nyud.net:8080 to your links. For example, if you access http://harry.sufehmi.com.nyud.net:8080, you’ll actually access a Coral server, serving a copy of my website from its cache.
I did say that it’s very easy, didn’t I ?
- RAM Upgrade : This will enable Squid to have bigger memory cache size, therefore increasing its effectiveness significantly.
- Roundrobin Edge servers : If the load is so high that even Squid is overwhelmed by it, then we can implement a cluster of Edge servers. People can volunteer their servers and have it act as the edge server for IlmuKomputer.com.
The incoming requests are spread over the edge servers by way of Roundrobin DNS feature. It’s not the best way to do it, but it’s very easy and the cost is almost nothing.
- Use lighttpd : Apache is a rather heavy webserver. I personally like its (amazing) flexibility (there’s a reason why it’s called the Swiss Army Knife of Webserver), but at times you’ll need something else. From my experience, lighttpd + fastcgi is very nice alternative to Apache + PHP. The features are now quite similar to Apache’s, but it’s much more lightweight. Its community is also quite helpful and happy to help a newbie within reasons. Recommended.
- And many other ways
Last, we’d like to say thanks to Mr. Romi for giving us the opportunity, it was very interesting ! Hope IKC will become even more successful in the future, therefore benefitting even more people. Well done pak.
March 8th, 2007 16:45
Hmmm,
I’ve been outside of web programming and optimization for quite a while, but I’d say using cache would be effective in this case. Either hard cache (generate the pages) or just cache it in the RAM. So you don’t need big traffic to MySQL. What I see needed to update is the page statistic (”Halaman ini dibaca x kali”), and this can be accomplished using ajax. So the page content is grabbed from cache, and the statistic is updated separately using ajax.
If the code is complex then the op-code cache helps alot. I remember there were some op-code cache for PHP out there, but don’t know whether they are still in development or not, namely turkmmcache, apc, etc.
How’s that sound?
March 8th, 2007 16:50
One more thing, downloadable files should be separated from the main server. Using small web server like thttpd would help a lot.
March 8th, 2007 19:07
try to use memcached
March 8th, 2007 19:15
try eaccelerator, its fork of turkmmcache
http://eaccelerator.net/
March 8th, 2007 19:34
Hi all, thanks for the suggestions. A bit more info, Pak Romi has tried eacclerator, but it doesn’t seem to work.
At the moment, Squid is also caching downloadable files, so most of the downloads at the moment are from squid (not apache).
.
Anyway, Apache’s performance was so bad, I decided to skip optimizing it altogether, and just try to stop as much traffic as possible from reaching it.
.
Now that the server has stopped falling down, then I can start looking to optimize Apache. However, this may not be possible because it’s tied to CPanel.
We may have to compile Apache ourselves to make it fully optimized.
.
For the time being though, Squid is holding up very well on its own. This gives us ample time to experiment with other solutions.
March 9th, 2007 10:28
Waduh, pake bahasa Inggris neh, Tapi yang bahasa Indonesia di blognya mas Romi Satriyo aku udah tahu kok
March 9th, 2007 18:45
@harry: well, it’s implemented to our intranet server for about 2 years ago till now.
It’s only PIII 700Mhz, with SDRAM 128, using minimal CentOS and served about 200 pc all day. Thx to squid for optimizing my server
anyway, have you optimized your apache? it could be optimized on httpd.conf
@saya: mr. mdamt, thx a lot for your tricks.
March 11th, 2007 09:41
Thanks banget om Harry, sudah nolongin dan langsung turun tangan
Seminggu ke depan ini kita coba dulu lihat kondisi. Aku server cukup stabil dan kondisi CPU dan Memory terjaga. Paling tidak untuk sementara sudah lumayan, sampai kita pikirkan lagi strategi lain. Sekali lagi, thanks.
March 14th, 2007 19:42
Keren emang om harry.sufehmi.com
March 15th, 2007 09:57
Isn’t ‘Ilmu Komputer’ should be translated into ‘Computer Science’?
March 15th, 2007 14:06
Very inspiring..! thx mas Harry
March 15th, 2007 14:51
Really nice.., but i still wondering, is WP strong to be used as a portal CMS.. I’ve wrote my doubt about this on my web..
May 7th, 2007 11:06
[...] Harry Sufehmi sudah menulis lengkap tentang penggunaan SQUID untuk server traffic tinggi di situsnya. Lengkap dengan implementasi untuk [...]
December 3rd, 2007 17:41
wordpress, php, and squid - they don’t work along well
.
Here’s an excerpt from a report made for one of my client :
.
.
Now their website can handle hundreds of requests per second without breaking a sweat.
.
Hope it helps someone.
February 1st, 2008 18:39
A very nice article for the Wordpress Comunity. Implementation like you said on the squid +Php and mysql with the Appache make wonders.
Thanks for this Tip
February 6th, 2008 02:14
Thanks for this tip.
February 6th, 2008 15:45
I agree with dikshie to use memcached.
http://maisonbisson.com/blog/post/11489/memcached-and-wordpress
And I suggest to use Lucene too for searching. But IKC team must modify the WP code to work with lucene.
Here information about lucene.
http://lucene.apache.org/
February 7th, 2008 16:28
@Dolly - problem is, on very high-load, even the Apache itself become the bottleneck.
.
Then the request will be passed by (already over-loaded) Apache to PHP.
.
Do you think memcached will be able to be of any use in this kind of scenario ?
.
About Lucene, it’s a great search engine. But, again, we need to understand our requirements first, and then use only the appropriate tools.
.
Sometimes, just using Google searchbox can be enough already.
.
Hope it explains it.
February 8th, 2008 09:04
@sufehmi
sorry, I think the bottleneck is MySQL only. So I suggest to use memcached for cache and use lucene to search. Its can reduce the MySQL load.
February 8th, 2008 10:43
Hi Dolly, truth to be told - Apache/PHP is not very scalable. In many cases, they can only serve about 2 - 5 requests per second.
.
This is still way better than others, for example, Ruby. But still, when your website are very popular and got bogged down by PHP, you need a solution.
.
So yes, we need to first find out where the bottleneck is. Only then we can find the right solution.
.
In your case (overloaded MySQL), I think indeed memcached can be of help.
.
Thanks.
March 15th, 2008 03:02
I second the memcache it’s probably your best bet! IMO
March 15th, 2008 03:04
Also i saw another site that was quite helpful too you may want to check it out!
Free Personal Webhosting free with no ads
Let me know what you think
March 25th, 2008 23:11
hi
they are very usefull informations . thanks very much
March 26th, 2008 06:07
Hi There,
Just came by your site searching for more infomation about web traffic info, and I found some great info here
I`m working on building the best! Viral Traffic site now so I do a lot of reviews all over the net.
Have A Great Day.
Regards,
Jack L
Viral Traffic Center.com
April 19th, 2008 21:36
Hi There,
Apache isn’t as bad you think. The performance of Apache 2.2 is excellent its super scalable thanks to enhanced multithreading and non-blocking IO support. A friend of mine switched a high traffic tech blog running Litespeed + LSAPI + XCache to a carefully configured Apache 2.2 + FastCGI + XCache and surprisingly, the performance is on par!
Before runing litespeed, the site ran on Apache 1.3 and died every now and then. Btw the server is a dual harpertown with 8GB of RAM.
April 29th, 2008 18:18
apache is a good soft. I’m using it for years!
May 1st, 2008 18:52
apache is a good soft. I’m using it for years!
May 4th, 2008 11:50
Nice article. Yeah, I’ve had a few issues with Wordpress on a few high traffic sites, especially sites I had a lot of plugins on. So, if you’re running on a shared host, optimization is really important.
May 4th, 2008 22:03
Great article. Thanks.
May 6th, 2008 09:52
[...] Harry Sufehmi sudah menulis lengkap tentang penggunaan SQUID untuk server traffic tinggi di situsnya. Lengkap dengan implementasi untuk IlmuKomputer.Com Softwarenya sendiri bisa [...]
May 10th, 2008 23:32
Thanks for this article!!!!
May 23rd, 2008 16:39
Wow…what a long article. I agree with the above comment the “catch” is the point. Thank you for your article.
May 30th, 2008 22:09
Would like more information on your product
June 25th, 2008 09:46
I’ve been working for a long time with worpress and I see no problem in dealing with php.
June 29th, 2008 19:19
Hello!
I have used a plugin before, and it converts all the wp pages on the fly to html, and that way the server load is DRASTICALLY lower, I have looked for the plugin, but don’t remember the name, i’ll check at my home laptop because that is where I have the login details for that blog!
July 5th, 2008 18:53
Wow! A 27% drop is very significant.
July 7th, 2008 22:34
27% is big indeed
July 11th, 2008 05:11
That was some great information and case study! I hope I have to use it someday when I have a lot of traffic on my website! Thanks
July 19th, 2008 19:09
Hi. I want to say to you somethings about optimization. Optimization or optimality is a term that may refer to:
Optimization (mathematics), trying to find maxima and minima of a functionOptimization (computer science), improving a system to reduce runtime, bandwidth, memory requirements, or other property of a system; in particularCompiler optimization, improving the performance or efficiency of compiled codeSearch engine optimization, in internet marketing, methodologies aimed at improving the ranking of a website in search engine listingsProcess optimization, in business and engineering, methodologies for improving the efficiency of a production processProduct
July 24th, 2008 16:12
Thanks for the article.
August 1st, 2008 18:26
I always find the main bottle neck is CPU, you cant throw enough at it on RS 3000
August 3rd, 2008 16:07
Artikel yang cukup bagus!!
August 9th, 2008 00:09
[...] Harry Sufehmi sudah menulis lengkap tentang penggunaan SQUID untuk server traffic tinggi di situsnya. Lengkap dengan implementasi untuk IlmuKomputer.Com Softwarenya sendiri bisa dipelajari [...]
August 25th, 2008 14:37
Great article. Thanks.
August 26th, 2008 22:58
Thanks for these tips! Good to know how to optimize the server!
August 31st, 2008 23:16
Thanks for writing about this, these tips are very interesting! I hope I need to use them someday! That would mean my website has a LOT of traffic! Hehe
September 3rd, 2008 13:40
Nice posted! But I am running on a shared host, So somebody suggested optimization is really important. why is so? If its just like improving a system to reduce runtime, bandwidth, memory requirements, or other property of a system. Well I agree but to get the Traffic do there is not any other method?
I even agree with efficiency of production, but let you know ever phase is not same all time. so more than efficiency an consistence is important. LOT of traffic come surely……!
September 3rd, 2008 13:43
HEY, I forget to say thanks
this I feel important, to say about lovely post!
September 9th, 2008 20:59
Thanks for this. It’s a really steep learning curve for me. Recently lost job and trying to make a living online. All these articles are really useful. There seems to be a lot of crap out there. Thanks for something useful for a change
September 12th, 2008 23:00
Hey thanks for this! My webhost sent me a message because I was using up too much resources on my shared server, this will help me for sure!
September 13th, 2008 01:18
Hey man! THanks for these tips, they really help get the load down on the server!
September 17th, 2008 22:01
Thanks for these really useful tips. Bookmarked, and I will be back!!
September 18th, 2008 16:27
Thanks for these tips! Good to know how to optimize the server server
September 19th, 2008 13:10
Thanks for these tips! Good to know how to optimize the server server
September 19th, 2008 18:30
Thank you for the good tip!
September 20th, 2008 12:04
[...] public links >> optimalisasi Comment on High-load Website (WordPress) Optimization :… Saved by tupacsPORN on Thu 18-9-2008 Perlu Optimalisasi Civil Society dan Parpol yang Berkarakter [...]
September 26th, 2008 08:01
Squid is the man!!
I think it is the best proxy. With squid you can improve your response times because it caches and reuses frequently-requested web pages. While squid is answering the request your web server will be happier and your MySQL even more happier. That’s why you see such great performance improvement.
I’m happy you managed to fix it. Nice blog BTW. And thanks for allow me to post in your blog.
October 7th, 2008 00:23
[...] Harry Sufehmi sudah menulis lengkap tentang penggunaan SQUID untuk server traffic tinggi di situsnya. Lengkap dengan implementasi untuk [...]
October 8th, 2008 13:39
This blog Is very informative , I am really pleased to post my comment on this blog . It helped me with ocean of knowledge so I really belive you will do much better in the future . Good job web master .
October 12th, 2008 02:08
What a great resource this post is! I’ve noticed a real slowdown with two of my wp sites, so I’m going to go through these potential solutions one by one and see if I can nail it down. Cheers!