Mr. Romi, founder of IlmuKomputer.com (IKC), yesterday asked me to help optimize this website. A bit about IlmuKomputer.com, it means “Computer Knowledge”, and contains a lot (and I mean it) free high quality computer tutorials.
As you can easily guess, the website is very popular. On peak hours, it’ll usually become overloaded, and will become unresponsive.

I’m only too happy if I can be of assistance to IKC’s team in their good cause. So I started working on it with help from one of my staff, Yopi.

Turned out that what we’ll be doing will be very different with what most others do. Anyway, IKC is a very popular website (and “slashdotted” daily, by leechers), so what works for most others doesn’t work for us.

The Bottlenecks

A bit of background – IKC uses WordPress as its CMS. It’s a very nice CMS, and makes your life easier. I’ve used WP myself since version 1.5.x. However, being database-based, there are a lot of points within its a WP-based infrastructure which can become a potential bottleneck. So if your website started to become popular with this CMS, you will need to start optimizing it.

After examining the situation for a while, it’s clear that MySQL was THE bottleneck. Output of top shows it using at least 8 times of CPU time than other service. Mr. Romi also told me how it kept on falling down on peak time.

Apache (and PHP, since it’s compiled as Apache module) is the next one; with each of its process using more than 10 MB of RAM. This may seem insignificant at first, but multiply that by (potentially) 150 processes – and you’ve got quite a memory hogger here.
Also CPU-usage wise; I’m quite surprised to see that each incoming request will cause the particular process’s CPU usage to spike to more than 50%.

Initial actions

I asked Mr. Romi to increase the size of MySQL’s internal cache size. He did, but the machine still fell down in daily basis.

He has also implemented caching on the app server (PHP) by way of wp-cache plugin. Still no joy too.

The Edge

I decided that we need to go straight to the “edge”, and stop the load there.

I proposed that I setup Squid in HTTP Acceleration mode. This way, most of the requests won’t even touch Apache, much less MySQL. Squid will bear most of the load, but since it’s very efficient, it should be able help a lot in making the website perform better.

Since I’ve got a few things to do myself, I asked Yopi to setup Squid in our test machine.
I just gave him pointers now and then, yet he managed to finish testing the setup and implement it in IKC’s server in just about 3.5 hours.

Then I showed him “tail -f /log/squid/access.log”, and we watched in amazement on how quickly the TCP_MISS lines are changing to TCP_HITs.
After about 12 hours, I increased the cache_mem size, and the TCP_HITs are slowly changing to TCP_MEM_HITs.

The result

Squid is working as we expected.

Average server load dropped from 30% plus to about 3%. While squid’s CPU usage increased from 0% to an average of only 2%. A very nice trade off.

After about a month, I checked the website’s logfiles, and saw some very nice numbers — traffic to IlmuKomputer.com has doubled ! Needless to say, Mr. Romi is very happy with it.

I also found that everyday there will be people downloading the contents using crawler software – such as Teleport Pro, wget, etc. I asked Mr. Romi if he’s got problem with it, and he says no. It is his mission to spread knowledge for free after all. So I let these leechers alone.

Come to think of it, it’s possible that these crawlers are the ones causing IKC server to fell down at peak hours. Example, Teleport Pro is able to download 10 links simultaneously at the same time. Then once any of it is finished, it will instantly start download the next one. When all 10 downloads access the database, and many crawlers at the same time, not many servers will be able to stand up to it. It’s like being machine gunned wearing just a simple leather cloth. If you have had the experience of having your website linked from Slashdot or Digg, you’ll understand what I’m talking about.

In this case, squid acted as a thick titanium armor, and taking most of the hits for your server. I suspect now the number of crawlers has increased than before, but it shouldn’t be a problem.

MySQL is a bit strange though. Sometimes its CPU usage can be as high as 160%. Thankfully this is very rare, so it’s probably just some internal clean-up routine.

One day, after happily watching the low load on the server for a while, suddenly everything froze. Even my SSH connection. Attempts to reconnect to the server failed.
After a while, I was finally able to connect again. Looking around, I noticed there’s some sort of bandwidth limiter daemon running on the server. After consulting with Mr. Romi, I killed it. The problem stopped.

Happy ending ?

I’m still monitoring the server as we speak for glitches. For example, squid seem to hang from time to time. This can be caused by anything from bad memory to problem with specific hardware configuration; so for now I’ve setup a cronjob which will restart it in certain intervals.
It seems to help, so I can troubleshoot the problem in peace.

Anyway, I’m sure that with the increased availability, even more people will visit the website (Ed: confirmed!). Then at some time in the future, we may find the server overloaded again.

In that case, there are still many things which we can do to keep IKC up & running in just one server :

• Coral-ize internal links : Coral is a global cache with servers all over the world. It has proven to help people with overloaded servers to lighten their load (when slashdotted, digged, etc). With the Coralize plugin, all of your internal links will point to its Coral cache.

  Actually, for most people, this may be the easiest and the best step they can do. I can setup Squid because IKC has its own dedicated server. Not everyone does, I personally also own a (shared) webhosting account. Coral CDN (Content Distribution Network) is a very nice & easy solution to us. It’s rarely mentioned though, so here you go.

  If you’re not using WordPress, you can still utilize Coral CDN easily ! Just append .nyud.net:8080 to your links. For example, if you access http://harry.sufehmi.com.nyud.net:8080, you’ll actually access a Coral server, serving a copy of my website from its cache.
  I did say that it’s very easy, didn’t I ? 🙂

• RAM Upgrade : This will enable Squid to have bigger memory cache size, therefore increasing its effectiveness significantly.
• Roundrobin Edge servers : If the load is so high that even Squid is overwhelmed by it, then we can implement a cluster of Edge servers. People can volunteer their servers and have it act as the edge server for IlmuKomputer.com.

  The incoming requests are spread over the edge servers by way of Roundrobin DNS feature. It’s not the best way to do it, but it’s very easy and the cost is almost nothing.

• Use lighttpd : Apache is a rather heavy webserver. I personally like its (amazing) flexibility (there’s a reason why it’s called the Swiss Army Knife of Webserver), but at times you’ll need something else. From my experience, lighttpd + fastcgi is very nice alternative to Apache + PHP. The features are now quite similar to Apache’s, but it’s much more lightweight. Its community is also quite helpful and happy to help a newbie within reasons. Recommended.
• And many other ways

Last, we’d like to say thanks to Mr. Romi for giving us the opportunity, it was very interesting ! Hope IKC will become even more successful in the future, therefore benefitting even more people. Well done pak.

One of the most asked questions I got from customer, in relation to their planned migration to Linux, is “Will Linux runs our legacy application ?”. And, 90% of the time, that legacy apps will be a VisualBasic 6 application. They feared that it won’t run on Linux, and their business will suffer.

I’ve always told my customers that “technical problems are not a problem“.
I can always help them find a solution for a technical issue. It’s the political ones that’s sometimes proved impossible to deal with 🙂
For example, once I met with a Canadian consultant, who happily informed me that he has been successful in making legacy apps running in older-than-dinosaur servers to talk with the web-apps on Linux, by creating a wrapper for these oldies. My inner geek bowed and saluted his hacking wizardry, and again my faith in our ability to overcome technical issues is strengthened.

Back to VisualBasic, what I did then is to observe their current situation. Each customer is unique, and a solution won’t always work for each of them. After the fact-finding session, then I usually able to prescribe the best solution for them.

Today I found one other possible solution for this.

Found a discussion on Slashdot, where it was noted that RealBasic is almost 100% compatible with VisualBasic and will run on non-Microsoft platform (even on Mac OS X).
Many will find its price (US$ 500) is way cheaper than to redevelop their corporate application.

Although not the solution for all, but it’s always good to have yet another choice, especially at this medium price range – it’s a clean solution and still affordable.

Just another reminder why I still check Slashdot from time to time — it’s not for the news, but the comments.
You guys rock. Thank you.

And to those looking to develop their corporate application – go web-based guys. Tying yourself to a single, proprietary, platform may prove very costly later.

And always, again, ALWAYS get the source code. Do not deal with a developer which will develop your corporate apps but won’t give you the source. Period.
You will thank me later for this, and when that happens, you may feel like transmitting a huge amount of money to my bank account. Don’t worry, it’s absolutely normal. In that case, just comment in this post, and you shall find my account details in your email in just a few minutes. 😀

OK, gotta code !

To be honest, I never expected to find an Open Source Business Intelligence (BI) application, yet here it is; Pentaho. Amazing.

It encompasses almost the whole spectrum of Business Intelligence concept; Reporting, Analysis, Data Mining, the BI platform, Dashboards/Management tools, and Workflow.

Good stuff. Now I know what to answer when my customers are asking me for an open BI solution.
Well done!

I’ve been managing many Windows and Unix servers in the last 10 years, and this I know for sure – backing up in Windows can be a painful experience, and most of the time it require significant investment in special backup software (which tend to cost thousands of dollars, usually more).
Even then, the software will become buggy once you start setting up complex backup scenario.

In Unix/Linux however, you have powerful scripting tools at your disposal, usually already included in the package. These tools are very flexible, enabling you to develop almost any kind of backup scheme.
It does require some sort of programming skill. Point-and-click admins will have a hard time at first, but let me tell you, do give it a try. You’ll find that it’s very much worth the trouble.

Both require investment in time & effort to develop a good backup strategy.

Before we progress, first here’s a few rules in regard to backup :

• You can never have too many backups.
  I backed up my personal data to several locations – my other PC, and also my brothers’ PC. So in total, I have 3 copies of it.
  One day, the hard drive in my main PC broke down. So I came to my brother, and asked him to copy my data which is in his PC. To my surprise, he said that his hard drive just died too. I ended up with only a single copy of my precious data.
  I quickly replaced the dead hard drive, restored the only copy of my data there, and made a new backup script for it. Nowadays, my data is usually available in 5 or more locations.
• Automate all of its processes.
  If it require even the tiniest amount of manual intervention, believe me, it will end up not being executed. For once of twice, you may still willing to intervene. But when you need to do that everyday, it just won’t happen.
• Check your backup.
  Check the result / logs everyday. Try restoring the backup about every week. Do NOT skip this, or you will find out that the backup is actually not restorable when that very important server died on you.

These are the most important ones, and I confess to have suffered from one or more of it in the past.
You don’t have to, but it’s your choice.

Anyway, here’s a sample script to get you started backing up in Linux.

Backing up the whole hard drive, over the network.

#!/bin/bash
mkdir /mnt/backup
mount -t ext3 /dev/hdb1 /mnt/backup
chmod 777 /mnt/backup
cd /mnt/backup
/usr/bin/rsync -avuz --progress --rsh="ssh -l root -i /root/.ssh/id_dsa" 192.168.0.1:/ /mnt/backup


The 1st line is important – it tells the computer that we’d like this script to be processed by bash. Different shell has different syntax. So we need to be precise about this.

The 2nd line create the directory for mounting the backup drive. 3rd line mounts /dev/hdb1 (first partition of second IDE device) to /mnt/backup. 4th line gives full access to the drive. the 6th line does the actual backup process, copying only changed files on 192.168.10.101 to /mnt/backup.

It may seem simple at first, but make no mistake, rsync is one powerful tool. For example, quoted from rsync manual:

The rsync remote-update protocol allows rsync to transfer just the differences between two sets of files across the network connection, using an efficient checksum-search algorithm described in the technical report that accompanies this package.

This capability has enabled me to backup a 200 GB hard drive, over 100 Mbps network, in under 2 hours.
Without disturbing the 15+ users which are on that network as well. Simply amazing.

The next one is probably the kind of backup script you’d more often encounter – backup, compress, store to a safe location.


#!/bin/bash
tar cvf /backup/accounting-$(date +%Y%m%d).tar /home/accounting
bzip2 -9 /backup/accounting-$(date +%Y%m%d).tar
/usr/bin/scp -2 -i ~/.ssh/id_dsa /backup/accounting-$(date +%Y%m%d).tar.bz2 smith@192.168.0.10:/data/backup/


The 2nd line bundles up the whole content of /home/accounting into a single file named /backup/accounting-(today’s date).tar; example; /backup/accounting-20061230.tar would be the resulting file if this script is run on the 30th December 2006.
This trick needed to avoid the backup replacing the same file everytime it runs. This way, we’ll have multiple backups over time, instead of just one.

The 3rd line compresses the file above, as strong as possible (with the -9 switch)
The last line copies the file (now with .bz2 extension after compressed by bzip2) into directory /data/backup/ in a server with IP address of 192.168.0.10, as user smith.

The last example is a more complicated backup script.
I developed this to backup groups.or.id‘s (kinda like Yahoogroups) member database automatically, everyday to servers on different countries. Therefore, in case of disaster, the administrator can quickly restore the service on another server with little problem.

Backup member database, over the Internet.


#!/bin/bash
daftar_milis=( $(ls ~/) )

exit 0


A bit of background, the server uses ezmlm as the mailing list software, which is usually controlled by user “alias”.

2nd line is already interesting. Basically, we execute ls (which shows the content of directory) ~/. What is directory ~/ ? Well, the tilde character (~) is a shortcut for our home directory. So, when running this script as alias user, the “ls ~/” actually means “ls /var/qmail/alias/”
The result (list of files and directories) will then be stored in an (array) variable named “daftar_milis”

4th and 5th line sets us up for a looping. It will loop as many times as there are data in “daftar_milis”.

7th line will output a line, which is “—- MILIS: (current data in “daftar_milis”) —-“, and append ( >>) it into a file named /backup/daftar-member-(today’s date).txt

8th line runs ezmlm-list, which will list the members of the mailing list, and store in into the same file as above.

When all the data in “daftar_milis” has been processed, then the 12th line will be executed. It will compress the backup file with bzip2 compression.
Note that this compression algorithm is much more complex than standard Zip compression, therefore on a slow processor it may take a very long time to finish.

The 13th line will copy the backup file to a server somewhere on the Internet, on a secure tunnel encrypted with SSH2 protocol.

So there you are, a few examples to get you started backing up in Linux. Hope you find it useful.