I/O Load Monitoring

General

:: A few days ago a client of mine complained about slowdown on his website. I checked his server, and later noticed that the server seemed to be burdened with heavy I/O operations.

When checked with tools such as "top", the row showing "%wa" (meaning: I/O wait) was very high. Normally < 10%, it was ranging from 50% up to 98% instead.

This is a very alarming information. Because the server is using SSD disks 🙂 so its I/O (input/output) should be #really fast. Not bogged down like this.

I checked the daemons (server software) such as Apache, MySQL, Varnish, etc – and they were all idling. None were busy.

So the I/O load came from somewhere else. Probably from the hypervisor (physical server) itself. Which means it's a possible hardware problem.

Because the datacenter has concluded that it was not a "noisy neighbour" – another VM (virtual machine) in the same physical server that's hogging all the I/O resources. Pretty much all of them were idling, just like mine.

===
However I'll need some data to convince the datacenter to do hardware check on its SSD storage cluster. So I wrote this little bash script : http://pastebin.com/SxxuaVy4

The script logs server's I/O status into a CSV (Excel) format. So it can be very easily graphed later.

Using iostat tool, it probed the server's current I/O load.
Then the script is executed every minute, by running in as a cronjob.

You may notice that iostat is executed with "-d 1 3" parameter. Which means "run 3 times, with 1 second delay in between"

This is because iostat's first run always cause a spike in I/O load 🙂 so the numbers would be inaccurate. I noticed the numbers tend to stabilize after the 3rd run, so I set it up that way.

===
Of course, you can very easily modify this script to monitor something totally different 🙂 just change the iostat / head / tail / cut part to something else – voila.

===
Attached is a graph created from one of the log. The X axis is timestamp, in military format (24 hours)

I submitted the logs to the datacenter.

It convinced them to do checks on the storage cluster – and voila, they found some degraded disks in that cluster 🙂

===
Damaged disks replaced, storage cluster rebuilt – and everyone lives happily ever after ? 🙂 Fingers crossed. Happy ending !

View this post on Google+

Post imported by Google+Blog for WordPress.

25 thoughts on “I/O Load Monitoring”

  1. How is it possible to be so brainy !! I wouldn’t have found the error myself

  6. Ayo segera bergabung bersama kami di serasiqq Agen Poker & Domino QQ Online TERBAIK dan TERPERCAYA Se-ASIA
    Nikmati promo-promo menarik dari kami ;
    -BONUS CASHBACK TERBESAR mencapai 0.5% TANPA SYARAT
    -BONUS REFERRAL 10% SEUMUR HIDUP
    -EXTRA BONUS REFERRAL 10%
    Kelebihan serasiqq :
    * Minimal Deposit sangat TERJANGKAU, Hanya Rp. 10.000,-
    * Minimal Wd Rp. 20.000,-
    * Tidak ada maksimal Withdraw (menang berapa aja pasti di bayar)
    * Jackpot hingga Ratusan JUTA rupiah
    * Proses Depo dan WD SUPER CEPAT
    * Bisa dimainkan di Android, Iphone dan Ipad
    * CS yang Ramah dan Profesional siap melayani 24jam Online
    * Dalam 1 ID dapat memainkan 7 Games : Aduq, bandarQ, Domino QQ, Poker, capsa, BandarPoker dan sakong
    * Transaksi via BCA, MANDIRI, BRI dan BNI
    * 100% TANPA ROBOT Real Player vs Player
    Mari bergabung dengan kami “serasiqq”
    Agen Poker dan Domino Terbaik dan Terpercaya
    Penasaran mau daftar ?
    Untuk info selengkapnya silahkan hubungi kami di :
    – PIN BB : 22BE23ERD
    – YMail : cs_serasiqq@yahoo.com
    – LINE : serasiqq

  8. Memberikan bonus deposit 10% setiap hari sehingga memanjakan setiap member nayg ingin bergabung. ao burun join dan daftar sekarang juga sebelun promnya habis.

  16. SERING MERASA BOSAN?? SUKA MAIN GAME ?? YUK GABUNG DI ERAQQ . INFO! SITUS POKER TERPERCATA DI INDONESIA YANG BISA DI MAINKAN LEWAT SMART PHONE KAMU. MODAL 20RB BISA MENANG RATUSAN JUTA LOH. LAGSUNG AJA BUKA WWW . ERAQQ . INFO <<- DARI BROWSER KAMU.

  17. Very interesting, good job and thanks for sharing such a good blog. Your article is so convincing that I never stop myself to say something about it. Keep it up. – Call Center Services Melbourne, Telemarketing Companies Melbourne

  18. More interesting blog. The information you provided is useful to everyone. I appreciate you taking the time to make this post. – Search Engine Optimization Services Australia, SEO Company Australia

  22. Thank you for sharing, for those of you who are looking for a good and safe diet for the body visit the following website;
    http://www.leptosinpusat.com/
    http://www.leptosinpusat.com/obat-diet-leptosin-yang-aman-dan-baik-untuk-tubuh-wa-sms-0821-2004-3737/
    http://www.leptosinpusat.com/peluang-bisnis-online-yang-menggiurkan-wa-0821-2004-3737/
    http://www.leptosinpusat.com/bolehkah-ibu-menyusui-minum-obat-diet-wa-0821-2004-3737/
    http://www.leptosinpusat.com/pusat-penjualan-obat-diet-yang-terdaftar-bpom-murah-dan-aman/
    http://www.leptosinpusat.com/obat-diet-untuk-melangsingkan-tubuh-dengan-cepat-selama-15-hari/
    http://www.leptosinpusat.com/obat-diet-herbal-untuk-yang-mempunyai-masalah-obesitas/
    http://www.leptosinpusat.com/obat-diet-wa-0821-2004-3737-yang-dapat-meluruh-lemak-dengan-cepat/
    http://www.leptosinpusat.com/pusat-obat-diet-cepat-pelangsing-aman-dan-penurun-berat-badan-yang-ampuh/

  23. I’m just commenting to make you understand what a useful encounter our girl encountered viewing your web page. She mastered lots of details, most notably what it’s like to possess a great helping style to make other folks just completely grasp chosen problematic matters. You actually did more than our own desires. Many thanks for churning out these powerful, healthy, informative as well as cool tips about that topic to Lizeth.

  24. Hello there! Quick question that’s totally off topic. Do you know how to make your site mobile friendly? My web site looks weird when browsing from my apple iphone. I’m trying to find a template or plugin that might be able to fix this problem. If you have any recommendations, please share. Thank you!

Leave a Reply

Your email address will not be published. Required fields are marked *