Translate this page :

I/O Load Monitoring

:: A few days ago a client of mine complained about slowdown on his website. I checked his server, and later noticed that the server seemed to be burdened with heavy I/O operations.

When checked with tools such as "top", the row showing "%wa" (meaning: I/O wait) was very high. Normally < 10%, it was ranging from 50% up to 98% instead.

This is a very alarming information. Because the server is using SSD disks 🙂 so its I/O (input/output) should be #really fast. Not bogged down like this.

I checked the daemons (server software) such as Apache, MySQL, Varnish, etc – and they were all idling. None were busy.

So the I/O load came from somewhere else. Probably from the hypervisor (physical server) itself. Which means it's a possible hardware problem.

Because the datacenter has concluded that it was not a "noisy neighbour" – another VM (virtual machine) in the same physical server that's hogging all the I/O resources. Pretty much all of them were idling, just like mine.

===
However I'll need some data to convince the datacenter to do hardware check on its SSD storage cluster. So I wrote this little bash script : http://pastebin.com/SxxuaVy4

The script logs server's I/O status into a CSV (Excel) format. So it can be very easily graphed later.

Using iostat tool, it probed the server's current I/O load.
Then the script is executed every minute, by running in as a cronjob.

You may notice that iostat is executed with "-d 1 3" parameter. Which means "run 3 times, with 1 second delay in between"

This is because iostat's first run always cause a spike in I/O load 🙂 so the numbers would be inaccurate. I noticed the numbers tend to stabilize after the 3rd run, so I set it up that way.

===
Of course, you can very easily modify this script to monitor something totally different 🙂 just change the iostat / head / tail / cut part to something else – voila.

===
Attached is a graph created from one of the log. The X axis is timestamp, in military format (24 hours)

I submitted the logs to the datacenter.

It convinced them to do checks on the storage cluster – and voila, they found some degraded disks in that cluster 🙂

===
Damaged disks replaced, storage cluster rebuilt – and everyone lives happily ever after ? 🙂 Fingers crossed. Happy ending !



Post imported by Google+Blog for WordPress.

7 Responses to “I/O Load Monitoring

  • 1
    Brand
    March 10th, 2017 04:57

    How is it possible to be so brainy !! I wouldn’t have found the error myself

  • 2
    Aqiqah Bekasi
    April 1st, 2017 21:05

    Nice. This is a very alarming information

  • 3
    sandra
    April 13th, 2017 12:25

    temukan ribuan film box office seru kesayanganmu hanya di AONMOVIE, website nonton gratis dengan kualitas sadis http://www.aonmovie.com

  • 4
    Jual Bunga
    May 22nd, 2017 21:40

    Keren load monitornya,,boleh juga tuuh Bunga Duka Cita

  • 5
    Banyak Bonus
    June 4th, 2017 16:16

    boleh juga ni, keren lho

  • 6
    Agen Poker Indonesia
    July 9th, 2017 21:41

    Ayo segera bergabung bersama kami di serasiqq Agen Poker & Domino QQ Online TERBAIK dan TERPERCAYA Se-ASIA
    Nikmati promo-promo menarik dari kami ;
    -BONUS CASHBACK TERBESAR mencapai 0.5% TANPA SYARAT
    -BONUS REFERRAL 10% SEUMUR HIDUP
    -EXTRA BONUS REFERRAL 10%
    Kelebihan serasiqq :
    * Minimal Deposit sangat TERJANGKAU, Hanya Rp. 10.000,-
    * Minimal Wd Rp. 20.000,-
    * Tidak ada maksimal Withdraw (menang berapa aja pasti di bayar)
    * Jackpot hingga Ratusan JUTA rupiah
    * Proses Depo dan WD SUPER CEPAT
    * Bisa dimainkan di Android, Iphone dan Ipad
    * CS yang Ramah dan Profesional siap melayani 24jam Online
    * Dalam 1 ID dapat memainkan 7 Games : Aduq, bandarQ, Domino QQ, Poker, capsa, BandarPoker dan sakong
    * Transaksi via BCA, MANDIRI, BRI dan BNI
    * 100% TANPA ROBOT Real Player vs Player
    Mari bergabung dengan kami “serasiqq”
    Agen Poker dan Domino Terbaik dan Terpercaya
    Penasaran mau daftar ?
    Untuk info selengkapnya silahkan hubungi kami di :
    – PIN BB : 22BE23ERD
    – YMail : cs_serasiqq@yahoo.com
    – LINE : serasiqq

  • 7
    angeli
    July 19th, 2017 08:54

    artikel yang bagus sob….buat yang hobi main poker online,referensi dari saya pribadi
    DOMINOHALO.COM
    Agen Poker Terpercaya

Leave a Reply

 

Subscribe without commenting

            








SEObox: Web Hosting Murah Unlimited Komik Indonesia Homeschooling Indonesia