
When checked with tools such as "top", the row showing "%wa" (meaning: I/O wait) was very high. Normally < 10%, it was ranging from 50% up to 98% instead.
This is a very alarming information. Because the server is using SSD disks 🙂 so its I/O (input/output) should be #really fast. Not bogged down like this.
I checked the daemons (server software) such as Apache, MySQL, Varnish, etc – and they were all idling. None were busy.
So the I/O load came from somewhere else. Probably from the hypervisor (physical server) itself. Which means it's a possible hardware problem.
Because the datacenter has concluded that it was not a "noisy neighbour" – another VM (virtual machine) in the same physical server that's hogging all the I/O resources. Pretty much all of them were idling, just like mine.
===
However I'll need some data to convince the datacenter to do hardware check on its SSD storage cluster. So I wrote this little bash script : http://pastebin.com/SxxuaVy4
The script logs server's I/O status into a CSV (Excel) format. So it can be very easily graphed later.
Using iostat tool, it probed the server's current I/O load.
Then the script is executed every minute, by running in as a cronjob.
You may notice that iostat is executed with "-d 1 3" parameter. Which means "run 3 times, with 1 second delay in between"
This is because iostat's first run always cause a spike in I/O load 🙂 so the numbers would be inaccurate. I noticed the numbers tend to stabilize after the 3rd run, so I set it up that way.
===
Of course, you can very easily modify this script to monitor something totally different 🙂 just change the iostat / head / tail / cut part to something else – voila.
===
Attached is a graph created from one of the log. The X axis is timestamp, in military format (24 hours)
I submitted the logs to the datacenter.
It convinced them to do checks on the storage cluster – and voila, they found some degraded disks in that cluster 🙂
===
Damaged disks replaced, storage cluster rebuilt – and everyone lives happily ever after ? 🙂 Fingers crossed. Happy ending !
Post imported by Google+Blog for WordPress.
How is it possible to be so brainy !! I wouldn’t have found the error myself
Nice. This is a very alarming information
temukan ribuan film box office seru kesayanganmu hanya di AONMOVIE, website nonton gratis dengan kualitas sadis http://www.aonmovie.com
Keren load monitornya,,boleh juga tuuh Bunga Duka Cita
boleh juga ni, keren lho
Ayo segera bergabung bersama kami di serasiqq Agen Poker & Domino QQ Online TERBAIK dan TERPERCAYA Se-ASIA
Nikmati promo-promo menarik dari kami ;
-BONUS CASHBACK TERBESAR mencapai 0.5% TANPA SYARAT
-BONUS REFERRAL 10% SEUMUR HIDUP
-EXTRA BONUS REFERRAL 10%
Kelebihan serasiqq :
* Minimal Deposit sangat TERJANGKAU, Hanya Rp. 10.000,-
* Minimal Wd Rp. 20.000,-
* Tidak ada maksimal Withdraw (menang berapa aja pasti di bayar)
* Jackpot hingga Ratusan JUTA rupiah
* Proses Depo dan WD SUPER CEPAT
* Bisa dimainkan di Android, Iphone dan Ipad
* CS yang Ramah dan Profesional siap melayani 24jam Online
* Dalam 1 ID dapat memainkan 7 Games : Aduq, bandarQ, Domino QQ, Poker, capsa, BandarPoker dan sakong
* Transaksi via BCA, MANDIRI, BRI dan BNI
* 100% TANPA ROBOT Real Player vs Player
Mari bergabung dengan kami “serasiqq”
Agen Poker dan Domino Terbaik dan Terpercaya
Penasaran mau daftar ?
Untuk info selengkapnya silahkan hubungi kami di :
– PIN BB : 22BE23ERD
– YMail : cs_serasiqq@yahoo.com
– LINE : serasiqq
artikel yang bagus sob….buat yang hobi main poker online,referensi dari saya pribadi
DOMINOHALO.COM
Agen Poker Terpercaya
Memberikan bonus deposit 10% setiap hari sehingga memanjakan setiap member nayg ingin bergabung. ao burun join dan daftar sekarang juga sebelun promnya habis.
Thx infonya
Thanks atas informasi yg anda berikan 😀
Agen Poker Indonesia
BandarQ Indonesia
Bandar Poker
Agen Capsa
Thanks atas informasi yg anda berikan 😀
Agen Poker Indonesia
BandarQ Indonesia
Bandar Poker
Agen Capsa
terimakasih informasinya sangat membantu sekali
Pake tool ya buatnya
Wow! What a Nice Articles ! i very very i like it bro
you are the real mvp for me
thank you very much again for the best of your webs ^^
Ihsan Magazine
Rumah Dijual di BSD City
Rumah Dijual di Pondok Cabe
Meskipun gak paham bahasanya tapi informasinya oke banget
SERING MERASA BOSAN?? SUKA MAIN GAME ?? YUK GABUNG DI ERAQQ . INFO! SITUS POKER TERPERCATA DI INDONESIA YANG BISA DI MAINKAN LEWAT SMART PHONE KAMU. MODAL 20RB BISA MENANG RATUSAN JUTA LOH. LAGSUNG AJA BUKA WWW . ERAQQ . INFO <<- DARI BROWSER KAMU.
Very interesting, good job and thanks for sharing such a good blog. Your article is so convincing that I never stop myself to say something about it. Keep it up. – Call Center Services Melbourne, Telemarketing Companies Melbourne
More interesting blog. The information you provided is useful to everyone. I appreciate you taking the time to make this post. – Search Engine Optimization Services Australia, SEO Company Australia
Artikel keren ni. Cek juga web Rendang Tambuah Ciek kami ya http://tambuahciek.com
boleh juga nih
obat spruten di wajah ampuh http://www.fuadherbalshop.com/
Terimakasih atas infonya gan
http://iroskesehatanherbal.blogspot.com/p/cara-pembelian-qnc-jelly-gamat.html
Thank you for sharing, for those of you who are looking for a good and safe diet for the body visit the following website;
http://www.leptosinpusat.com/
http://www.leptosinpusat.com/obat-diet-leptosin-yang-aman-dan-baik-untuk-tubuh-wa-sms-0821-2004-3737/
http://www.leptosinpusat.com/peluang-bisnis-online-yang-menggiurkan-wa-0821-2004-3737/
http://www.leptosinpusat.com/bolehkah-ibu-menyusui-minum-obat-diet-wa-0821-2004-3737/
http://www.leptosinpusat.com/pusat-penjualan-obat-diet-yang-terdaftar-bpom-murah-dan-aman/
http://www.leptosinpusat.com/obat-diet-untuk-melangsingkan-tubuh-dengan-cepat-selama-15-hari/
http://www.leptosinpusat.com/obat-diet-herbal-untuk-yang-mempunyai-masalah-obesitas/
http://www.leptosinpusat.com/obat-diet-wa-0821-2004-3737-yang-dapat-meluruh-lemak-dengan-cepat/
http://www.leptosinpusat.com/pusat-obat-diet-cepat-pelangsing-aman-dan-penurun-berat-badan-yang-ampuh/
I’m just commenting to make you understand what a useful encounter our girl encountered viewing your web page. She mastered lots of details, most notably what it’s like to possess a great helping style to make other folks just completely grasp chosen problematic matters. You actually did more than our own desires. Many thanks for churning out these powerful, healthy, informative as well as cool tips about that topic to Lizeth.
Hello there! Quick question that’s totally off topic. Do you know how to make your site mobile friendly? My web site looks weird when browsing from my apple iphone. I’m trying to find a template or plugin that might be able to fix this problem. If you have any recommendations, please share. Thank you!
a very useful post. let me share this post