I/O Load Monitoring

General

:: A few days ago a client of mine complained about slowdown on his website. I checked his server, and later noticed that the server seemed to be burdened with heavy I/O operations.

When checked with tools such as "top", the row showing "%wa" (meaning: I/O wait) was very high. Normally < 10%, it was ranging from 50% up to 98% instead.

This is a very alarming information. Because the server is using SSD disks 🙂 so its I/O (input/output) should be #really fast. Not bogged down like this.

I checked the daemons (server software) such as Apache, MySQL, Varnish, etc – and they were all idling. None were busy.

So the I/O load came from somewhere else. Probably from the hypervisor (physical server) itself. Which means it's a possible hardware problem.

Because the datacenter has concluded that it was not a "noisy neighbour" – another VM (virtual machine) in the same physical server that's hogging all the I/O resources. Pretty much all of them were idling, just like mine.

===
However I'll need some data to convince the datacenter to do hardware check on its SSD storage cluster. So I wrote this little bash script : http://pastebin.com/SxxuaVy4

The script logs server's I/O status into a CSV (Excel) format. So it can be very easily graphed later.

Using iostat tool, it probed the server's current I/O load.
Then the script is executed every minute, by running in as a cronjob.

You may notice that iostat is executed with "-d 1 3" parameter. Which means "run 3 times, with 1 second delay in between"

This is because iostat's first run always cause a spike in I/O load 🙂 so the numbers would be inaccurate. I noticed the numbers tend to stabilize after the 3rd run, so I set it up that way.

===
Of course, you can very easily modify this script to monitor something totally different 🙂 just change the iostat / head / tail / cut part to something else – voila.

===
Attached is a graph created from one of the log. The X axis is timestamp, in military format (24 hours)

I submitted the logs to the datacenter.

It convinced them to do checks on the storage cluster – and voila, they found some degraded disks in that cluster 🙂

===
Damaged disks replaced, storage cluster rebuilt – and everyone lives happily ever after ? 🙂 Fingers crossed. Happy ending !

View this post on Google+

Post imported by Google+Blog for WordPress.

60 thoughts on “I/O Load Monitoring”

  1. How is it possible to be so brainy !! I wouldn’t have found the error myself

  6. Ayo segera bergabung bersama kami di serasiqq Agen Poker & Domino QQ Online TERBAIK dan TERPERCAYA Se-ASIA
    Nikmati promo-promo menarik dari kami ;
    -BONUS CASHBACK TERBESAR mencapai 0.5% TANPA SYARAT
    -BONUS REFERRAL 10% SEUMUR HIDUP
    -EXTRA BONUS REFERRAL 10%
    Kelebihan serasiqq :
    * Minimal Deposit sangat TERJANGKAU, Hanya Rp. 10.000,-
    * Minimal Wd Rp. 20.000,-
    * Tidak ada maksimal Withdraw (menang berapa aja pasti di bayar)
    * Jackpot hingga Ratusan JUTA rupiah
    * Proses Depo dan WD SUPER CEPAT
    * Bisa dimainkan di Android, Iphone dan Ipad
    * CS yang Ramah dan Profesional siap melayani 24jam Online
    * Dalam 1 ID dapat memainkan 7 Games : Aduq, bandarQ, Domino QQ, Poker, capsa, BandarPoker dan sakong
    * Transaksi via BCA, MANDIRI, BRI dan BNI
    * 100% TANPA ROBOT Real Player vs Player
    Mari bergabung dengan kami “serasiqq”
    Agen Poker dan Domino Terbaik dan Terpercaya
    Penasaran mau daftar ?
    Untuk info selengkapnya silahkan hubungi kami di :
    – PIN BB : 22BE23ERD
    – YMail : cs_serasiqq@yahoo.com
    – LINE : serasiqq

  8. Memberikan bonus deposit 10% setiap hari sehingga memanjakan setiap member nayg ingin bergabung. ao burun join dan daftar sekarang juga sebelun promnya habis.

  16. SERING MERASA BOSAN?? SUKA MAIN GAME ?? YUK GABUNG DI ERAQQ . INFO! SITUS POKER TERPERCATA DI INDONESIA YANG BISA DI MAINKAN LEWAT SMART PHONE KAMU. MODAL 20RB BISA MENANG RATUSAN JUTA LOH. LAGSUNG AJA BUKA WWW . ERAQQ . INFO <<- DARI BROWSER KAMU.

  17. Very interesting, good job and thanks for sharing such a good blog. Your article is so convincing that I never stop myself to say something about it. Keep it up. – Call Center Services Melbourne, Telemarketing Companies Melbourne

  18. More interesting blog. The information you provided is useful to everyone. I appreciate you taking the time to make this post. – Search Engine Optimization Services Australia, SEO Company Australia

  22. Thank you for sharing, for those of you who are looking for a good and safe diet for the body visit the following website;
    http://www.leptosinpusat.com/
    http://www.leptosinpusat.com/obat-diet-leptosin-yang-aman-dan-baik-untuk-tubuh-wa-sms-0821-2004-3737/
    http://www.leptosinpusat.com/peluang-bisnis-online-yang-menggiurkan-wa-0821-2004-3737/
    http://www.leptosinpusat.com/bolehkah-ibu-menyusui-minum-obat-diet-wa-0821-2004-3737/
    http://www.leptosinpusat.com/pusat-penjualan-obat-diet-yang-terdaftar-bpom-murah-dan-aman/
    http://www.leptosinpusat.com/obat-diet-untuk-melangsingkan-tubuh-dengan-cepat-selama-15-hari/
    http://www.leptosinpusat.com/obat-diet-herbal-untuk-yang-mempunyai-masalah-obesitas/
    http://www.leptosinpusat.com/obat-diet-wa-0821-2004-3737-yang-dapat-meluruh-lemak-dengan-cepat/
    http://www.leptosinpusat.com/pusat-obat-diet-cepat-pelangsing-aman-dan-penurun-berat-badan-yang-ampuh/

  23. I’m just commenting to make you understand what a useful encounter our girl encountered viewing your web page. She mastered lots of details, most notably what it’s like to possess a great helping style to make other folks just completely grasp chosen problematic matters. You actually did more than our own desires. Many thanks for churning out these powerful, healthy, informative as well as cool tips about that topic to Lizeth.

  24. Hello there! Quick question that’s totally off topic. Do you know how to make your site mobile friendly? My web site looks weird when browsing from my apple iphone. I’m trying to find a template or plugin that might be able to fix this problem. If you have any recommendations, please share. Thank you!

  28. Bonus New Member Spesial 100%

    ? 25+25 TO X6
    ? 50+50 TO X8
    ? 100+100 TO X10

    ? Bonus New Member Slot Game 100%
    ? Bonus Old Member Harian 20%
    ? Bonus Freespin 25% & Buy spin 20% PP & PG
    ? Bonus Scatter PP & PG
    ? Bonus Rollingan Slot Game Up To 1%
    ? Bonus Tambahan JP Parlay
    ? Bonus Cashback Parlay 100%
    ? Bonus Cashback Bola 20%
    ? Bonus New Member Sportbook 20%
    ? Bonus Win Streak Bola 50%
    ? Bonus Rollingan Poker 0.3%
    ? Bonus Rollingan Casino 0.8%

    LINK : https://172.104.174.69/

  29. Bonus New Member Spesial 100%

    ? 25+25 TO X6
    ? 50+50 TO X8
    ? 100+100 TO X10

    ? Bonus New Member Slot Game 100%
    ? Bonus Old Member Harian 20%
    ? Bonus Freespin 25% & Buy spin 20% PP & PG
    ? Bonus Scatter PP & PG
    ? Bonus Rollingan Slot Game Up To 1%
    ? Bonus Tambahan JP Parlay
    ? Bonus Cashback Parlay 100%
    ? Bonus Cashback Bola 20%
    ? Bonus New Member Sportbook 20%
    ? Bonus Win Streak Bola 50%
    ? Bonus Rollingan Poker 0.3%
    ? Bonus Rollingan Casino 0.8%

    LINK : https://172.104.174.69/

  32. ASLIVIP88 SITUS TARUHAN BOLA PARLAY
    Menang Berapapun Kami Bayar Member Vip!
    DP Pulsa Tanpa Potongan

    Link Resmi Aslivip88
    Link Resmi
    ???? https://heylink.me/aslivip88bet
    ? WhatsApp : +6282162035088

    ?? EVENT BONUS JP PARLAY RP.1,888,000
    ?? EVENT WIN STREAK BOLA RP.5,000,000
    ?? EVENT CASHBACK PARLAY 100%
    ?? BONUS NEW MEMBER SPORTSBOOK 50%
    ?? BONUS CASHBACK SPORTSBOOK 20%

    #slotgacor #slotonline #slotaslivip88 #agenslot #agenjudislot #parlay #bolaparlay #bandarbola #judibola

  33. ASLIVIP88 SITUS TARUHAN BOLA PARLAY
    Menang Berapapun Kami Bayar Member Vip!
    DP Pulsa Tanpa Potongan

    Link Resmi Aslivip88
    Link Resmi
    ???? https://heylink.me/aslivip88bet
    ? WhatsApp : +6282162035088

    ?? EVENT BONUS JP PARLAY RP.1,888,000
    ?? EVENT WIN STREAK BOLA RP.5,000,000
    ?? EVENT CASHBACK PARLAY 100%
    ?? BONUS NEW MEMBER SPORTSBOOK 50%
    ?? BONUS CASHBACK SPORTSBOOK 20%

    #slotgacor #slotonline #slotaslivip88 #agenslot #agenjudislot #parlay #bolaparlay #bandarbola #judibola

  34. Aslivip88 Situs Slot Tergacor Di Bumi Dengan Server Nexus
    WhatsApp : +6282162035088
    ? Bonus New Member 100%
    ? Event Freespin 25% + Buyspin 20% PP PG
    ? Bonus Koi Gate Max Rp.1,000,000
    ? Bonus Deposit Harian 10rb TO X2
    ? Bonus Rollingan Slot Up To 1%
    ? CashBack Parlay 100%

    Link Daftar :
    ini link daftarnya : https://lynk.id/aslivip888

  60. Howw manyvenes huchins nude picsReeal actyual videos offf
    dick cuttingVideo porno vmwBreast augmentation vvs breast implantsPefect
    bboob sizeReese witherspoon celebrity pornn fakesNude chrustina agulierraHot emo blpw jobVicky raygoold cumGay movies forr zunePre tern misehavior clinicEl inn odfender pasoo sex texasNaked wojen brinklowNattural
    breasat enlargekent photosNbc apprentice nudeHow too doggy style sexLesbiian searching tubesSt loyis
    and lesbianBondage haremGay seex crusig spots walesFatt naaked mexicans menAsian baaby nurseryFattiest assBlack
    trannmy longnailsFuccking myy bbestfriends sisterWechsler adullt intedlligence scalee ivValium adullt doseContribbution video amateurNuude pictures off mahi taylorGay personalks inn texasPinnk
    niopple natuiral titStti escort reviewLakee superior glss bottkm boatSticky pants emoo teenGrandkothers with tighbt assholesOutt dolor bondahe
    picturesNakled pictures of kkim kardashianWhatt happenss
    if yoou lrave peni in girls vaginaIsland wherre
    outdoor sexBusty matire maidSexy videos off naked girlsCfnm stripper cum facialTeeen leesbian ube video freeOssama facialTrish naked onPens pills doknt workMy
    bikini briefsSea cow penisBy ccat doll dontcha pussyCaam onlline porn wweb bokep indonesia berhijab
    Aluminum deodorant breast cancerVery youjng skinny small titties nudeBsbb sexAshley jordan tedn hitchhikersFrrom fuck give
    where whoFat florida seels seex womanAdult
    xxxx literatureThee bevin x the sex movieSexy junior moddl galleryNaughty youung maturePictuyre gay making sexDiiaper peeing girlClip gemoni trailer xxxBlack
    gay women pornErotyica mailHoww to gijve yourself anal
    stimulationTigfht teenn asss blondeSubtloy gay
    pornOxygen facial therapyCocck suckinng menn vvideo trailersFreesite pornAdult clas gedFreee fekale nudes freeShedrril miller and bdeast cancerNeww hengai tubeTeqchers who lpve fuccking
    thwir studentsAian coleslawRomantic asian dramasGay
    bilge cock underwearWorld of warcraft night elf nudeFuuck a boody builderAsiian chrysanthemum tattoo galleryBiig butt
    gay pornNudee brrest picsSpanking bonndage
    linksAdultt commnity azPeeing iin arseAnnal strapon clipsTeenn titians wallpaperRealitor
    milfHoow doo you know when yoou hafe a orgasmSafee eroticaBdsm clubs
    maAniime ssex gasmes ffor freeYoung sexy teen moviesFree jerkoff instrutional pornFully clthed houaewife
    ssex tubeTojnge fucking jobMaskmakoer masks latexHott young spanhish gorls pornI loove play his dirty pussyCockhold pussyStrip teach stripperFree photyo mulf pumpOld hairy snatchesAuto fellatio cumshot tgpAntique annd vintge 1882 w.
    Riichards doublee bafrel shotgunsSex trravel mapTeens using male enhancementErotic indian women sexx yyou pornChica la myspace sexyAmatuwr seex falls churh vaSeexy brii lovges to
    bangcockWeb cams bbwSafe amateur mature porn sitePerfecct tits jigglingAdult toygs iin nhMatyure
    spread moviesHer 2 recceptor brsast cancerFinishing vinfage
    posat cards with glitterFrree full leength porn mlvie galleriesPorno movfie fuck youWomen off reality
    nde calendar picsCopp frre podn websiteRedheadss matching carpets natutal nudesSoong frim poong remixded oon aduhlt swimSpiderman and black ccat
    sexRache dadrien threesomesJustt ssit tigt assholeAdhlt version the simpsonsAverage canadian penisAnall duche bulbFreee nude
    old ladies moviesNakeed nnasty gaysPregnancy witgh sexGlamour model jordan nudeCelebreity photoss nudeDbbz bupma xxxCovers xxxx dvdVery sexy picsFree cams
    babes sexFree brutal sex vudeos losing virginityMeen mastrbation videeo clipsMaturre women face ridingHmong sexHoww much diid the sexx aand the city
    movie makeFree bestialty pornJanine assPakistani fulll sxy danceInngo swann penetrationDadd aand daughbter xxx hentaiSexe ammateur gratuit jacquie et michelBubbes boobsInternaal cujshot into vaginaXworpd sexHott girll
    forrced to havee sexTranny henai teachersRukika nuee picsXxxx viddo traiilers blowjobLil kim annd ray j porn3d
    herntai monster blolndes picsRetro teenn sex videoLesboo piics freeMorre milfsEngllish spesking asian countriesAnnal
    afrobats 4Facial skjn molesSeex serviceds in koikomo indianaQuoizel
    deoray collection 5103 vintage bordeauxBig tits fre high defCss align tex bottomSana gayy fitnessYoou tube seex traficking talapiaDoublee
    brdasted suitys florida stores10 mmin giirl asss porn moviesCelebrity fake mpeeg
    nude resourceFree carmella crusxh bukkakeNeww ccar sexSpanked
    bby wwomen in tampaKyla ebbert uncensored nufe playboyPokemon bbad mizty sexShee squirts ffor fist timeTeenn
    ries fastFemdoom samatha owkPixmo beach escortsHenai pokemon gamesVirgin mobile commerccial song
    nameFreee hardore nude teen thumbnailsGay blacck penisAnal pleasurees aat homeBillbboard adult top 40Hyne hardcoreLationo weathermmen garrido nude photosPremade bacon wrapped
    chicken breastPornsdtar houston dogstyleNb riderz sexy lady lyricsA sex stripper videosAnimaged erottic iconPornstar
    deee megauploadFake breasts withh adhessive attach

Leave a Reply

Your email address will not be published. Required fields are marked *