I deal with a lot of big files at work. While storage capacity is not infinite indeed. So it’s in my interest to keep the file sizes as low as possible.
One way to achieve that is by using compression. Especially when dealing with log files, or database archive, you can save a ton of space with the right compression tool.
But space saving is not the only consideration.
You also need to weighs in other factors. Such as :
- File type : different tool will compress different type of file differently
- CPU multi-core capabilities
- Compression speed
- Compression size
- Decompression time
But there are so many great compression tools available in Unix / Linux. It can be really confusing to choose which one to use even for a seasoned expert.
So I created X Compression Tool Benchmarker to help with this.
- Test any kind of file : just put the file’s name as the parameter when calling the script. Then it will be tested against all the specified compression tools.
- Add more compression tool easily : just edit the compressor_list & ext_file variable, and that’s it
- Fire and forget : just run the script, and forget it. It will run without needing any intervention
- CSV output : ready to be opened with Libre Office / Excel, and made into graphs in seconds.
Here’s a sample result for a Database archive file (type MySQL dump) :
The bar chart on top of this article is based from this result.
As you can see, currently this script will benchmark the following compression tools automatically : pigz – gzip – bzip2 – pbzip2 – lrzip – rzip – zstd – pixz – plzip – xz
The result, for each different file types, may surprise you 🙂
For example ; I was surprised to see rzip beat lrzip – because lrzip is supposed to be the enhancement of rzip.
Then I was even more surprised to find out that :
- I was testing Debian Buster’s version of rzip, which turned out to be pretty old – it does not even have multi-thread/core capability
- But when I tested the latest version of rzip, which can use all the 16 cores in my server – it turned out to be slower than the old rzip from Debian Buster !
- No, disk speed is not an issue – I made sure that all the benchmark was run from NVME SSD
So I was grinning at how Debian Buster packaged a very old version of rzip instead of the new one – turned out the joke’s on me : the old rzip perform better than the new one. Even without the multi-core capability.
Also it was amazing to see how really REALLY fast zstd is, while still giving decent compression size. When you absolutely need compression speed, this not so well known compression tool turned out to be the clear winner.
And so on, etc
Yes, indeed I had fun 🙂
I hope you will too. Enjoy !
UPDATE : My friend , Eko Juniarto, published his results here and have permitted me to publish it here as well – thanks. Very interesting, indeed.
31 thoughts on “XCTB – X Compression Tool Benchmarker”
Ù…Ø±Ú©Ø² Ù¾Ø®Ø´ Ø¹Ù…Ø¯Ù‡ Ø§Ø³Ù„Ø§ÛŒÙ… Ø¨Ø§ Ø¨Ø±Ù†Ø¯ Ù…Ø¬ÛŒÚ©Ø› Ø§Ø±Ø§Ø¦Ù‡ Ø¯Ù‡Ù†Ø¯Ù‡ Ø§Ù†ÙˆØ§Ø¹ Ø§Ø³Ù„Ø§ÛŒÙ… Ø¨Ø§ Ø±Ù†Ú¯ Ù‡Ø§ÛŒ Ù…ØªÙ†ÙˆØ¹ Ùˆ Ú©ÛŒÙÛŒØª Ø¨Ø§Ù„Ø§ Ø´Ù…Ø§ Ù…ÛŒ ØªÙˆØ§Ù†ÛŒØ¯ Ø¯Ø± Ù‡Ø± Ø´Ù‡Ø±ÛŒ Ú©Ù‡ Ù‡Ø³ØªÛŒØ¯ Ø¨ØµÙˆØ±Øª Ø¹Ù…Ø¯Ù‡ Ø¨Ø§Ø± Ø®ÙˆØ¯ØªØ§Ù† Ø±Ø§ Ø§Ø² ÙØ±ÙˆØ´Ú¯Ø§Ù‡ Ù…Ø¬ÛŒÚ© Ø§Ø³Ù„Ø§ÛŒÙ… Ø³ÙØ§Ø±Ø´ Ùˆ ØªØÙˆÛŒÙ„ Ø¨Ú¯ÛŒØ±ÛŒØ¯. Ù‡Ù…Ø§Ù†Ø·ÙˆØ± Ú©Ù‡ Ù…ÛŒ Ø¯Ø§Ù†ÛŒØ¯ Ø§Ø³Ù„Ø§ÛŒÙ… Ù‡Ø§ÛŒ Ú¯ÙˆÙ†Ø§Ú¯ÙˆÙ†ÛŒ Ø¯Ø± Ø¨Ø§Ø²Ø§Ø± Ù…ÙˆØ¬ÙˆØ¯ Ù…ÛŒ Ø¨Ø§Ø´Ø¯
Nice artilce you can visit my website Sewa Container
Dear sufehmi.com admin, Your posts are always a great source of information.
Dear sufehmi.com administrator, You always provide great examples and real-world applications, thank you for your valuable contributions.
Hi sufehmi.com admin, Thanks for the in-depth post!
Dear sufehmi.com owner, Your posts are always well organized and easy to understand.
Hi sufehmi.com webmaster, Thanks for the well-researched and well-written post!
Hello sufehmi.com admin, Thanks for the well-written and informative post!
Hello sufehmi.com admin, Your posts are always a great source of knowledge.
To the sufehmi.com admin, Thanks for the well-organized post!
Hi sufehmi.com owner, Your posts are always well structured and easy to follow.
Hello sufehmi.com owner, Thanks for the well-organized and comprehensive post!
Hi sufehmi.com admin, Great content!
Dear sufehmi.com owner, Excellent work!
To the sufehmi.com admin, Thanks for the great post!
To the sufehmi.com administrator, Your posts are always well-referenced and credible.
Dear sufehmi.com admin, Thanks for the informative post!
Dear sufehmi.com admin, You always provide helpful diagrams and illustrations.
Hi sufehmi.com owner, Your posts are always well written and informative.
Hi sufehmi.com administrator, Your posts are always informative and well-explained.
Dear sufehmi.com administrator, Thanks for the post!
Hi sufehmi.com admin, Thanks for the well-organized post!
To the sufehmi.com webmaster, Your posts are always well written and informative.
Hi sufehmi.com admin, Keep up the great work!
Dear sufehmi.com administrator, Great content!
Hi sufehmi.com admin, Keep up the great work!
Hi sufehmi.com admin, Your posts are always well-supported and evidence-based.
Hi sufehmi.com owner, Keep up the good work, admin!
Hello sufehmi.com webmaster, Your posts are always well-timed and relevant.
Hello sufehmi.com admin, Your posts are always well-referenced and credible.
To the sufehmi.com owner, Your posts are always well-received and appreciated.