bigls : process millions of files with ease

General
:: I have had to deal with millions of files in a single directory, and if you ever experienced it, you'll already know that it's always a major pain. 

So I created a simple PHP script, which will do :

# List millions of files with no problem

# default output : machine parseable, for easy post-processing

# human-readable output option also available

The result turned out to be quite powerful. I used it to process tens of millions of files, no sweat.

An example : 

cd /mnt/tmp/tmp1 ; bigls.php | while read OUT ; do cond=20130901 ; x=`echo $OUT | cut -b 1-8` ; f=`echo $OUT | cut -b 10-255` ; if [ $cond -ge $x ]; then mv  $f /mnt/tmp/tmp2/ >> /tmp/files-moved.log ; fi ; done

That single line of commands will move all files created before 1 Sep 2013 from /mnt/tmp/tmp1/ to /mnt/tmp/tmp2/

Again, pretty powerful stuff 🙂 now we can process million of files with no problem.

So here it is, enjoy !

bigls : https://raw.github.com/sufehmi/bigls/master/bigls.php

bigls website : https://github.com/sufehmi/bigls

note: you'll need at least PHP 5.3.0 to run bigls

bigls

View this post on Google+

Post imported by Google+Blog for WordPress.

20 thoughts on “bigls : process millions of files with ease”

  17. I am new to PHP and Linux. Can you tell me how to install bigls?

Leave a Reply

Your email address will not be published. Required fields are marked *