GlusterFS : Powerfully simple SAN for Enterprises. :: cheap – powerful – easy : pick all three. That's GlusterFS, a software-based SAN (Storage Area Network) from RedHat.
By using commodity / generic hardware, you can have a resilient, very scalable SAN, with very high availability.

In this article, I'll show you how to get a GlusterFS cluster up & running in very little time. This tutorial will install GlusterFS from source, so even though it was written for Debian 6 Linux, it should be very easy to adapt it for other Linux distros as well.

ASSUMPTIONS :

(1) There are 2 servers; called server1 & server2

(2) Each have at least 1 GB of RAM : this is the minimum recommended RAM for GlusterFS.

(3) Each have 2 partitions for GlusterFS, called /dev/sdb1 and /dev/sdc1

(4) GlusterFS volume to be setup : Distributed Replicated Volume (DRV)

HOWTO :

####################################################
########## EXECUTE ON SERVER1 & SERVER2 ############
####################################################

### install required software packages & libraries
sudo apt-get install openssh-server screen sudo build-essential flex bison libssl-dev libaio-dev xfsprogs attr

mkdir tmp
cd tmp

### get the latest version of GlusterFS
wget http://download.gluster.org/pub/gluster/glusterfs/LATEST/glusterfs-3.4.0.tar.gz

tar xzvf glusterfs-3.4.0.tar.gz
cd glusterfs-3.4.0

### install GlusterFS :
# this will install both the server & client
# GlusterFS will be install on /opt/gfs
mkdir -p /opt/gfs
./configure –prefix=/opt/gfs
sudo make all
sudo make install

### make sure GFS is running everytime the server restarts
# insert the following line on the fifth line in /etc/rc.local
sudo sed -i "5i /opt/gfs/sbin/glusterd" /etc/rc.local

### create bricks

sudo fdisk /dev/sdb
# then create partition /dev/sdb1
sudo su
# use XFS filesystem, this is the only recommended filesystem by GlusterFS
mkfs.xfs /dev/sdb1
mkdir -p /mnt/disk1
mount /dev/sdb1 /mnt/disk1
mkdir -p /mnt/disk1/brick1
echo "/dev/sdb1 /mnt/disk1 xfs defaults 0 0"  >> /etc/fstab

fdisk /dev/sdc
# then create partition /dev/sdc1
mkfs.xfs /dev/sdc1
mkdir -p /mnt/disk2
mount /dev/sdc1 /mnt/disk2
mkdir -p /mnt/disk2/brick2
echo "/dev/sdc1 /mnt/disk2 xfs defaults 0 0"  >> /etc/fstab

####################################################
############### EXECUTE ON SERVER1 #################
####################################################

### execute GlusterFS daemon on server1
# this will automatically create the "Trusted Storage Pool" (TSP) on server1
/opt/gfs/sbin/glusterd

### join server2 to TSP
/opt/gfs/sbin/gluster peer probe server2

### check TSP status
/opt/gfs/sbin/gluster peer status

### create Distributed Replicated Volume (DRV)
/opt/gfs/sbin/gluster volume create gfsvol1 replica 2 transport tcp server1:/mnt/disk1/brick1 server2:/mnt/disk1/brick1 server1:/mnt/disk2/brick2 server2:/mnt/disk2/brick2

### start volume
/opt/gfs/sbin/gluster volume start gfsvol1

### check DRV volume's status
/opt/gfs/sbin/gluster volume info

### Success !
# Volume Name: gfsvol1
# Type: Distributed-Replicate
# Volume ID: 7f7e35ea-fbae-480b-a876-6244d365cfb2
# Status: Started
# Number of Bricks: 2 x 2 = 4
# Transport-type: tcp
#
# Bricks:
# Brick1: server1:/mnt/disk1/brick1
# Brick2: server2:/mnt/disk1/brick1
# Brick3: server1:/mnt/disk2/brick2
# Brick4: server2:/mnt/disk2/brick2

####################################################
################ EXECUTE ON Client #################
####################################################

### install required software packages & libraries
sudo apt-get update
sudo apt-get install openssh-server screen sudo build-essential flex bison libssl-dev libaio-dev xfsprogs attr fuse fuse-utils

mkdir tmp
cd tmp

### get the latest version of GlusterFS
wget http://download.gluster.org/pub/gluster/glusterfs/LATEST/glusterfs-3.4.0.tar.gz

tar xzvf glusterfs-3.4.0.tar.gz
cd glusterfs-3.4.0

### install GlusterFS :
# this will install both the server & client
# GlusterFS will be install on /opt/gfs
mkdir -p /opt/gfs
./configure –prefix=/opt/gfs
sudo make all
sudo make install

sudo modprobe fuse

sudo mkdir /mnt/1
sudo mount -t glusterfs server1:/gfsvol1 /mnt/1

# check, make sure that it is mounted
df

# done !

# test GFS

dd if=/dev/zero of=/tmp/file-1GB bs=10M count=100

date; cp /tmp/file-1GB /mnt/1/; date

ls -l /mnt/1/

TIPS / TRICK

### Sometimes you need to erase & reuse a brick in a volume. This is how to do it
### You can not just reuse a brick again, 
### first you'll need to clear the special GFS attributes on it
# ref: http://linuxsysadm.wordpress.com/2013/05/16/glusterfs-remove-extended-attributes-to-completely-remove-bricks/

cd /mnt/disk1/brick1

for i in `attr -lq /mnt/disk1/brick1`; do   setfattr -x trusted.$i .; done

attr -lq /mnt/disk1/brick1