GlusterFS : Powerfully simple SAN for Enterprises. :: cheap – powerful – easy : pick all three. That's GlusterFS, a software-based SAN (Storage Area Network) from RedHat.
By using commodity / generic hardware, you can have a resilient, very scalable SAN, with very high availability.
In this article, I'll show you how to get a GlusterFS cluster up & running in very little time. This tutorial will install GlusterFS from source, so even though it was written for Debian 6 Linux, it should be very easy to adapt it for other Linux distros as well.
ASSUMPTIONS :
(1) There are 2 servers; called server1 & server2
(2) Each have at least 1 GB of RAM : this is the minimum recommended RAM for GlusterFS.
(3) Each have 2 partitions for GlusterFS, called /dev/sdb1 and /dev/sdc1
(4) GlusterFS volume to be setup : Distributed Replicated Volume (DRV)
HOWTO :
####################################################
########## EXECUTE ON SERVER1 & SERVER2 ############
####################################################
### install required software packages & libraries
sudo apt-get install openssh-server screen sudo build-essential flex bison libssl-dev libaio-dev xfsprogs attr
mkdir tmp
cd tmp
### get the latest version of GlusterFS
wget http://download.gluster.org/pub/gluster/glusterfs/LATEST/glusterfs-3.4.0.tar.gz
tar xzvf glusterfs-3.4.0.tar.gz
cd glusterfs-3.4.0
### install GlusterFS :
# this will install both the server & client
# GlusterFS will be install on /opt/gfs
mkdir -p /opt/gfs
./configure –prefix=/opt/gfs
sudo make all
sudo make install
### make sure GFS is running everytime the server restarts
# insert the following line on the fifth line in /etc/rc.local
sudo sed -i "5i /opt/gfs/sbin/glusterd" /etc/rc.local
### create bricks
sudo fdisk /dev/sdb
# then create partition /dev/sdb1
sudo su
# use XFS filesystem, this is the only recommended filesystem by GlusterFS
mkfs.xfs /dev/sdb1
mkdir -p /mnt/disk1
mount /dev/sdb1 /mnt/disk1
mkdir -p /mnt/disk1/brick1
echo "/dev/sdb1 /mnt/disk1 xfs defaults 0 0" Â >> /etc/fstab
fdisk /dev/sdc
# then create partition /dev/sdc1
mkfs.xfs /dev/sdc1
mkdir -p /mnt/disk2
mount /dev/sdc1 /mnt/disk2
mkdir -p /mnt/disk2/brick2
echo "/dev/sdc1 /mnt/disk2 xfs defaults 0 0" Â >> /etc/fstab
####################################################
############### EXECUTE ON SERVER1 #################
####################################################
### execute GlusterFS daemon on server1
# this will automatically create the "Trusted Storage Pool" (TSP) on server1
/opt/gfs/sbin/glusterd
### join server2 to TSP
/opt/gfs/sbin/gluster peer probe server2
### check TSP status
/opt/gfs/sbin/gluster peer status
### create Distributed Replicated Volume (DRV)
/opt/gfs/sbin/gluster volume create gfsvol1 replica 2 transport tcp server1:/mnt/disk1/brick1 server2:/mnt/disk1/brick1 server1:/mnt/disk2/brick2 server2:/mnt/disk2/brick2
### start volume
/opt/gfs/sbin/gluster volume start gfsvol1
### check DRV volume's status
/opt/gfs/sbin/gluster volume info
### Success !
# Volume Name: gfsvol1
# Type: Distributed-Replicate
# Volume ID: 7f7e35ea-fbae-480b-a876-6244d365cfb2
# Status: Started
# Number of Bricks: 2 x 2 = 4
# Transport-type: tcp
#
# Bricks:
# Brick1: server1:/mnt/disk1/brick1
# Brick2: server2:/mnt/disk1/brick1
# Brick3: server1:/mnt/disk2/brick2
# Brick4: server2:/mnt/disk2/brick2
####################################################
################ EXECUTE ON Client #################
####################################################
### install required software packages & libraries
sudo apt-get update
sudo apt-get install openssh-server screen sudo build-essential flex bison libssl-dev libaio-dev xfsprogs attr fuse fuse-utils
mkdir tmp
cd tmp
### get the latest version of GlusterFS
wget http://download.gluster.org/pub/gluster/glusterfs/LATEST/glusterfs-3.4.0.tar.gz
tar xzvf glusterfs-3.4.0.tar.gz
cd glusterfs-3.4.0
### install GlusterFS :
# this will install both the server & client
# GlusterFS will be install on /opt/gfs
mkdir -p /opt/gfs
./configure –prefix=/opt/gfs
sudo make all
sudo make install
sudo modprobe fuse
sudo mkdir /mnt/1
sudo mount -t glusterfs server1:/gfsvol1 /mnt/1
# check, make sure that it is mounted
df
# done !
# test GFS
dd if=/dev/zero of=/tmp/file-1GB bs=10M count=100
date; cp /tmp/file-1GB /mnt/1/; date
ls -l /mnt/1/
TIPS / TRICK
### Sometimes you need to erase & reuse a brick in a volume. This is how to do it
### You can not just reuse a brick again,Â
### first you'll need to clear the special GFS attributes on it
# ref: http://linuxsysadm.wordpress.com/2013/05/16/glusterfs-remove-extended-attributes-to-completely-remove-bricks/
cd /mnt/disk1/brick1
for i in `attr -lq /mnt/disk1/brick1`; do  setfattr -x trusted.$i .; done
attr -lq /mnt/disk1/brick1