Labs: installation & configuration of GlusterFS as synchronous data storage solution.
By: Pascal Charest, Freesoftware consultant
Date: September, 2008.
Synchronization of files in a cloud environment is a challenge in the path of high-{availability, performance}. From simple load balanced web sites to full-blown applications - some files always need to be in sync. Peoples, for simplicity, rely on asynchronous transfer (ie: rsync ), others deploy bigger solutions (ie: block device replication through DRBD or shared storage through AoE protocol & concurrency management with OCFSv2) or even go for the “lazy” “no-shared-storage” solution through NFS.
To address this problem in the PraizedMedia software stack, I decided to give FUSE based GlusterFS a try. Awesome, really ! The technical knowledge to deploy a basic solution is very very low. The modularity of the program also help to have “something working right now”. This isn’t meant as a direct alternative to DRBD or a good SAN deployment but in my use case, it fit perfectly.
In this lab, I will guide you through the installation of GlusterFS on 2 networked systems. They will be both used as “servers” & “client” for the GlusterFS filesystem. They will be sharing a directory (on both system : /var/production/brick), re-mounted as /var/production/static through GlusterFS. Any write I/O on this directory (of any client server) will be synchronized to the pool. This last feature is called “AFR” (for automatic file replication) and is a module (called a translator) to the GlusterFS file system.
The specificity of my environment is around the file-locking management : I don’t need any. By design, the application will never try to write the same file twice on any of the server.
#Installation of requirement (standard tools)
apt-get install flex bison libfuse-dev linux-headaers-`uname -r` curl
#download of the sources
cd /usr/local/src/
curl -O http://ftp.zresearch.com/pub/gluster/glusterfs/1.3/glusterfs-CURRENT.tar.gz
tar zxf glusterfs-CURRENT.tar.gz
# configure
cd glusterfs-1.3.11
./configure --prefix=/usr/local/glusterfs-1.3.11
make && make install
ln -s /usr/local/glusterfs-1.3.11 /usr/local/glusterfs
So we now have a basic 2 servers GlusterFS systems installed. Lets be honest, that wasn’t really hard! We are still missing configuration files though.
#Editing /usr/local/glusterfs/etc/glusterfs/glusterfs-server.vol
#
# glusterfs-servers definition
# volume definition are on first lvl, other are on second lvl (tabbed)
volume brick
type storage/posix
option directory /mnt/production/brick
end-volume
volume server
type protocol/server
option transport-type tcp/server
option auth.ip.brick.allow *
subvolumes brick
end-volume
#Editing the /usr/local/glusterfs/etc/glusterfs/glusterfs-client.vol
#
# glusterfs-client.vol
# volume definition are on first lvl, other are on second lvl (tabbed)
#
volume remote1
type protocol/client
option transport-type tcp/client
option remote-host 002.praized.com
option remote-subvolume brick
end-volume
volume remote2
type protocol/client
option transport-type tcp/client
option remote-host 001.praized.com
option remote-subvolume brick
end-volume
volume mirror0
type cluster/afr
subvolumes remote1 remote2
end-volume
#Launching services (servers and clients)
mkdir -p /mnt/production/brick
/usr/local/glusterfs-1.3.11/sbin/glusterfsd -f /usr/local/glusterfs-1.3.11/etc/glusterfs/glusterfs-server.vol
mkdir -p /mnt/production/static
/usr/local/glusterfs-1.3.11/sbin/glusterfs -f /usr/local/glusterfs-1.3.11/etc/glusterfs/glusterfs-client.vol /mnt/production/static/
You now possess a synchronized directory between your two systems. Please note that GlusterFS require TCP/6996 port to be open. There is also some improvement that can be done to this setup through adding a locking mechanism & i/o thread - I don’t currently need them, but you might.
Enjoy!
Debugging notes ; after starting the server process you should have a kernel process call glusterfs. All log files are in /usr/local/glusterfs/var/log/glusterfs*. After starting the client, “df -h” should show you your new mount point. Careful with UID/GID (&Permission), there is no such thing as root_squash_fs in GlusterFS yet.
Other notes ; Using Amazon EBS would have been the perfect solution if they did allow multiple servers-volume mount and lets us deal with concurrency / lock problems. But, they don’t.
I had a meeting with a potential client today. He made me think when he said to me : “there is a lot of GNU/Linux sysadmin available to work in Montreal, why “you” ?”. While this is true and valid (GNU/Linux isn’t the “beast” it used to be), I think that we are seeing the same effect that has happen for years in the Windows universe : “Click + Click = functional server”.
A LAMP based stack can now be deployed in couple of seconds using most of the distribution available on the web. While having a Apache web server and a MySQL database might work for small domain (couple of “unused websites”), it doesn’t survive that well under the “web2.0 dynamic” charge.
Optimization of the code is good up to a point - and I think this is where most “good idea” fail. The cost of a developer for a week can be around 1400$ (think 35$ * 8 hours * 5 days). In comparison the cost of a single EC2 instance for a whole year is around 1600$ (each month: 50GB in, 500GB out - instance open 24/7). The online calculator is available here.
This mean that while code optimization might be interesting, the first thing web related application should do is to scale horizontally. The technology is available. There is a true plethora of technology available to scale web based infrastructure.
In the last couple of month, I’ve been working with some clients to build scalable infrastructures. The bottom line would be that everything can scale with the good infrastructure. One very good post about this statement has been done by on Nati Shalon’s blog (here, he is speaking of twitter).
In the next couples of days, I’ll be writing about some of those “scalable” systems.
Après “The Cathedral & The Bazaar” de Eric S. Raymond, “The Success of Open Source” de Steven Weber et “Free as in Freedom” de Sam Williams, je progresse sur mon parcours de relecture de ces classiques avec “Code 2.0” de Lawrence Lessig.
J’ai été relativement chanceux d’avoir trouver ce dernier chez Indigo au centre ville (de Montréal), sinon, c’est surement un live sur la photographie qui aurait passer le seuil de ma bibliothèque. Pas mauvais en sois, mais moins intéressant à terme.
J’ai un peu hate de voir les modifications qui ont été apportées en rapport à la première version - dont je n’ai pas souvenir d’avoir vu depuis quelques années, ce qui laisse présager que je l’ai passer à très long terme a quelqu’un.
6. Use the Open Source certification mark to keep things pure
One of the threats we faced was the possibility that the term ‘open source’ would be "embraced and extended" by Microsoft or other large vendors, corrupting it and losing our message.
[...]
It eventually developed that the U.S. Patent and Trademark office would not issue a trademark for such a descriptive phrase [Open Source]
[...]
The sorts of serious abuse we feared have not (at least, no yet as of November 2000) actually materialized.The Cathedral & the Bazaar :: Revenge of the Hackers, Eric S. Raymond, page 178
A couple years ago, I almost skipped this part of the Revenge of the Hacker manifesto since it was evident that Microsoft would not try something that obvious. I was so wrong…
It did took them a couple of years to figure it out, but then, they push forward evenement like {Open Source} Heroes happen here where you can [...]
Order you own Hero Hack Pack and get started with Open Source. Each Hack Pack contains a trial copy of Windows Server 2008 and Visual Studio 2008, plus a chance to win a free pass to OSCON 2008!
I am well aware that free software doesn’t need to be free "as in beer", yet It kinda make me sad that people can pull that kind of scheme and still sleep at night. They are clearly piggy backing on a wave they don’t control and have nothing to do with. You can receive a trial version of a software that is closed source, which is non-free, to run a software that is closed source, also non-free to program open-source. Hey, you can’t say no to that offer!
Why don’t they start their own little revolution and try to gain leverage… because, lets be honest here, Microsoft doesn’t even own any open source application. They have the money and the technical talent to, once again, shake the whole computing world with innovation. Their loud "we support open source" is based on money they gave their own concurrent while keeping their standard closed - they really should not play on this level.
Port 25, Microsoft blogging platform for their open source community, is more of a joke than anything else. They have some freak labs, acces to some of the best programmers in the world and … their messages all look, at least, to me, like :
Microsoft is involved in open source since Apache web server run on their server. That’s without saying anything about the price/availability of IIS… And this is no joke, the last few post from port 25 are installation procedures for Apache.
Microsoft is involved in open source since they have a partnership with XEN (this is more present on their website than the port25 blog). They still push forward their own virtualisation system, but with VMWare currently stealing all the "high value target" and Xen taking everything else…. not much left for VirtualPC.
But, there is one thing that does intrigue me : Shared Source at Microsoft. Well designed, this program could have some leverage, but I guess that everything is released under a reference only licence, that a very big entree fee is required and that your soul must be sold to the devil, or something like that.
And when they really want to go open source, the underlying "help us get more money" is so evident that their isn’t even any fun reporting it. The project look cool though : Singularity.
Update: I’ve been pointed toward this image, sad that it hasn’t been kept up-to-date.
