Labs: installation & configuration of GlusterFS as synchronous data storage solution.
By: Pascal Charest, Freesoftware consultant
Date: September, 2008.
Synchronization of files in a cloud environment is a challenge in the path of high-{availability, performance}. From simple load balanced web sites to full-blown applications - some files always need to be in sync. Peoples, for simplicity, rely on asynchronous transfer (ie: rsync ), others deploy bigger solutions (ie: block device replication through DRBD or shared storage through AoE protocol & concurrency management with OCFSv2) or even go for the “lazy” “no-shared-storage” solution through NFS.
To address this problem in the PraizedMedia software stack, I decided to give FUSE based GlusterFS a try. Awesome, really ! The technical knowledge to deploy a basic solution is very very low. The modularity of the program also help to have “something working right now”. This isn’t meant as a direct alternative to DRBD or a good SAN deployment but in my use case, it fit perfectly.
In this lab, I will guide you through the installation of GlusterFS on 2 networked systems. They will be both used as “servers” & “client” for the GlusterFS filesystem. They will be sharing a directory (on both system : /var/production/brick), re-mounted as /var/production/static through GlusterFS. Any write I/O on this directory (of any client server) will be synchronized to the pool. This last feature is called “AFR” (for automatic file replication) and is a module (called a translator) to the GlusterFS file system.
The specificity of my environment is around the file-locking management : I don’t need any. By design, the application will never try to write the same file twice on any of the server.
#Installation of requirement (standard tools)
apt-get install flex bison libfuse-dev linux-headaers-`uname -r` curl
#download of the sources
cd /usr/local/src/
curl -O http://ftp.zresearch.com/pub/gluster/glusterfs/1.3/glusterfs-CURRENT.tar.gz
tar zxf glusterfs-CURRENT.tar.gz
# configure
cd glusterfs-1.3.11
./configure --prefix=/usr/local/glusterfs-1.3.11
make && make install
ln -s /usr/local/glusterfs-1.3.11 /usr/local/glusterfs
So we now have a basic 2 servers GlusterFS systems installed. Lets be honest, that wasn’t really hard! We are still missing configuration files though.
#Editing /usr/local/glusterfs/etc/glusterfs/glusterfs-server.vol
#
# glusterfs-servers definition
# volume definition are on first lvl, other are on second lvl (tabbed)
volume brick
type storage/posix
option directory /mnt/production/brick
end-volume
volume server
type protocol/server
option transport-type tcp/server
option auth.ip.brick.allow *
subvolumes brick
end-volume
#Editing the /usr/local/glusterfs/etc/glusterfs/glusterfs-client.vol
#
# glusterfs-client.vol
# volume definition are on first lvl, other are on second lvl (tabbed)
#
volume remote1
type protocol/client
option transport-type tcp/client
option remote-host 002.praized.com
option remote-subvolume brick
end-volume
volume remote2
type protocol/client
option transport-type tcp/client
option remote-host 001.praized.com
option remote-subvolume brick
end-volume
volume mirror0
type cluster/afr
subvolumes remote1 remote2
end-volume
#Launching services (servers and clients)
mkdir -p /mnt/production/brick
/usr/local/glusterfs-1.3.11/sbin/glusterfsd -f /usr/local/glusterfs-1.3.11/etc/glusterfs/glusterfs-server.vol
mkdir -p /mnt/production/static
/usr/local/glusterfs-1.3.11/sbin/glusterfs -f /usr/local/glusterfs-1.3.11/etc/glusterfs/glusterfs-client.vol /mnt/production/static/
You now possess a synchronized directory between your two systems. Please note that GlusterFS require TCP/6996 port to be open. There is also some improvement that can be done to this setup through adding a locking mechanism & i/o thread - I don’t currently need them, but you might.
Enjoy!
Debugging notes ; after starting the server process you should have a kernel process call glusterfs. All log files are in /usr/local/glusterfs/var/log/glusterfs*. After starting the client, “df -h” should show you your new mount point. Careful with UID/GID (&Permission), there is no such thing as root_squash_fs in GlusterFS yet.
Other notes ; Using Amazon EBS would have been the perfect solution if they did allow multiple servers-volume mount and lets us deal with concurrency / lock problems. But, they don’t.
I’ve been asked about the possibility of harnessing the power “of the cloud” in the context of an email server. For the simplicity of this blog post, I’ll assume the definition of “cloud computing” to be equivalent to “Amazon AWS” offer.
When emails goes in
This is the easy part. Receiving email in an EC2 (Elastic Cloud Computing) instance is as easy as receiving it anywhere. You launch 2 instances in different availability zone, grab 2 IP and change your MX records. With the recent availability of EBS (Elastic blocks store), you even have access to permanent storage for email. In hours (big maximum) you have a complete setup supporting fail-over and backup capability (leave your queue/data store on EBS for persistence and snapshot for backup).
Being in a full virtual environment also negate most scaling problems. You dynamically start and stop anti-{spam,virus} scanning instances following the need of your clients and customers. This setup is also very cost-effective: you don’t have to pay for hardware (servers, switches, hard drive..), maintenance, power and all the network management involved in having public infrastructure (bgp, firewall, etc…).You don’t even have to vouch for a long term contract.
For your customer, this represent a very decent offer: speed and latency in the Amazon cloud are very nice - way better than most small technical shop can afford.
Then emails have recipient
Emails are not only coming IN your infrastructure, they - sometime - must be transmitted to other people’s networks. This is where archaic email management style really fail. Emails as a services is a dynasty based on the conception that internet proprieties are big, controllable, static and permanent. This is the exact opposite of what you would get placing an email server inside Amazon Cloud.
You do not control IP space/range - even if, you are leased “1″ IP. This is the big “bug”. You have no idea what peoples do in their instances. Get used to it, your range will be tagged, {grey,black} listed often in dns based blocking list. Very often. White list will refuse your queries, since you cannot vouch for Amazon customer use of the cloud.
Solution, you can still use a smtp server install somewhere else, but… kind of defeat the whole purpose. The financial exercise of fighting dnsbl vs maintaining hardware infrastructure is left to the reader.
Amazon Elastic Block Store (EBS)
Amazon Elastic Block Store (EBS) provides block level storage volumes for use with Amazon EC2 instances. Amazon EBS volumes are off-instance storage that persists independently from the life of an instance. Amazon Elastic Block Store provides highly available, highly reliable storage volumes that can be attached to a running Amazon EC2 instance and exposed as a device within the instance. Amazon EBS is particularly suited for applications that require a database, file system, or access to raw block level storage.
source: Amazon AWS
Cloud Computing: Coming out of the Fog
By Gerrit Huizenga, from IBM @ Ottawa Linux Symposium
Its very strange to hear a talk about cloud computing from an IBM employee since they have not yet shown any serious stats about their Blue Cloud system. Still, it’s a good review of the cloud computing field.
Despite the fact that the presentation is built upon the statement that cloud computing isn’t about servers provisioning, it clearly revolve around the two following points of views:
From outside of the cloud :
You want your applications (complex systems) to be deployed fast, with next to no configurations to be done. 3Tera system is shown as a “good” way of doing that - personally, having built something similar for a client, Amazon EC2 is also a good contender for the title. From my POV, this is really about provisioning and the capacity of building virtual appliance.
From inside of the cloud :
You want to have a fully (automatic, dynamically) managed data center. Technology is already there. This is ALL about server provisioning.
The presentation moved from this “reviewing definition” to “why it is presently not everywhere” and “how to build a general interface for cloud system”. Guess this speaker is reading the cloud computing mailing list at Google Group since those are hot subjects right now.
As a closure (this wasn’t mentioned in the presentation) : here is a quick stock index of related corporation.
Executive summary : Give me 10k$, a month, 3 poweredge servers, a gigabytes capable switch and I’ll build you a scalable cloud infrastructure ;-).
And, the post:
Last year dominant meme was "Virtualization". Since you can’t have the same focus for two consecutives years (must be a law about that written somewhere), they (for various definition of "they") had to enhance it. Here come "Cloud Computing".
Cloud computing, as defined here, here, here, here, here and… is still in condensation phase. Ideas appear and usability should emerge… soon.
While this is concentrated fun for theorician, I would prefer a more technical discussion. I am aware of Montreal based corporations currently studying Cloud/Grid systems. One of the next big player, in Montreal/North-Eastern USA, might be iWeb Technologies - they already have hardware, a customer base and so much to gain on the scalability aspect of cloud computing. Think about dynamically closing unused shared hosting system and relocating instance in relation of their impact on server resources. A lot of other corporation are also present in the field.
But I don’t have access to the same quantity of hardware as they have, so lets see what is available / can be built in my small lab.
SunGrid Engine, as an online service, no hardware needed, have more of a grid heritage than a cloud computing future. Application are launch, run, and a specific output is gathered and sent. The list of application, while impressive, doesn’t have "Apache" - this is a system meant for raw processing power, not offering services.
IBM’s BlueCloud is still more of a vapor cloud around a press release than anything that has to do with computing. Though, I’m sure it look awesome in their lab. But, again, I’m sure their whole lab look nice.
3TERA’s apps logic does look neat, yet, there is no public price tag. This also look like the kind of system that is built around templates "which should not be modified". I have no idea how the system reliability goes when customization are made. And I won’t know… no price tag is a straight no-go for me. If you are ashamed of your pricing model, there is a problem. If its not the case, there is no reason not to show "figures".
Another online service, Amazon AWS (EC2 & S3), is one of the current market leader. Based on XEN, you can have a remote instance for couples of cents an hour. The main concern with EC2 is the volatile aspect of the storage, which kinda defeat most of services real purpose, dealing with informations.
So ?
While I don’t have much hardware, I still have a labs of 4 dev + 2 prod systems. Lets see what can be done. Lets design a home brewed cloud infrastructure.
Nodes types
ConfigNode :
role : CNode is a standard Debian sys. It is the DHCP + PXE + tftp server. It hold the HardwareNode kernel. All cloud configuration happen on those systems.
min : 1 sys.
normal : 2 sys.. {Primary/Slave}. with software raid + drbd + heartbeat.
Scalable: no use. 2 systems is more than enough, there isn’t really any CPU/Network load.
StorageNode :
role: SNode is a network booted GNU/Linux system. It serve AoE devices on the network. All nodee (except ConfigNode) use SNODE as root filesystem.
min: 1 sys.
prefered: 2 sys, {Primary/Primary} with software raid + drbd. MD-device Multipathing is required from clients to preserve the P/P coherence and reliability to network failure.
Scalable : This is a building block. The limit of SNODE is defined by the network fabric speed.
HardwareNode :
role : HNode is a network booted GNU/Linux/XEN-dom0 system. It use a SNODE array as its root filesystem. This is where INODE will be launched. This node is diskless.
min: 1 sys.
prefered: no limit.
Scalable: This is a building block of the infrastructure. The limit of HNODE is defined by the acceptable speed of the root file system located on a SNODE.
Instance :
role : an Instance is a network booted GNU/Linux/XEN-domU system. In the presence of VT technologies, it can also be an unmodified guest operating system (hear full-fledge GNU/Linux or Microsoft Windows). It is started on a specific HNODE using SNODE resources.
min : 1 sys.
prefered : no limit.
Scalable : Currently limited to the underlying HNODE ressources.
Summary : Using a specific configuration node we start a StorageNode and an hardware node. Then, once the infrastructure is "running", Instances can be dynamically started on HardwareNode.
Since Instances are XEN/domU based, running on shared storage, they can be migrated LIVE without downtime between HardwareNode. A ping to the virtual instance would not fail, even in the middle of the live migration.
Since HardwareNode are network booted, adding new server is as simple as adding the MAC addrs in the dhcp configuration and tagging it as HNODE. As long as system are able to PXE boot, it is really a matter of minutes to add new nodes.
Since HardwareNode are network booted with remote root filesystem, they do not need to have hard drive. This remove one of the main failing pieces of current infrastructure. There isn’t much to fail in a server with only a CPU, memory and network interfaces.
The storage aspect is taken care of Storage node where good raid + redundancy + hard-drive snapshot can be used to control the environment. The only limit on the number of storage node is the network… but then, link aggregation is your friend.
Since multipathing is used, with DRBD and AoE, a storage node can be shutdown without impacting running instances.
The creating of new InstanceNode is easy : either copy an instance or debootstrap a new system. Doing something similar from 3Tera would be fairly easy at this point, creating template and preparing configuration interfaces/scripts.
What now ?
Took me a week-end day. I have a running ConfigNode, StorageNode (using NFS, but AoE /multipathing is next), HardwareNode and an Instance. Much of the time was spent waiting for kernel compilation and deploying distcc on my lan. Had little problems pxe booting a dom0, but found a fix.
I wonder what someone working full time could accomplish in a month…. Someone want to pay me to see ? ;-). Haaa.. and it would cost you (in addition to my salary for a month) a copy of Nicolas Carr’s BigSwitch book (which I haven’t read yet, but plan to, as soon as I can get my hand on a copy). I can even do a little presentation first for some kind of financial retribution (yeah, money drive me ;-)).
Seriously, such setup would be fully scalable and so easy to dynamically configure through scripts/GUI. One of the limiting factor is the CPU/Memory resources limit that instance have since they are linked to a single hardware node but if Xen (as a commercial solution) is able to create a resource pool, I’m sure there is way to go around that limitation.
Jeez, using VT enabled hardware node, you could even start Microsoft Windows instance in your cloud…
Btw, I know that everything i’ve spoke about can be done through VMWare infrastructure with vmotion (and maybe 3Tera’s Apps) but…. then, think about the fact that a 2 CPU licence for VMWARE Infrastructure is a little bits over 6900$USD….
I just don’t understand why there isn’t more cloud out there. This isn’t all that hard to deploy… not even time consuming…
