Labs: installation & configuration of GlusterFS as synchronous data storage solution.
By: Pascal Charest, Freesoftware consultant
Date: September, 2008.
Synchronization of files in a cloud environment is a challenge in the path of high-{availability, performance}. From simple load balanced web sites to full-blown applications - some files always need to be in sync. Peoples, for simplicity, rely on asynchronous transfer (ie: rsync ), others deploy bigger solutions (ie: block device replication through DRBD or shared storage through AoE protocol & concurrency management with OCFSv2) or even go for the “lazy” “no-shared-storage” solution through NFS.
To address this problem in the PraizedMedia software stack, I decided to give FUSE based GlusterFS a try. Awesome, really ! The technical knowledge to deploy a basic solution is very very low. The modularity of the program also help to have “something working right now”. This isn’t meant as a direct alternative to DRBD or a good SAN deployment but in my use case, it fit perfectly.
In this lab, I will guide you through the installation of GlusterFS on 2 networked systems. They will be both used as “servers” & “client” for the GlusterFS filesystem. They will be sharing a directory (on both system : /var/production/brick), re-mounted as /var/production/static through GlusterFS. Any write I/O on this directory (of any client server) will be synchronized to the pool. This last feature is called “AFR” (for automatic file replication) and is a module (called a translator) to the GlusterFS file system.
The specificity of my environment is around the file-locking management : I don’t need any. By design, the application will never try to write the same file twice on any of the server.
#Installation of requirement (standard tools)
apt-get install flex bison libfuse-dev linux-headaers-`uname -r` curl
#download of the sources
cd /usr/local/src/
curl -O http://ftp.zresearch.com/pub/gluster/glusterfs/1.3/glusterfs-CURRENT.tar.gz
tar zxf glusterfs-CURRENT.tar.gz
# configure
cd glusterfs-1.3.11
./configure --prefix=/usr/local/glusterfs-1.3.11
make && make install
ln -s /usr/local/glusterfs-1.3.11 /usr/local/glusterfs
So we now have a basic 2 servers GlusterFS systems installed. Lets be honest, that wasn’t really hard! We are still missing configuration files though.
#Editing /usr/local/glusterfs/etc/glusterfs/glusterfs-server.vol
#
# glusterfs-servers definition
# volume definition are on first lvl, other are on second lvl (tabbed)
volume brick
type storage/posix
option directory /mnt/production/brick
end-volume
volume server
type protocol/server
option transport-type tcp/server
option auth.ip.brick.allow *
subvolumes brick
end-volume
#Editing the /usr/local/glusterfs/etc/glusterfs/glusterfs-client.vol
#
# glusterfs-client.vol
# volume definition are on first lvl, other are on second lvl (tabbed)
#
volume remote1
type protocol/client
option transport-type tcp/client
option remote-host 002.praized.com
option remote-subvolume brick
end-volume
volume remote2
type protocol/client
option transport-type tcp/client
option remote-host 001.praized.com
option remote-subvolume brick
end-volume
volume mirror0
type cluster/afr
subvolumes remote1 remote2
end-volume
#Launching services (servers and clients)
mkdir -p /mnt/production/brick
/usr/local/glusterfs-1.3.11/sbin/glusterfsd -f /usr/local/glusterfs-1.3.11/etc/glusterfs/glusterfs-server.vol
mkdir -p /mnt/production/static
/usr/local/glusterfs-1.3.11/sbin/glusterfs -f /usr/local/glusterfs-1.3.11/etc/glusterfs/glusterfs-client.vol /mnt/production/static/
You now possess a synchronized directory between your two systems. Please note that GlusterFS require TCP/6996 port to be open. There is also some improvement that can be done to this setup through adding a locking mechanism & i/o thread - I don’t currently need them, but you might.
Enjoy!
Debugging notes ; after starting the server process you should have a kernel process call glusterfs. All log files are in /usr/local/glusterfs/var/log/glusterfs*. After starting the client, “df -h” should show you your new mount point. Careful with UID/GID (&Permission), there is no such thing as root_squash_fs in GlusterFS yet.
Other notes ; Using Amazon EBS would have been the perfect solution if they did allow multiple servers-volume mount and lets us deal with concurrency / lock problems. But, they don’t.
Amazon Elastic Block Store (EBS)
Amazon Elastic Block Store (EBS) provides block level storage volumes for use with Amazon EC2 instances. Amazon EBS volumes are off-instance storage that persists independently from the life of an instance. Amazon Elastic Block Store provides highly available, highly reliable storage volumes that can be attached to a running Amazon EC2 instance and exposed as a device within the instance. Amazon EBS is particularly suited for applications that require a database, file system, or access to raw block level storage.
source: Amazon AWS
OpenSSH client keep a fingerprint of servers to which connections (ssh-client) have been made. Such fingerprints are stored in .ssh/know_hosts and are automatically compared with the current server fingerprint on connection acknowledgment.
Hence, the .ssh/known_hosts file is crucial to system security against man-in-the-middle attack in a networked environment. This file is also a very very good vector of attack on system administrator computer and hashing the content of the file is a good practices. Especially with the current wave of big bugs hitting GNU/Linux systems.
The first step is to enable hashing of the new fingerprints:
# cat /etc/ssh/ssh_config
Host *
SendEnv LANG LC_*
HashKnownHosts yes
GSSAPIAuthentication no
GSSAPIDelegateCredentials no
TCPKeepAlive yes
ServerAliveInterval 60
The “HashKnownHosts yes” configuration option is the way to go - it is a general setting affecting all users on your system (Host *). If you don’t have access to the central ssh_config option, don’t forget you have personalized user setting in .ssh/config.
This enable the hashing of future fingerprints. To modify your actual file, use the following ssh-keygen command. Your unmodified know_hosts will be save as know_hosts.old .
# ssh-keygen -H -f .ssh/know_hosts
Have fun, stay safe.
The subject of the week seem to be information security, so I’ll get on with another post that should keep you awake - well… if you are a system administrator doing his job.
With the DNS vulnerability, we thought that this was the bottom of the barrel. Yet researcher are always able to amaze us: Attacks on package managers.
Ok, I must admit that it isn’t as bad as others bugs. Most of the risk can me mitigated by requesting meta-data verification (openssl) from your packager source or selecting a trusted repository. Still - I’ll verify all my sources…
2 years ago, I’ve posted an open letter to sysadmin in which I whined against the incessant flood of system administrators whose only background was firewall configuration … this might be considered as a follow up.
In the last few weeks, I’ve seen the extent of the ignorance of some consultant/sysadmin. Lets start with one example:
On MLUG, I’ve had trouble convincing peoples that RAID arrays are not “backup system” - they are the representation of an high-availability/high-reliability solution for information system. I’ve receive couple of private email on the subject saying that I was wrong and that I should read some more on the system, that they were good backup tech. Ouch!
I really don’t know what to say… this is just too wrong. Lets be honest, I’m in fright that, one day, they’ll be called as experts in court proceeding. This sentence really is tainted by my interest in law, but still… If you are a consultant / intend to be one, please, heed my words.
Being a computer consultant is a job full of interesting challenges, and the pay is great (especially in Montreal where the cost of living is so low) but your name is the one thing you must protect at all cost… I’ve been able, in the last 5 years (taking exception for 1 year working @ SFL) to easily get contracts, without ever doing any promotional work. Currently there is so much demand that, even me - selling “myself”, I had to take a partner (which I’m very interested to be working with). So: Please, think before you post something on a public forum / email another consultant … I’m nice enough not to place names, but others might not..
Btw - If you are searching for a job in the domain, you can get in touch with me. I can give you a couple names that are in need… but be ready for a harsh response if you fail to answer some basic question before I refer you…