Cloudweavers » amazon

Explorer les nuages ; veille technologique pour batisseurs.

pascal.charest — Tue, 03 Feb 2009 04:21:25 +0000

Finally got some time and finished:

“Explorer les nuages ; veille technologique pour batisseurs”
Par Pascal Charest, Feb 2009, édition française

To be published in Linux+DVD, European edition.

ec2 arrive en europe

pascal.charest — Wed, 10 Dec 2008 12:35:00 +0000

Tel que prévu dans les dernières semaines et stipulé dans mon dernier article pour Linux+DVD (qui devrait bientot parraitre), l’infrastructure Amazon EC2 est maintenant disponible en europe.

Cette annonce va assèner un coup dur à tous les autres providers de services de cloud computing qui ne font pas la distinction géographiquement… On parle tout de meme de 50ms de latence de moins si la connection n’a pas a passer par l’atlantique…

Quoi qu’il en soit, je complete présentement un autre article pour une revue qui accompagne un sysadmin dans son premier déploiement dans le réseau d’Amazon. En espérant qu’il arrive assez vite ;-)

Source: Amazon EC2 detail page, Amazon EC2 Description page.

Amazon CloudFront

pascal.charest — Tue, 18 Nov 2008 16:54:00 +0000

The long awaited new feature of Amazon AWS is now available: CloudFront. To be short, its a content delivery network based on their S3 storage solution. Should have some benchmark later in the week

Cloudbursting

pascal.charest — Thu, 18 Sep 2008 02:25:00 +0000

Cloudbursting

Technical term referring to the transition to a dynamic state of a network infrastructure as an events mitigation process. It entail to both asynchronous and synchronous expansion toward or inside a computer cloud. Each such burst is followed by a period of quiescence before the next burst occurs.

Event such as high workload and extreme traffic spike can cause a correctly configured environment to cloudburst.

The term seem to have been used first, in July 2008, by William Fellows (Principal Analyst @ 451group) in a report and pushed forward by Jeffrey Barr (Technology evangelist of Amazon AWS) on a blog in august 2008.

This is my personal definition, yet I think it summarize the process. Anything that needs to be added ?

glusterfs & synchronous data storage

pascal.charest — Fri, 05 Sep 2008 04:11:31 +0000

Labs: installation & configuration of GlusterFS as synchronous data storage solution.
By: Pascal Charest, Freesoftware consultant
Date: September, 2008.

Synchronization of files in a cloud environment is a challenge in the path of high-{availability, performance}. From simple load balanced web sites to full-blown applications – some files always need to be in sync. Peoples, for simplicity, rely on asynchronous transfer (ie: rsync ), others deploy bigger solutions (ie: block device replication through DRBD or shared storage through AoE protocol & concurrency management with OCFSv2) or even go for the “lazy” “no-shared-storage” solution through NFS.

To address this problem in the PraizedMedia software stack, I decided to give FUSE based GlusterFS a try. Awesome, really ! The technical knowledge to deploy a basic solution is very very low. The modularity of the program also help to have “something working right now”. This isn’t meant as a direct alternative to DRBD or a good SAN deployment but in my use case, it fit perfectly.

In this lab, I will guide you through the installation of GlusterFS on 2 networked systems. They will be both used as “servers” & “client” for the GlusterFS filesystem. They will be sharing a directory (on both system : /var/production/brick), re-mounted as /var/production/static through GlusterFS. Any write I/O on this directory (of any client server) will be synchronized to the pool. This last feature is called “AFR” (for automatic file replication) and is a module (called a translator) to the GlusterFS file system.

The specificity of my environment is around the file-locking management : I don’t need any. By design, the application will never try to write the same file twice on any of the server.

#Installation of requirement (standard tools)
apt-get install flex bison libfuse-dev linux-headaers-`uname -r` curl

#download of the sources
cd /usr/local/src/ curl -O http://ftp.zresearch.com/pub/gluster/glusterfs/1.3/glusterfs-CURRENT.tar.gz tar zxf glusterfs-CURRENT.tar.gz

# configure
cd glusterfs-1.3.11 ./configure --prefix=/usr/local/glusterfs-1.3.11 make && make install ln -s /usr/local/glusterfs-1.3.11 /usr/local/glusterfs

So we now have a basic 2 servers GlusterFS systems installed. Lets be honest, that wasn’t really hard! We are still missing configuration files though.

#Editing /usr/local/glusterfs/etc/glusterfs/glusterfs-server.vol
# # glusterfs-servers definition # volume definition are on first lvl, other are on second lvl (tabbed) volume brick type storage/posix option directory /mnt/production/brick end-volume volume server type protocol/server option transport-type tcp/server option auth.ip.brick.allow * subvolumes brick end-volume

#Editing the /usr/local/glusterfs/etc/glusterfs/glusterfs-client.vol
# # glusterfs-client.vol # volume definition are on first lvl, other are on second lvl (tabbed) # volume remote1 type protocol/client option transport-type tcp/client option remote-host 002.praized.com option remote-subvolume brick end-volume volume remote2 type protocol/client option transport-type tcp/client option remote-host 001.praized.com option remote-subvolume brick end-volume volume mirror0 type cluster/afr subvolumes remote1 remote2 end-volume

#Launching services (servers and clients)
mkdir -p /mnt/production/brick /usr/local/glusterfs-1.3.11/sbin/glusterfsd -f /usr/local/glusterfs-1.3.11/etc/glusterfs/glusterfs-server.vol mkdir -p /mnt/production/static /usr/local/glusterfs-1.3.11/sbin/glusterfs -f /usr/local/glusterfs-1.3.11/etc/glusterfs/glusterfs-client.vol /mnt/production/static/

You now possess a synchronized directory between your two systems. Please note that GlusterFS require TCP/6996 port to be open. There is also some improvement that can be done to this setup through adding a locking mechanism & i/o thread – I don’t currently need them, but you might.
Enjoy!

Debugging notes ; after starting the server process you should have a kernel process call glusterfs. All log files are in /usr/local/glusterfs/var/log/glusterfs*. After starting the client, “df -h” should show you your new mount point. Careful with UID/GID (&Permission), there is no such thing as root_squash_fs in GlusterFS yet.

Other notes ; Using Amazon EBS would have been the perfect solution if they did allow multiple servers-volume mount and lets us deal with concurrency / lock problems. But, they don’t.

Amazon EBS (Elastic Block store) is out!

pascal.charest — Thu, 21 Aug 2008 13:56:28 +0000

Amazon Elastic Block Store (EBS)

Amazon Elastic Block Store (EBS) provides block level storage volumes for use with Amazon EC2 instances. Amazon EBS volumes are off-instance storage that persists independently from the life of an instance. Amazon Elastic Block Store provides highly available, highly reliable storage volumes that can be attached to a running Amazon EC2 instance and exposed as a device within the instance. Amazon EBS is particularly suited for applications that require a database, file system, or access to raw block level storage.

source: Amazon AWS

@ Linux Symposium – Cloud Computing

pascal.charest — Wed, 23 Jul 2008 16:14:24 +0000

Cloud Computing: Coming out of the Fog
By Gerrit Huizenga, from IBM @ Ottawa Linux Symposium

Its very strange to hear a talk about cloud computing from an IBM employee since they have not yet shown any serious stats about their Blue Cloud system. Still, it’s a good review of the cloud computing field.

Despite the fact that the presentation is built upon the statement that cloud computing isn’t about servers provisioning, it clearly revolve around the two following points of views:

From outside of the cloud :

You want your applications (complex systems) to be deployed fast, with next to no configurations to be done. 3Tera system is shown as a “good” way of doing that – personally, having built something similar for a client, Amazon EC2 is also a good contender for the title. From my POV, this is really about provisioning and the capacity of building virtual appliance.

From inside of the cloud :

You want to have a fully (automatic, dynamically) managed data center. Technology is already there. This is ALL about server provisioning.

The presentation moved from this “reviewing definition” to “why it is presently not everywhere” and “how to build a general interface for cloud system”. Guess this speaker is reading the cloud computing mailing list at Google Group since those are hot subjects right now.

As a closure (this wasn’t mentioned in the presentation) : here is a quick stock index of related corporation.