Archive - Technique RSS Feed

A walk in the cloud

March 17, 2008 in , , , , , , , with 2 Comments

Executive summary : Give me 10k$, a month, 3 poweredge servers, a gigabytes capable switch and I’ll build you a scalable cloud infrastructure ;-).

And, the post:

Last year dominant meme was "Virtualization". Since you can’t have the same focus for two consecutives years (must be a law about that written somewhere), they (for various definition of "they") had to enhance it. Here come "Cloud Computing".

Cloud computing, as defined here, here, here, here, here and…  is still in condensation phase. Ideas appear and usability should emerge… soon.

While this is concentrated fun for theorician, I would prefer a more technical discussion. I am aware of Montreal based corporations currently studying Cloud/Grid systems. One of the next big player, in Montreal/North-Eastern USA, might be iWeb Technologies – they already have hardware, a customer base and so much to gain on the scalability aspect of cloud computing. Think about dynamically closing unused shared hosting system and relocating instance in relation of their impact on server resources. A lot of other corporation are also present in the field.

But I don’t have access to the same quantity of hardware as they have, so lets see what is available / can be built in my small lab.

SunGrid Engine, as an online service, no hardware needed, have more of a grid heritage than a cloud computing future. Application are launch, run, and a specific output is gathered and sent. The list of application, while impressive, doesn’t have "Apache" – this is a system meant for raw processing power, not offering services. 

IBM’s BlueCloud is still more of a vapor cloud around a press release than anything that has to do with computing. Though, I’m sure it look awesome in their lab. But, again, I’m sure their whole lab look nice.

3TERA’s apps logic does look neat, yet, there is no public price tag. This also look like the kind of system that is built around templates "which should not be modified". I have no idea how the system reliability goes when customization are made. And I won’t know… no price tag is a straight no-go for me. If you are ashamed of your pricing model, there is a problem. If its not the case, there is no reason not to show "figures".

Another online service, Amazon AWS (EC2 & S3), is one of the current market leader. Based on XEN, you can have a remote instance for couples of cents an hour. The main concern with EC2 is the volatile aspect of the storage, which kinda defeat most of services real purpose, dealing with informations.

So ?

While I don’t have much hardware, I still have a labs of 4 dev + 2 prod systems. Lets see what can be done. Lets design a home brewed cloud infrastructure.

Nodes types
ConfigNode :
    role : CNode is a standard Debian sys. It is the DHCP + PXE + tftp server. It hold the HardwareNode kernel. All cloud configuration happen on those systems.
    min : 1 sys.
    normal : 2 sys.. {Primary/Slave}. with software raid + drbd + heartbeat.
    Scalable: no use. 2 systems is more than enough, there isn’t really any CPU/Network load.

StorageNode :
    role: SNode is a network booted GNU/Linux system. It serve AoE devices on the network. All nodee (except ConfigNode) use SNODE as root filesystem.
    min: 1 sys.
    prefered: 2 sys, {Primary/Primary} with software raid + drbd. MD-device Multipathing is required from clients to preserve the P/P coherence and reliability to network failure.
    Scalable : This is a building block. The limit of SNODE is defined by the network fabric speed.

HardwareNode :
    role : HNode is a network booted GNU/Linux/XEN-dom0 system. It use a SNODE array as its root filesystem. This is where INODE will be launched. This node is diskless.
    min: 1 sys.
    prefered: no limit.
    Scalable: This is a building block of the infrastructure. The limit of HNODE is defined by the acceptable speed of the root file system located on a SNODE.

Instance :
    role : an Instance is a network booted GNU/Linux/XEN-domU system. In the presence of VT technologies, it can also be an unmodified guest operating system (hear full-fledge GNU/Linux or Microsoft Windows). It is started on a specific HNODE using SNODE resources.
    min : 1 sys.
    prefered : no limit.
    Scalable : Currently limited to the underlying HNODE ressources.

Summary : Using a specific configuration node we start a StorageNode and an hardware node. Then, once the infrastructure is  "running", Instances can be dynamically started on HardwareNode.

Since Instances are XEN/domU based, running on shared storage, they can be migrated LIVE without downtime between HardwareNode. A ping to the virtual instance would not fail, even in the middle of the live migration.

Since HardwareNode are network booted, adding new server is as simple as adding the MAC addrs in the dhcp configuration and tagging it as HNODE. As long as system are able to PXE boot, it is really a matter of minutes to add new nodes.

Since HardwareNode are network booted with remote root filesystem, they do not need to have hard drive. This remove one of the main failing pieces of current infrastructure. There isn’t much to fail in a server with only a CPU, memory and network interfaces.

The storage aspect is taken care of Storage node where good raid + redundancy + hard-drive snapshot can be used to control the environment. The only limit on the number of storage node is the network… but then, link aggregation is your friend.

Since multipathing is used, with DRBD and AoE, a storage node can be shutdown without impacting running instances.

The creating of new InstanceNode is easy : either copy an instance or debootstrap a new system. Doing something similar from 3Tera would be fairly easy at this point, creating template and preparing configuration interfaces/scripts. 

What now ?

Took me a week-end day. I have a running ConfigNode, StorageNode (using NFS, but AoE /multipathing is next), HardwareNode and an Instance. Much of the time was spent waiting for kernel compilation and deploying distcc on my lan. Had little problems pxe booting a dom0, but found a fix.

I wonder what someone working full time could accomplish in a month…. Someone want to pay me to see ? ;-). Haaa.. and it would cost you (in addition to my salary for a month) a copy of Nicolas Carr’s BigSwitch book (which I haven’t read yet, but plan to, as soon as I can get my hand on a copy). I can even do a little presentation first for some kind of financial retribution (yeah, money drive me ;-)).

Seriously, such setup would be fully scalable and so easy to dynamically configure through scripts/GUI. One of the limiting factor is the CPU/Memory resources limit that instance have since they are linked to a single hardware node but if Xen (as a commercial solution) is able to create a resource pool, I’m sure there is way to go around that limitation.

Jeez, using VT enabled hardware node, you could even start Microsoft Windows instance in your cloud…

Btw, I know that everything i’ve spoke about can be done through VMWare infrastructure with vmotion (and maybe 3Tera’s Apps) but…. then, think about the fact that a 2 CPU licence for VMWARE Infrastructure is a little bits over 6900$USD….

I just don’t understand why there isn’t more cloud out there. This isn’t all that hard to deploy… not even time consuming…

.cloud computing

load-balancing

March 12, 2008 in , , , with Comments Off

Note: The setup I’m about to describe is not only UNSUPPORTED by MySQL but also clearly "Not a good idea", following posts all over MySQL sites. I’m not kidding : Here (second paragraph), here (big orange warning in the center) and so on. Read the conclusion of this post, it also bring one important point.

Yesterday, while helping a friend with a server, I’ve decided to improve my lab setup.  The configuration in question involve dual LAMP system accessing shared-storage system to deliver webpages. The load-balancing part will be secured by PEN software on my router. I’ve integrated a small php function to show which server is answering the request, this information is available under my LinkedIn tag in the last sidebar.

So I have 2 " servers" systems, MD devices + DRBD 8.2.5 + OCFSv2 + Apache2 + Php5 + Mysql 5.0.51, with dual network interface {one toward the LAN, one as a cross-over}.

Over the cross-over, DRBD sustain a primary/primary array, with OCFSv2, creating a shared-disk device. Procedures to create such setup is available on mass-storage.org. We still need to create the rest of the infrastructure, this will be a 3 steps job.

Load-Balancing

The current setup is a load-balanced environment, there isn’t much high-availability value since our load-balancer is going to be the point of failure. We will be using PEN load-balancer on the router. Why ? Because it is very easy to configure and was the fastest to install. In time, I’ll post a true walkthrough to get an HA infrastructure, but not today.

apt-get install pen
pen -T 10 10.0.0.25:80 10.0.0.26:80 10.0.0.27:80

And I NAT all my inbound web traffics towards 10.0.0.25, which, in turn, load balance over 10.0.0.26 and 10.0.0.27. "Connection tracking" is kept for 10 seconds, giving the same servers for most of the sessions. This isn’t really needed, but allow me to give an excuse for the "DO NOT hammer my server to play with the load-balancing ;-)" part of my speach to my friends.

Apache2 + Php5

This really is the easy part, we configure both Apache service toward the same files, it work. If you intend to do heavy use of sessions (or small), you better use a modules to relocate them on you shared storage.

MySQL 5.0.51

Here goes the tricky part. MySQL doesn’t like shared storage, so you better really know what you are doing. The first thing we do is change some default configuration:

in /etc/mysql/my.cnf, we need to create the external-locking mode

skip-external-locking  -> external-locking

To remove the caching:

delay-key-write = OFF
query-cache-size = 0

And we need to be sure we are using MyISAM tables. Innodb is really a big NO-GO here. Kinda cool to know that MySQL default mode/tables are already using the MyISAM storage engine.

Conclusion

If you have any services which require you to think about load-balancing, but are ignorant enough to base decisions on the lab-notes I post on this blog, I would advice you to get in touch with a storage/network consultant. Won’t cost you that much and you will have a written confirmation that your infrastructure is going to work for your given application.

Getting technical – When bugs strike

March 11, 2008 in , with 65 Comments

Because most people does not care about my camera / lens, i’ll be giving you a melting-pot post I’ve been working on:

1)
This might be one of the worst bug affecting Microsoft Windows for years. I’d expect to see this news in a couple of week, on april 1th.

The description of the bug goes as follow:

A bug has been discovered in the way that Windows Home Server manages file transfer and balancing across multiple hard drives. In certain cases, depending on application use patterns, timing, and the workload that is placed on the Windows Home Server-based computer, certain files may become corrupted.

source: Microsoft Knowledge base KB946676.

And the fix:

Until a software fix for Windows Home Server is available, users may choose to act to help limit the possibility of being affected by this issue. As a precautionary measure, users should use Windows Explorer or a command-line tool to copy files to and from the Windows Home Server-based computer. Do not use applications to directly edit or change files that are stored on the Windows Home Server-based computer.

Emphasis is mine,
source: Microsoft Knowledge base KB946676.

Following the link, you will see a list of application you should NOT use on this operating system. Notable one include Microsoft Excel, Mozilla Thunderbird, Adobe Lightroom and Adobe Photoshop….

2)
Speaking of bad bugs, here is another one concerning device encryption.

A paper from the Center for Information technology policies, Princeton university describe the risk of cold boot attacks against data remanence in RAM. The paper is kinda technical and doesn’t disclose the exact tools used to retreive informations. To do a very blunt summary : Your ram isn’t as volatile as your might think. This is a disastrous news for all device encryption scheme, and kind of a very good one for all forensic analyst.

Weasley Mcgrew, from McGrew Security, has made available a tool that can be pushed on a removable media and be used to rip a copy of the ram fingerprint. I wouldn’t bid anything on the fingerprint size of the tool, but it does seem kinda small.

I’ve tried "msramdmp" on my Asus-EEE from the SD-card reader. Took me about 10 minutes to create the ripping device (make, sysconfig and libtool were not installed), but it simply work. Does take a long time to rip RAM though, but again, I was writting to SD Card interfaced through USB… around 1.4MB/s. But time isn’t an issue in labs and off-line preparation can be quite as long as you want if you get some R-134A (to cool your memory, only needed if you want to leave the computer closed while you prepare).

Leaving you on a quote from the Princton paper FAQ, have nice nightmare:

We have demonstrated practical attacks against several popular disk encryption systems: BitLocker (a feature of Windows Vista), FileVault (a feature of Mac OS X), dm-crypt (a feature of Linux), and TrueCrypt (a third-party application for Windows, Linux, and Mac OS X). Since these problems result from common design limitations of these systems rather than specific bugs, most similar disk encryption applications, including many running on servers, are probably also vulnerable.

Des nouvelles

February 28, 2008 in , , with 1 Comment

C’est dernier temps, il y a tellement d’événements qui affectent ma vie !

J’ai passé, il y a deux fds, de beaux moments avec ma copine.  J’arrive d’une conférence de VMWare (avec les gars d’Equallogic et leur SAN iSCSI) au Hyatt de Montréal. Je désir visiter New York bientôt. J’ai recu des offres pour sponsoriser ce blog. Mass-storage est toujours en construction et je travail encore trop d’heures par semaine ;-). J’ai detruit mon compte facebook.

Je reviens avec un vrai post demain. Je suis presentement dans la phase finale de preparation de 2 contrats vraiment intéressant, je vous en reparle quand je termine cette grosse grippe.

High performance SSH/SCP

February 18, 2008 in , , , , with Comments Off

The article here, about high performance optimization to the SSH process, has been doing quite a few waves. I still post it on my blog since some people might have missed it (shame on you!).

In a very short summary : lets say that it’s a patch to allow for dynamic resizing of the internal flow control buffer in OpenSSH. It also multi-thread the crypto part. In other words, you get speed, you don’t lose much, on link where you own both end-point.

You still need to be a bit careful, there is a lot of talk of why it WON’T be integrated in  the official OpenSSH release. But hey, if Leif Nixon and Robert G. Brown like it, jeez…. I guess it must not be that bad.

Note : For those who don’t know those names, they are kind of "local" heros in the Beowulf clustering field.

Page 1 of 2112345»1020...Last »