In the midst of all that, I’ve got a mandate involving MySQL databases and the import of a big chunk of data (about 8gb raw). Yeah, I know, some peoples reading my blog will know whom I speaking of. The data come with the schema, the load data queries and the CSV files. Should be pretty easy, long, but easy.

Got that one wrong. I’ve been bitten by all the bugs that were on the road:
– MySQL LoadData (from CSV) use the TMPDIR variable (which was on root filesystem) and LoadData generate a temporary queries in that directory. In the present case, the 8GB raw csv represented over 40GB of required tmpdir space.
– MySQL Replication doesn’t live really well with importing large amount of data. The first (failed) import broke all the replication (ring of masters + slaves).
– and…

MySQL’s innodb log file cannot shrink its size. That’s pretty simple as a rule, but it creates lots of problems when you have a failed import due to free space issue. Whatever you do, MySQL will NOT give back this hard drive space unless you dump all and re-import. The current case being a database of over 20GB, it’s really a no-go (and since you are lacking free space, dump is a no-go).

The important things to take away from that encounter is: always enable innodb_file_per_table on MySQL. The performance overhead will be minimal (very) – you might even get better performance depending on your file system tolerance for fragmentation – AND – you will be able to drop table and regain the disk space… even if its only for a couple hours. The drop of a 10GB database should always give you some free space ;-). Question is: why haven’t MySQL put this setting as default ? Its been around for long enough… is stable and does help the general user (a sysadmin) experience.

The next few hours will see little sleep and lots of action ; More precisely I’ll be deploying lots of hardware (2 IBM SAN, 2 core servers, 2 switchs, 2 APC, 5 branchs servers – supporting up to 20 ‘leaf’/virtual servers), and then somes (3 couples of 2 systems in high redundancy (wackamole IP ‘fencing’, shared-storage through DRBD). All that will go in ‘my’ new 48U cage @Hypertec (old nortel building) to act as a demo for some clients.

Once that’s completed, the true fun start: A very big part of this infrastructure is going to be self-healing, failure resitant and high performance. We are speaking of :

This is going to be solid, scalable, fast : the holy grail of a lot of service provider that are aiming at automatization of their ‘hosting’ business. The result of a lot of planning and testing ; the cutting-edge of cloud-computing.

Note: This is a selection of very early draft of a document I’m writing – As such, those are extract of “working notes” and should be considered as beta (not Google definition of beta ; true beta)… lots will change.

[...]
Internet being a jungle (or a city, whatever you find most dangerous), your infrastructure will be preyed upon. it can be by customers requiring services (too much of them can create difficult situations) or by malevolent individuals wanting to see your service off Internet.
[...]
Of the techniques available, dos/ddos might be the worst. Here’s a quick non technical theory review:

DOS: Denial of services
For a single attacker, cutting access to your services can be accomplished by solving this equation:
Attacker resource * resource(attack function) > Defender resource * resource(defense function)
The defense against the attack is simply the reverse of the equation. Using decent servers (for processing power) in a decent datacenter (for bandwidth) can help solve this equation to the defender advantage without having to modify services. If it doesn’t work, modifying the defense function (such as implementing a firewall correlating a source IP and the attacker function) will allow required resources for defense to be minimal and thus win the fight.
[...]

DDOS: Distributed Denial Of Services
The DDOS add the dimension of multiple (in the order of hundreds or thousands) attackers systems. This will bypass of most of the standard defenses “resource reduction function” since the resulting traffic will be tangent to a normal usage pattern. Randomly blocking visitor (or user) cannot be accomplished without risking blocking valid one and user pattern analysis is generally resource intensive.
[...]

How to survive DDOS
A lot of services and devices are available to mitigate the attack of a DDOS. Some can be implemented by the end user (server administrator) or by the upstream provider. However, most of them must be deployed as a planned feature, not while the network is under attack.
* drop spoofed/invalid packets at upstream provider (packets with invalid source IP (see RFC 1918), implement ingress filtering (see RFC 2267)) – it is also call dark address filtering.
* prepare rate-limiting function ‘per-vhost’ (if service = webpage), or ‘per-services’, and ‘per-source’.
* implement black hole filtering procedure (an in-line router / packet analyzer able to black hole packet will leave your server doing service computing, not routing).
* request analysis. SNORT is a well know and very good ingress filtering agent that can be used to filter traffic that does not match normal usage pattern.
* enable syn cookie (valid only against syn flood).
* always allow establish connections priority over new ones.
* off load as much as you can (mainly: DNS services in separate network, dropping both is harder).

And I’ll allow a bit of additional informations on this last one, because it is often overlooked and can represent your salvation when you are attacked. Either the attacker will use a specific IP, which is easy to mitigate by changing to any other you reserved for that and changing the DNS (5 minutes downtime is nothing in a major DDOS) OR the attacker is resolving your domain name through your DNS. This latest fact is quite important, because it mean the attack can be mitigated by using geo-localisation on your DNS system : different servers will answers requests from different part of the world. MaxMIND does offer a very up-to-date database of IP/Country and IP/Town ; and using Amazon AWS (cloud computing service by Amazon), new servers can be launched at minutes notice and your DNS (when properly configured) can be modified to provide specific IP “to-peoples-outside-your-normal-business-area”. You don’t even have to involve your upstream provider and you will be able to offset a very big part of the attack (as long as your normal business area is not russia + china).

Or, if implementing those recommendation are not a possibility, there is always services/devices available for sales. Be ready to pay a very big price for them.
[...]