I’ve been suggesting, going as far a setting a demo system, that their document revision system be migrated from Rational ClearCase to GIT. The current setup support code (developers) and documentation (infrastructure, products, management teams).

Change rational: I don’t have anything specific against ClearCase, but their licences are ending and, as a corporate decision, they are looking into cutting recurring costs. Going for a widely deployed & supported open-source/free software solution looked like a safe bet. While not involved in the decision process, I suggested GIT as an alternative to whatever they could be thinking of. I’m that kind of ‘consultant’, always with an opinion on everything, hmm… computer related.

The bug is: Why the hell did I suggest GIT ? I’ve entered a weird mental pattern. Through the years, I’ve convinced myself that peoples would be supported by an outdated system (ie: CVS), then upgrade to a more recent one (ie: SVN), then change their ways to a true developer friendly revision control system (ie: GIT).

GIT and SVN aren’t the same type of product.

Sometimes, you DO need a central repository – mainly if all you are doing hard-to-merge binary file commit. Sometimes, you DO need a locking mechanism. Think of 10 employees working on different part of the same Visio document. A project complete and you get 10 ‘branch’ merge request ? You’ll want to kill yourself. At least with locking, they will fight amongst themselves!

But the GIT vs SVN vs ‘whatever‘ isn’t the point here. The problem was that I took a ready-made solution (that I deployed long ago) and went as far as proposed it ‘to the outside world’. That would have never happen while I was leading the infrastructure decisions of 5+ startup (and maintaining a lot more). I’m losing my edge. It’s time to start posting a bit more (on this blog) and bouncing ideas off my entourage.

Then with games:

Then with cloths:

And… There’s more coming! With Christmas I’ve got a cart of interesting items LabsPhoenix is going to buy for the ‘home’ lab. At the very least, a new mac-mini, a Razer mouse, a eSATA external enclosure (with the HD), 2 SSD drives, 2 Infiniband network card (with associated cables), OEM board (ie: Soekris, Wrap, …) with gigabytes connections and an ubiquity setup (3 antennas + router board). If I’m lucky, I might even buy this 24U enclosure I’ve been watching for a few months. There should also be a couple spending session for the LabsPhoenix own project (3x New Servers, vSphere licences, …) but thats another thing altogether.

– Les Laboratoires Phoenix welcomes a new managed client, at the same time as I got my two first contractual employee (with enough job to drive them for years).
– I’ll be giving a talk at ConFoo, March 12th, called ‘Massive Scalability’. Be there, its going to be a pretty good one.
– I’ve been mandated to write another article for the European edition of Linux+DVD. Deadline is in a couples days.
– I’ve started dancing classes. (No comments please ;-))
– With the wedding happening soon, we are totally swamped with stuff to do. From food tasting to getting whatever I will wear, going through hotel reservation, decoration choices… By themself, each task is quite easy to manage, but add to that the fact we are doing most of it remotely and that Catherine schedule is just crazy.

At least, this morning, I’ve got a 20 minutes break, waiting for the bus thats going to take me to Montreal – on yet another ‘business trip’.

But, beside everything that might be happening with my business (work log never been that full, started to get employees), I decided to take 2 minutes for a big fact of life (for sysadmin/management). Watch it, watch it: Nobody (especially YOU) care about backup. You care about successful restore. There! Now, I don’t want to hear about how great your backup are, if you don’t do regular restore (even as test), they’re not worth anything.

Reread those last sentences. Important point in a sysadmin life.

PHP version 5.3.1 was just released. This release contains a patch for a denial of service condition we’ve reported on 27 October 2009. The problem is related with PHP’s handling of RFC 1867 (Form-based File upload in HTML).

Source: http://www.securityfocus.com/archive/1/507982

Exploit already on PacketStorm…

The next few hours will see little sleep and lots of action ; More precisely I’ll be deploying lots of hardware (2 IBM SAN, 2 core servers, 2 switchs, 2 APC, 5 branchs servers – supporting up to 20 ‘leaf’/virtual servers), and then somes (3 couples of 2 systems in high redundancy (wackamole IP ‘fencing’, shared-storage through DRBD). All that will go in ‘my’ new 48U cage @Hypertec (old nortel building) to act as a demo for some clients.

Once that’s completed, the true fun start: A very big part of this infrastructure is going to be self-healing, failure resitant and high performance. We are speaking of :

This is going to be solid, scalable, fast : the holy grail of a lot of service provider that are aiming at automatization of their ‘hosting’ business. The result of a lot of planning and testing ; the cutting-edge of cloud-computing.

1) what’s wrong with:
void f() {
char buf[2048];
gets(buf)
}

void main() {
f();
}

(note ; this is the modified version of this function. Read comment 1 on this blog post for more info)

2) With current systems, IPV6 is becoming standard feature. What security problems do you see with that statement and how would you go to secure an IPV4 network knowing those problems ?

3) There have been quite a few problems with SSL theory and OPENSSL implementation in the last few years – please, name a few and explain them.

4) What is entropy or prng ?

About Hypertec

As a rule, never visit somewhere without background info : Hypertec-BCDR (Business Continuity and Disaster Recovery) (they also use the name Hypertec-AS for the french version) is the hosting, datacenter & high availability services division of the Hypertec Group. The group look like an umbrella corporation which also hold the Hypertec Systems division (kind of a computer retail shop). The exact financial details are private (the group is private / NOT available in the stock market), but from what I’ve heard, the whole group have about 120+ employee and a sales figure of about 20M$/years. Those are very rough numbers, I could be totally off the track, and include all their activities (don’t know for the data center aspect only). There seems to be office in a couple locations (Montreal, Quebec, Ottawa, Toronto… ).

So, its quite strange that I haven’t heard about them… especially since they are located inside the old Nortel building in Saint-Laurent. I’ve also contact friends about them, and they were virtually unknown!

The visit

… and this is why I’m doing a blog post on them: because Jonathan Ahdoot, sales manager, walked me through their data center and I must say, he was able to impress me. The main surface is reserved for tier-4 dedicated cages to which you can add a small quantity of tier-2 rack (about 60) setup. As a reminder, in datacenter higher tier speak of better quality (scale from 1 to 4 – as defined by the uptime institute)(different from Internet peering tier).

The visit make clear quite fast why I hadn’t heard about them : they fish for the big ones and government (which can be considered a big one) contracts. They have rooms for rent that act as office away from office for couples of days, they have a 10 posts technical room, a cafeteria (which can become 24h) and … behold: a lounge. Yes ! a true lounge with satellite TV and couches. How many time would I have given everything (my clients own ;-)) for a nice couch while waiting for a file copy between the SAN and the server I’m restoring @ 2h AM. They also make their conference room available to clients (which is another nice feature, especially for office-less consultant (me!)).

I’m far from being a data center specialist: I build infrastructure and I rack them somewhere – this is mainly what I do. So I cannot go into big details about all the nice features the data center seemed to have or in the small point why it might not be as great as I think. However, there is one thing that did impress me: There is 5 flywheel energy storage system in the main engineering room, all being provided by electricity (Hydro) and hooked on a generator. This was also the first time I’ve heard about flywheel energy storage (FES), but I do find the idea quite neat. There must be a lot of energy lost through friction (even if they are in vaccum), but it does look like a system way more secure than batteries (UPS) for data center. Secure as in : I’ve already been screwed twice by “this was a planned maintenance and the ups didn’t turned on, or the tech turned off the wrong line”.

But the sky is not totally blue: Since they do seem to target tier-4 clients, they lack a bit of the standard facility we require in a tier-2: renting 48U racks rarely leave you the space for screen, mouse, keyboard, screwdriver… you expect them to be readily available on site. From what I’ve saw, they were either lacking or in bad shape (tier-2, again… the tier-4 look awesome). Anyway, if you got a cage (with multiple rack) and you don’t have space for tools, you have others problems.

Anyway, a couples contracts will require me to be in data center for the next few months (migrating 35U, deploying 20U, re-designing 24U…). So I guess I will be posting more reviews as time goes.

The swekey is a small USB key that secures access to any swekey enabled web sites.
Swekey secured web sites won’t let you login without your swekey plugged to your computer.
The swekey can also be used to secure corporate’s intranet, unix servers access, and database administration.
[...]

Swekey device, Photo by Pascal Charest

usb 2-8: new full speed USB device using ohci_hcd and address 3
usb 2-8: configuration #1 chosen from 1 choice
Initializing USB Mass Storage driver…
scsi10 : SCSI emulation for USB Mass Storage devices
usbcore: registered new interface driver usb-storage
USB Mass Storage support registered.
usb-storage: device found at 3
usb-storage: waiting for device to settle before scanning
usb-storage: device scan complete
scsi 10:0:0:0: CD-ROM Musbe Swekey 1.03 PQ: 0 ANSI: 0
sr1: scsi-1 drive
sr 10:0:0:0: Attached scsi CD-ROM sr1
sr 10:0:0:0: Attached scsi generic sg3 type 5
cdrom: This disc doesn’t have any tracks I recognize!
usb 2-8: reset full speed USB device using ohci_hcd and address 3

Then : Nothing. No auto-mount, no dialog box… Kinda of left there. The partition cannot be mounted…

Going to their website, I learn the official working steps: “BUY” (pseudo-done), “PLUG” (done), “REGISTER” (ugh?) and I’m “READY”. The REGISTER (the step I’m at, right ?) section give me an error of ‘missing plug-in’ from Mozilla Firefox 3.0.14. Ok, browsing “Support”/”Download” inform me of missing dependencies (a software must be installed) to access the device. I download the x64 GNU/Linux version and … hum ?

pcharest@hydra:~/Desktop/swekey$ cat README
Swekey client
This package install:
- the swekey-client command line tool
- the swekey HAL module
- the swekey Mozilla plugin

The swekey-client command line tool gives you the list of plugged swekeys
and let you calculate OTPs with them.

type:
swekey-client –help
to get the available options

To install swekey-client just type:
sudo ./install
or
./install
if you are root

To uninstall swekey-client just type:
sudo ./uninstall
or
./uninstall
if you are root

I have no idea what is an OTP but let say I try installing the client:

sudo ./install

and validate the device is detected:

./swekey-client –list

It work and give me a device ID. Good, at least the device is known by the system. I still don’t know how it should work. I guess I should be installing the Mozilla plug-in the readme mentionned, but… I never found it. I guess the client install worked (and it was included) because after a Mozilla reload, the Manage section of their web page give (or might also be one of the random file I clicked on) :

Registration is not mandatory but it will allow you to disable a lost or stolen Swekey.

So… I don’t really need to register the key… lets try it then (which I’ve been trying to do for quite a long time at this point).

I own quite a few Zabbix servers, so, from the list of supported service :

ZABBIX is an enterprise-class open source distributed monitoring solution.
A swekey integration exists, it is still a patch but you can ask for it if you need to test it.

Ok, still want to test the device – So i try with MediaWiki:

And it started to work well : creation of an account (user+password), then I get asked if I want to bind this account to my Swekey. This won’t allow me to auto-login but will require the key to be present in any computer (with the installed software) to access the account.

Summary: As a summary, I’d say that while it give a boosted security (require the Swekey to log) – it does seem to go a bit over the limit of the permanent fight between conviviality and security. Installing the software is complicated and might be very problematic on system without administrator access… Personally, having tried both, I would prefer Paypal key ID to be integrated to more website. There is no need to ‘install’ the software on any computer and it give you the same added security the Swekey does.

Note: This is a selection of very early draft of a document I’m writing – As such, those are extract of “working notes” and should be considered as beta (not Google definition of beta ; true beta)… lots will change.

[...]
Internet being a jungle (or a city, whatever you find most dangerous), your infrastructure will be preyed upon. it can be by customers requiring services (too much of them can create difficult situations) or by malevolent individuals wanting to see your service off Internet.
[...]
Of the techniques available, dos/ddos might be the worst. Here’s a quick non technical theory review:

DOS: Denial of services
For a single attacker, cutting access to your services can be accomplished by solving this equation:
Attacker resource * resource(attack function) > Defender resource * resource(defense function)
The defense against the attack is simply the reverse of the equation. Using decent servers (for processing power) in a decent datacenter (for bandwidth) can help solve this equation to the defender advantage without having to modify services. If it doesn’t work, modifying the defense function (such as implementing a firewall correlating a source IP and the attacker function) will allow required resources for defense to be minimal and thus win the fight.
[...]

DDOS: Distributed Denial Of Services
The DDOS add the dimension of multiple (in the order of hundreds or thousands) attackers systems. This will bypass of most of the standard defenses “resource reduction function” since the resulting traffic will be tangent to a normal usage pattern. Randomly blocking visitor (or user) cannot be accomplished without risking blocking valid one and user pattern analysis is generally resource intensive.
[...]

How to survive DDOS
A lot of services and devices are available to mitigate the attack of a DDOS. Some can be implemented by the end user (server administrator) or by the upstream provider. However, most of them must be deployed as a planned feature, not while the network is under attack.
* drop spoofed/invalid packets at upstream provider (packets with invalid source IP (see RFC 1918), implement ingress filtering (see RFC 2267)) – it is also call dark address filtering.
* prepare rate-limiting function ‘per-vhost’ (if service = webpage), or ‘per-services’, and ‘per-source’.
* implement black hole filtering procedure (an in-line router / packet analyzer able to black hole packet will leave your server doing service computing, not routing).
* request analysis. SNORT is a well know and very good ingress filtering agent that can be used to filter traffic that does not match normal usage pattern.
* enable syn cookie (valid only against syn flood).
* always allow establish connections priority over new ones.
* off load as much as you can (mainly: DNS services in separate network, dropping both is harder).

And I’ll allow a bit of additional informations on this last one, because it is often overlooked and can represent your salvation when you are attacked. Either the attacker will use a specific IP, which is easy to mitigate by changing to any other you reserved for that and changing the DNS (5 minutes downtime is nothing in a major DDOS) OR the attacker is resolving your domain name through your DNS. This latest fact is quite important, because it mean the attack can be mitigated by using geo-localisation on your DNS system : different servers will answers requests from different part of the world. MaxMIND does offer a very up-to-date database of IP/Country and IP/Town ; and using Amazon AWS (cloud computing service by Amazon), new servers can be launched at minutes notice and your DNS (when properly configured) can be modified to provide specific IP “to-peoples-outside-your-normal-business-area”. You don’t even have to involve your upstream provider and you will be able to offset a very big part of the attack (as long as your normal business area is not russia + china).

Or, if implementing those recommendation are not a possibility, there is always services/devices available for sales. Be ready to pay a very big price for them.
[...]