AoE + OCFSv2 (storage fun, part 3)

February 13, 2008 in , , , , , with 90 Comments

NOTE: Now on www.mass-storage.org

I have a running {DRBD 8.2.4 (P/P) + OCFSv2} 2 nodes cluster. More Info here.

Kinda nice for small workload (think load-balanced webservers, fileservers, sql servers (careful, Oracle is OK, mysql need specific configuration for external lock)) but a bit on the limited side as scalability goes.

Removing the storage aspect from applications servers is the way to go. This is what SAN are for. Lets modify my two nodes (ruby and crystal) cluster to allow dynamic growth in term of application and storage nodes.

For this test, i’ll be bringing a third and fourth system : "jade" & "glouton", two debian based fileservers.

The setup will be as follow :

(jade & glouton): SAN target, exporting device through AOE
(ruby & crystal): SAN initiator + application server

Lexical info: an Initiator is a SAN client, whereas Target are servers.
  
Exporting through AoE

(glouton&jade)# apt-get install aoetools vblade
(glouton)# vblade 0 1 eth0 /dev/sdb1
(jade)# vblade 1 1 eth0 /dev/sdb1

Note 1: My current setup make me use the above configuration. In a true production environment dual NIC would be preferred (using linux bonding module) & the exported device would be a MD array. There is also a lot of fine-tuning that can be done along the way (jumbo frame, multipath algo, scheduling algo, kernel hacking … )

Note 2: I would against going with an integrated list of MAC addrs. in the vblade export command. The option is present, but the list is then static. Using ebtables seem to be a valid alternative since it can be dynamically modified.

Importing through AoE

(ruby&crystal)# apt-get install aoe-tools
(ruby&crystal)# modprobe aoe

If the file systems are already exported (from jade & glouton), they will be automatically available in /dev/etherd, or else, use "aoe-discover".

Creating MD device for redundancy.

(ruby&crystal)# apt-get install mdadm
(ruby)# mdadm –create /dev/md0 -l1 -n2 /dev/etherd/e0.1 /dev/etherd/e1.1
(crystal)# mdadm –assemble /dev/md0 /dev/etherd/e0.1 /dev/etherd/e1.1

So at this point, there is two md raid devices which use the same resources. They aren’t mounted yet. Using OCFSv2 will allow us to control the concurrent access.

Still using the same /etc/ocfs2/cluster.conf file (see previous post), we format the raid device in OCFS2 format (note: I now use label, it simplify the creation process of identical configuration files):

(ruby)# mkfs.ocfs2 -L "san" /dev/md0 
(ruby & crystal)# mount -t ocfs2 -L "san" /storage

There we go, once again, a shared storage between ruby & crystal.

Note 01 : This such configuration can easily saturate your network. Do not even try if your max speed is 100Mb/s. This would give awful perfs (trust me!). Go for giga or even infiniband if you can afford it.

Note 02 : There is a lot of alternative options, you might want to check the md module documentation, under multipath. I know I will ;-)

But how exactly is this system scalable ?

Application node : If a system is built with aoetools, md-device support and ocfs2 installed, they can be hot-added to the network. No restart of any running sys. needed. However, It is still a very good idea to modify each cluster.conf file.

Storage node : A system with devices exported through AoE can be hot-added up to a certain point, depending on the underlying raid type (md-device), but I would advice against it. Anyway, you need to take OCFS2 offline to issue a resize command.

Filesystem size : Currently, due to 32 bits adressing, there seem to be a limit @ 16TB for a file system. A good reminder though is that AoE target can export more than one devices….

310-200 would have easier if the professionals would have approved of 650-178 or 70-292 before 70-431. However, one can also go for 70-528 if planning to attempt SY0-101 later.

90 Responses to “AoE + OCFSv2 (storage fun, part 3)”

  1. mazowieckie January 23, 2011 at 2:56 PM #

    I have always knew it.

  2. right size smoothies January 25, 2011 at 5:03 PM #

    This is definitely an incredible site!

  3. Correll January 26, 2011 at 6:02 PM #

    It’s rare for me to find something on the web that’s as entertaining and intriguing as what you have got here. Your page is sweet, your graphics are great, and what’s more, you use source that are relevant to what you’re saying. You are certainly one in a million, well done!

  4. Dale Sweatfield January 27, 2011 at 11:07 AM #

    This looks to be a very active site. How do you keep up with filtering out all the comments?

  5. Arguin January 31, 2011 at 2:08 AM #

    You was interested on my post and asked me about dvd.read this answer.. “Vod Videos on demand in bester Qualität als online Videothek mit all den deutschen Pornostars und auch internationale!Wir haben uns kostenlos registriert und konnten nach 1 Minute 3 Gratis Sexfilme anschauen und wir müssen sagen, das dieses Angebot unschlagbar ist für die , die gern Pornos aus dem Internet in HD Qualität laden wollen.”

  6. Malcolm Hoye February 3, 2011 at 2:50 PM #

    You genuinely are a really informed person!

  7. Janae Caballero February 3, 2011 at 7:40 PM #

    Ta for sharing the link – but unhappily it seems to be unable to pull up? Does anyone have a another link or another source?

  8. Kiersten Vandeputte February 5, 2011 at 11:17 PM #

    I’ve been around for quite a long time, but at long last decided to show my appreciation of your work!Thumbs up, and keep it going!Ta.

  9. Car Hire Malaga Airport February 10, 2011 at 1:44 PM #

    Enormous post on %BLOGTITLE% – and solid domain by the way!

  10. microfiber towel car February 11, 2011 at 4:12 AM #

    Great blog. Many websites like yours cover subjects that can’t be found in magazines and newspapers. I don’t know how we got on 12 years ago with just newspapers and magazines.

  11. Tricia Harken February 13, 2011 at 3:38 PM #

    Hi. I am quite new to this commenting as I normally just read the comments but I should get into the online communities a bit more.

  12. Biura rachunkowe Białystok February 13, 2011 at 5:57 PM #

    It’s very interesting what the person wrote above, but I can not agree with this …

  13. does clomid cause late period February 16, 2011 at 9:47 PM #

    Zune and iPod: Most people compare the Zune to the Touch, but after seeing how slim and surprisingly small and light it is, I consider it to be a rather unique hybrid that combines qualities of both the Touch and the Nano. It’s very colorful and lovely OLED screen is slightly smaller than the touch screen, but the player itself feels quite a bit smaller and lighter. It weighs about 2/3 as much, and is noticeably smaller in width and height, while being just a hair thicker.

  14. Erin Empson February 17, 2011 at 8:14 PM #

    This seems a very active website. How on earth do you keep up with looking at all the comments?

  15. Robert February 18, 2011 at 5:08 PM #

    I love this. finally i got som info..

  16. Liane Afable February 25, 2011 at 1:42 PM #

    verbessern

  17. Zyprexa dosage February 27, 2011 at 12:06 AM #

    Hands down, Apple’s app store wins by a mile. It’s a huge selection of all sorts of apps vs a rather sad selection of a handful for Zune. Microsoft has plans, especially in the realm of games, but I’m not sure I’d want to bet on the future if this aspect is important to you. The iPod is a much better choice in that case.

  18. bathroom storage units March 7, 2011 at 8:23 AM #

    Great job here. I in truth reveled what you had to tell.

  19. bathroom cabinets March 8, 2011 at 8:49 AM #

    I like your logo. I can’t master Photoshop at all.

  20. Seth Taverab March 8, 2011 at 1:26 PM #

    I wouldn’t agree with it.

  21. lorena b March 11, 2011 at 3:53 AM #

    Wonderful stuff.. really full of usefull information. I’ll grab the RSS feed and will stay tuned for more. Oh, and I threw you a StumbleUpon vote ;)

  22. acia berry March 11, 2011 at 4:25 AM #

    Anybody who values the experience and the atmosphere will shortly see why they need to at all times be included on a camping excursion.

  23. Autokar March 13, 2011 at 8:15 PM #

    These tips are so accurate

  24. photoshop actions March 21, 2011 at 3:40 AM #

    Anybody who values the expertise and the environment will rapidly see why they need to all the time be included on a tenting excursion.

  25. Paleo Recipes March 21, 2011 at 5:09 PM #

    I realy respect you guys taking the time to update these websites as a result of they realy help me before i do anything to hastey.

  26. koty March 22, 2011 at 11:01 AM #

    Ja mam jednego kota pospolitego, który śpi w ganku i jeszcze nigdy kot mi nie uciekł. Moim zdaniem lubi swoich gospodarzy :D

  27. contemporary bathroom cabinets March 23, 2011 at 2:58 PM #

    Howdy blogger, thank you for providing this article.. I found it first-class.

  28. Keflex March 26, 2011 at 8:45 PM #

    Zune and iPod: Most people compare the Zune to the Touch, but after seeing how slim and surprisingly small and light it is, I consider it to be a rather unique hybrid that combines qualities of both the Touch and the Nano. It’s very colorful and lovely OLED screen is slightly smaller than the touch screen, but the player itself feels quite a bit smaller and lighter. It weighs about 2/3 as much, and is noticeably smaller in width and height, while being just a hair thicker.

  29. Urządzenia biurowe March 30, 2011 at 11:45 AM #

    Sprzedajemy różne rzeczy od laptopów po meble aby ładnie przyrządzić swoje miejsce pracy.

  30. lady gaga born this way single version April 10, 2011 at 5:42 AM #

    lady gaga téléphone traduction

  31. get twitter followers April 11, 2011 at 8:47 PM #

    The title pretty much says it all.

  32. Lyndon Taula April 12, 2011 at 4:45 PM #

    We are a grouping of unpaid workers and starting a new scheme in our residential area. Your website supplied us with worthy info to help us get started.You have done an proud task!

  33. Leila April 13, 2011 at 3:41 AM #

    lady gaga twitter news

  34. http://andymcknight41.blog.com/2011/04/17/very-hot-tub-addresses-amp-spa-covers-measuring-for-a-substitute/ April 18, 2011 at 5:59 AM #

    This is my first time i visit here. I found so many interesting stuff in your blog especially its discussion. From the tons of comments on your articles, I guess I am not the only one having all the enjoyment here! keep up the good work.

  35. Deborah Channing April 21, 2011 at 11:31 PM #

    You got a really useful blog I have been here reading for about an hour. I am a newbee and your success is very much an inspiration for me.

  36. auction automobile April 29, 2011 at 11:34 PM #

    I just sent this post to a bunch of my friends as I agree with most of what you’re saying here and the way you’ve presented it is awesome.

  37. Eldridge Laremont May 1, 2011 at 9:33 AM #

    You ought to really think about working on developing this blog into a major authority in this market. You evidently have a grasp handle of the topics everyone is searching for on this website anyways and you could certainly even earn a buck or two off of some advertisements. I would explore following recent topics and raising the amount of write ups you put up and I guarantee you’d begin seeing some amazing targeted traffic in the near future. Just a thought, good luck in whatever you do!

  38. bc goverment auctions May 1, 2011 at 2:52 PM #

    You certainly have some agreeable opinions and views. Your blog provides a fresh look at the subject.

  39. online dating May 2, 2011 at 7:24 AM #

    Normally I don’t leave any comments but I just wanted to say how great this site is, thanks for the great info!

  40. bomboniere comunione May 2, 2011 at 2:18 PM #

    I intended to compose you one little remark so as to say thanks again for your personal remarkable secrets you’ve shared on this page. It’s so surprisingly open-handed with people like you to deliver easily what exactly some people would’ve sold for an electronic book to end up making some profit for their own end, most notably now that you could possibly have tried it in the event you desired. Those good ideas likewise acted to provide a fantastic way to comprehend some people have similar eagerness really like my personal own to find out a little more with regards to this issue. Certainly there are numerous more enjoyable periods ahead for people who read through your website.

« Older Comments