26
Lets dwell into the technical world that had me captive for a couples hours, last week:
The installation of NGINX & Phusion Passenger should be very straight forward and no cause for a blog post. Except, and the fun start here, if you are compiling both from source, inside Amazon AWS, on a small EC2 instance (32 bits) and your dom0 is 64bits (no way of knowing, but most should) and Amazon gave you a slice of memory over the 4gb of normally addressable slot (small windows of opportunity, but still). How bad can it get ? Well, you will be flooded by the dreaded “4gb seg fixup” error message in your log files and your ruby process will drop to an almost standstill speed.
Your god, Google, will advice you to do stuff (which you should have already tried):
apt-get install libc6-xen
echo ‘hwcap 0 nosegneg’ > /etc/ld.so.conf.d/libc6-xen.conf ; ldconfig
mv /lib/tls /lib/tls.disabled
And this is where you will start to despair, because, of the 26 200 Google results for “4gb seg fixup“, 26 000 are either linking to a post asking the same question or answering with one of those two answers and 200 are for 4gb usb key. Neither of which will help the message to go away from your syslog and bring it back NGINX/Ruby to decent speed.
So. At this point, where you are starting to think about wiping everything and starting back from scratch (which won’t help), try this little procedure. The principle is to remove the passenger gem from your system, reinstall it (which will only download the source), modify the makefile, recompile NGINX (which in turn automatically compile the Phusion Passenger module) and take a beer while your system serve ruby pages without (systems) errors.
#> gem uninstall passenger
#> gem install passenger
We have a valid passenger gem source code in /var/lib/gems/1.8/gems/passenger-2.2.4 - version can vary and location is valid for Ubuntu/Debian, but could change on others distro. We will be modifying the optimization flags given to the compiler. Since Phusion Passenger does not accept command line argument and variables declarations, we have no other choices than to modify the rake file pre-compilation.
#> sed ’s/EXTRA_CXXFLAGS = “-Wall #{OPTIMIZATION_FLAGS}”/EXTRA_CXXFLAGS = “-Wall -mno-tls-direct-seg-refs #{OPTIMIZATION_FLAGS}”/g’ /tmp/rakefile
#> mv /tmp/rakefile /var/lib/gems/1.8/gems/passenger-2.2.4/Rakefile
This being done, we will start an NGINX compilation process which will, in turn, start passenger-2.2.4 compilation. Using the -mno-tls-direct-seg-refs will allows us to work arround the 4gb seg fixup error.
#> CFLAGS=”-mno-tls-direct-seg-refs” CXXFLAGS=”-mno-tls-direct-seg-refs” ./configure –prefix=’/usr/local/nginx-0.7.61′ –add-module=’/var/lib/gems/1.8/gems/passenger-2.2.4/ext/nginx’ –with-http_ssl_module –with-http_stub_status_module
There you go.
tagged with : , consultant, freesoftware, labsphoenix, L{AN}M{PR}, nginx, phusion passenger, ruby, sysadmin, technical | permalink
No Responses to “a bit technical ; nginx, passenger, 4gb seg fixup”
-
Anonymous Says:
September 2nd, 2009 at 4:03 pmThanks a lot – it work perfectly and the bug is resolved! They really should correct the RakeFile to take arguments/variables into account.
-
jberryman Says:
November 18th, 2009 at 9:05 pmThanks for posting this. First question: are those initial step that you posted necessary before your solution will work? I have not, for example, done
$> mv /lib/tls /lib/tls.disabled
Second question: what kind of performance hit were you experiencing? Stalling with 5 simultaneous connections? Slow with 200 connections?
Thanks again.
-
Pascal Charest Says:
November 24th, 2009 at 11:02 pmI was speaking about a very very hard hit on performance. Something in the order of 1-2 requests / seconds in a Ruby apps normally able to stream 10-15 rq/s.
I’ve seen the first serie of solutions work in some case, but doing the last one is the best way to be sure its going to be working (making sure everything is compiled with this flag).