Experiences using nginx to front-end apache

[Shanti Subramanyam]

I thought I'd document our experiences trying to run nginx to
front-end a couple of apache servers. This is a experimental setup in
a lab to test the max. performance and as such our results may not
apply to you.
The test setup and load is described in
http://blogs.sun.com/shanti/entry/olio_on_nehalem.
We can drive a steady 5000 connection load against each apache
instance using a keepalive timeout of 10 seconds. This results in over
4500 httpd processes and obviously consumes a huge amount of memory.

Our intention in using nginx was to see if we can cut down the number
of apache processes - we expected that the additonal network latency
may not add much to the response time if we could get a corresponding
boost in performance by the system having to handle far less processes
with a reduction in memory to boot.
But in reality, nginx uses http 1.0 to connect to apache causing the
connection to be closed after every request (which I'm sure all of you
know). This results in a severe pressure on port availability as
closed ports go into time_wait for a period of 60 seconds (system
default). Tuning it down to 5 seconds, I could get 300 users to
connect but 500 failed.  So I reduced the time_wait_interval further
down to 1 second. This didn't work either as all the connections
simply went into  CLOSE_WAIT and the clients were stuck without any
ports to connect to and I had to lower the smallest_anon_port. At this
point, I could get all 500 connections to stay up.

After all this struggle and tuning, I'm giving up. I just can't
imagine how I will get this to scale to 5000 connections. I can
achieve the same reduction in apache processes if I simply turn off
keepalive or reduce it to a very low value (say 1 second). Essentially
with keepalive off, apache will behave exactly the same way in
handling clients directly as having nginx in front.

I can certainly see value in using nginx as a load balancer if one had
multiple small apache instances running. But for larger configurations
which tend to use a more sophisticated load balancer, I'm not exactly
sure what the value is ? I do understand that nginx does well serving
static content and one can certainly use that by re-directing all
static requests to an nginx instance. My question specifically is
w.r.t using nginx in front of apache.

Comments ?
Shanti

-- 

There is never time to do it right, but always time to do it over.
--- Murphy's law of computing