Tuning workers and connections

[Avleen Vig]

Hi folks, I have some questions about tuning nginx for best
performance, on a site which handle ~1000 requests per second.

In our config, we have:

    worker_processes  16;
    worker_rlimit_nofile 32768;
    events {
        worker_connections  8192;
    }

This is on a server with 8 CPU cores and 8Gb RAM.
My understanding is that this should allow nginx to establish up to
32k network connections, because that is the limit of
worker_rlimit_nofile. We serve very few files off the disk, maybe a
dozen CSS and JS files. Everything else is handed to backends using
proxy_pass.
So with one fd for the browser, and one fd to the upstream, we should
be able to handle 16k concurrent connections.

However what we're seeing, is that around 5k connections we get a
performance hit. Connections slow down, take longer to establish, etc.
The load on the box is almost zero, nginx is clearly working very fast
and efficiently, but I'm not sure why it slows down.
Any thoughts?


Also, I'm thinking about enabling the multi_accept option, but I
couldn't find much documentation on how this works.
It sounds like a high-performance tweak which we should be using if we
get this many requests per second.
Could it be that not using multi_accept is the problem?
Nginx otherwise has to handle the incoming connection requests in
serial and this is a bottleneck?


Thanks :)