Tuning workers and connections

[Avleen Vig]

> Directive worker_rlimit_nofile just raises open files limit for
> nginx processes if system allows this, nothing more.  It's usually
> needed only if you started nginx with low limit, then raised the
> system one and want nginx to raise it's limit without restarting.

Ah thanks :)

>> We serve very few files off the disk, maybe a
>> dozen CSS and JS files. Everything else is handed to backends using
>> proxy_pass.
>> So with one fd for the browser, and one fd to the upstream, we should
>> be able to handle 16k concurrent connections.
>>
>> However what we're seeing, is that around 5k connections we get a
>> performance hit. Connections slow down, take longer to establish, etc.
>> The load on the box is almost zero, nginx is clearly working very fast
>> and efficiently, but I'm not sure why it slows down.
>> Any thoughts?
>
> There are several things to check:
>
> 1. Which event method used?  Make sure you use kqueue for *BSD,
> epoll for Linux.

epoll.

> 2. What's in error_log?  Your system logs/memory stats/etc?  You
> may run out of some resources (file descriptors, network buffers,
> connection states in your firewall, ...).

I saw nothing about resource starvation in the error log.
Also I should mention this was on 0.6.35.
We've also upgraded to 0.7.59.

> 3. In which states nginx workers are?  If they are disk bound it's
> probably a good idea to check why and e.g. try to tune proxy
> buffers / output buffers / sendfile_max_chunk.

They don't appear to be disk bound. I will check again tonight at our
peak hours.

>> Also, I'm thinking about enabling the multi_accept option, but I
>> couldn't find much documentation on how this works.
>> It sounds like a high-performance tweak which we should be using if we
>> get this many requests per second.
>> Could it be that not using multi_accept is the problem?
>> Nginx otherwise has to handle the incoming connection requests in
>> serial and this is a bottleneck?
>
> With multi_accept nginx tries to accept all connections from
> listen queue at once.  Without it - only one connection will be
> accepted on each return from event function.  The bad thing with
> multi_accept that if you have constant stream of incomming
> connections at high rate - it may overflow your worker_connections
> without any chance to process already accepted connections.

Ok :-)
I've massively increased the number of worker_connections now, and the
box is otherwise quite powerful. It should be able to cope, but
there's only one way to find out!

Thanks guys!