Tuning workers and connections

[Stefan Caunter]

On Tue, Aug 21, 2012 at 3:23 AM, liangrubo <nginx-forum at nginx.us> wrote:
> Hello, we meet similar problem.
>
> if we set "multi_accept off;" and "accept_mutex on;", it sometimes took 3 or
> 9 or even 21 seconds to connect to nginx on port 80 from the same server. if
> we set "multi_accept on;" and "accept_mutex off;",  we can connect to nginx
> on port 80 instantly but it may take long to get the response (more than 3
> seconds for small css files on the same server).
>
> Our deploy structure is as follows:
> frontend nginx listen on port 80, servering static files. For dynamic
> requests, nginx on port 80 reverse proxy to nginx on port 81 via tcp socket,
> nginx on port 81 again reverse proxy to uwsgi via unix sokcet.
>
> the site is handling about 1200 requests per second.
>
> the server has 24 cpu cores and 96G memory.  the system has low load, lots
> of free memory and no IO bottleneck.

What is the system load? You have a bottleneck, you just have not
found it. If static file access is problematic, look to the disk i/o
subsystem. Are you caching? How many files is nginx accessing? What is
the filesystem? Show the output of a filesystem utility under 1200r/s.
If nginx is accessing many small static files from one partition, is
it on its own partition, or is logging writing to the same partition?
Concurrent access times to disk reduce geometrically as requests grow
arithmetically. At 1200 per second, nginx processes are likely waiting
for the disk to find request.

> some of the nginx configurations are as follows:
> worker_processes 8; //it was originally 4, I doubled it but it didn't help
> worker_rlimit_nofile 15240;
> worker_connections  15240;
>
> use epoll;
>
> we have keepalive setup for upstream(80 to 81 reverse proxy) as follows:
> keepalive 32;
>
>
> information shown by status_sub:
> Active connections: 4320
> server accepts handled requests
>  9012300 9012300 37349248
> Reading: 118 Writing: 199 Waiting: 4003
>
> we examined the dynamic request handling is fast and even if it was slow, it
> should not affect static file servering on port 80 anyway, right?

Right. Your dynamic processing is not i/o bound from what you have described.

> it seems nginx can't process the request fast enough but we can't find what
> is the bottleneck.
>
> any help is greatly appreciated.
>

Stefan Caunter