Tuning workers and connections

[liangrubo]

Hello, we meet similar problem.

if we set "multi_accept off;" and "accept_mutex on;", it sometimes took 3 or
9 or even 21 seconds to connect to nginx on port 80 from the same server. if
we set "multi_accept on;" and "accept_mutex off;",  we can connect to nginx
on port 80 instantly but it may take long to get the response (more than 3
seconds for small css files on the same server).

Our deploy structure is as follows:
frontend nginx listen on port 80, servering static files. For dynamic
requests, nginx on port 80 reverse proxy to nginx on port 81 via tcp socket,
nginx on port 81 again reverse proxy to uwsgi via unix sokcet.

the site is handling about 1200 requests per second.

the server has 24 cpu cores and 96G memory.  the system has low load, lots
of free memory and no IO bottleneck.

some of the nginx configurations are as follows:
worker_processes 8; //it was originally 4, I doubled it but it didn't help
worker_rlimit_nofile 15240;
worker_connections  15240;

use epoll;

we have keepalive setup for upstream(80 to 81 reverse proxy) as follows:
keepalive 32;


information shown by status_sub:
Active connections: 4320 
server accepts handled requests
 9012300 9012300 37349248 
Reading: 118 Writing: 199 Waiting: 4003 

we examined the dynamic request handling is fast and even if it was slow, it
should not affect static file servering on port 80 anyway, right?

it seems nginx can't process the request fast enough but we can't find what
is the bottleneck.

any help is greatly appreciated.

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,3638,229939#msg-229939