nginx failing when too many connections

[Juraj Ziegler]

Hi.

We have been using nginx for quite some time now, for serving static  
images, CSS files (and 301 / 302 redirects :). It worked well, until  
the number of concurrent connections reached the limit of  
"worker_processes" * "worker_connections" (4 * 4096). Then it started  
to behave very strangely - the number of connections dropped to 0 and  
no requests were being served. I understand why the 16385th  
connection could not be served, but the previous 16384 should have  
been served, I think.

After locating the problem, I increased "worker_connections" to 8192  
(giving a maximum of 32768 parallel connections). At the same time, I  
increased "worker_rlimit_nofile" to 32768. It looked like it could help.

That worked fine, until today. Requests timed out, but randomly. Req/ 
s rate jumped up and down between between 500 and 1100. 1100 is  
normal for the given time of day.

Restarting nginx helped only for a minute, then all was back to the  
bad state.
Going back from 0.5.33 to 0.5.32 did not help.
Rebooting the whole server did not help.
Switching to just one worked did not help

There was nothing special in the error log, just expected errors  
about missing files. "truss" showed all workers doing something.  
"snoop" showed traffic flowing. CPU was (and is) almost idle. Plenty  
of free RAM. iostat showed normal numbers. For completeness - I'm  
running nginx on Solaris/x86 10u2, reasonably patched. Compiled with  
Sun Studio 12.

Finally, I set "worker_connections" to 6000 and  
"worker_rlimit_nofile" to 16384. Since then, it works OK. Currently  
having 12316 concurrent connections, maximum being 15330 for the last  
hour. Serving 1224 req/s (max. 1537).

Is there some internal limit to "worker_processes" *  
"worker_connections" I have hit?
What else can I check, when nginx starts misbehaving?


j.