Handling 500k concurrent connections on Linux

[John Watson]

1) Error logs are clean (except for some 404s)

2) nginx.conf and sysctl.conf: https://gist.github.com/0b3b52050254e273ff11

Set TX/RX descriptors to 4096/4096 (maximum):
    ethtool -G eth1 tx 4096 rx 4096

Disabled irqbalanced and pinned IRQs to CPU0-7 for NIC

Don't know exact amount, but a good majority of the connections are
sitting idle for 90s before being closed.

Some graphs on the network interface for past couple days:
https://www.dropbox.com/s/0bl304ulhqp6a4n/push_stream_network.png

Thank you,

John W

On Wed, Oct 10, 2012 at 01:05:05PM +0400, Andrew Alexeev <andrew at nginx.com> wrote:
> John,
> 
> On Oct 9, 2012, at 11:10 PM, John Watson wrote:
> 
> > I was wondering if anyone had some tips/guidelines for scaling Nginx on
> > Linux to >500k concurrent connections. Playing with the
> > nginx_http_push_stream module in streaming mode. Noticing periodic slow
> > accept and/or response headers. I've scoured the Internet
> > looking/learning ways to tune Nginx/Linux but I think I've exhausted my
> > abilities.
> > 
> > Any help would be appreciated.
> > 
> > Hardware
> > Dual Nehalem 5520
> > 24G RAM
> > Intel 82576 (igb)
> > Ubuntu 12.04.1 (3.2.0-31-generic x86_64)
> > 
> > Thank You,
> > 
> > John W
> 
> I'd assume you've already checked/fixed the following, right?
> 
> 1) Error logs - anything wrong seen in there?
> 
> 2) http://nginx.org/en/docs/ngx_core_module.html#multi_accept and http://nginx.org/en/docs/ngx_core_module.html#accept_mutex - did you try it on/off?
> 
> 3) file descriptors limits (cat /proc/sys/fs/file-max, sudo - nginx && ulimit, worker_rlimit_nofile)
> 
> 4) sysctl net.ipv4.ip_local_port_range (if you're aiming at proxying all those connections to upstreams)
> 
> Additional information about what's happening in all those 500k connections might be helpful, as well as the relevant configuration section :)
> 
> Hope this helps
> 
> 
> -- 
> AA @ nginx
> http://nginx.com/support.html
> 
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 834 bytes
Desc: not available
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20121010/b2d53833/attachment-0001.bin>