Handling 500k concurrent connections on Linux

[John Watson]

After a bit more digging I discovered that Nginx sets the backlog on the
listen socket to only 511 (at least on Linux), not the -1 in the docs.
By increasing that to a much larger number I haven't noticed slow
accepts/response headers. Also for reference, backlog on a listen
socket is silently limited to net.core.somaxconn (which defaults to
128) so make sure to increase that and other necessary tunings as well.

On Wed, Oct 10, 2012 at 01:11:42PM -0700, John Watson <john at disqus.com> wrote:
> 1) Error logs are clean (except for some 404s)
> 
> 2) nginx.conf and sysctl.conf: https://gist.github.com/0b3b52050254e273ff11
> 
> Set TX/RX descriptors to 4096/4096 (maximum):
>     ethtool -G eth1 tx 4096 rx 4096
> 
> Disabled irqbalanced and pinned IRQs to CPU0-7 for NIC
> 
> Don't know exact amount, but a good majority of the connections are
> sitting idle for 90s before being closed.
> 
> Some graphs on the network interface for past couple days:
> https://www.dropbox.com/s/0bl304ulhqp6a4n/push_stream_network.png
> 
> Thank you,
> 
> John W
> 
> On Wed, Oct 10, 2012 at 01:05:05PM +0400, Andrew Alexeev <andrew at nginx.com> wrote:
> > John,
> > 
> > On Oct 9, 2012, at 11:10 PM, John Watson wrote:
> > 
> > > I was wondering if anyone had some tips/guidelines for scaling Nginx on
> > > Linux to >500k concurrent connections. Playing with the
> > > nginx_http_push_stream module in streaming mode. Noticing periodic slow
> > > accept and/or response headers. I've scoured the Internet
> > > looking/learning ways to tune Nginx/Linux but I think I've exhausted my
> > > abilities.
> > > 
> > > Any help would be appreciated.
> > > 
> > > Hardware
> > > Dual Nehalem 5520
> > > 24G RAM
> > > Intel 82576 (igb)
> > > Ubuntu 12.04.1 (3.2.0-31-generic x86_64)
> > > 
> > > Thank You,
> > > 
> > > John W
> > 
> > I'd assume you've already checked/fixed the following, right?
> > 
> > 1) Error logs - anything wrong seen in there?
> > 
> > 2) http://nginx.org/en/docs/ngx_core_module.html#multi_accept and http://nginx.org/en/docs/ngx_core_module.html#accept_mutex - did you try it on/off?
> > 
> > 3) file descriptors limits (cat /proc/sys/fs/file-max, sudo - nginx && ulimit, worker_rlimit_nofile)
> > 
> > 4) sysctl net.ipv4.ip_local_port_range (if you're aiming at proxying all those connections to upstreams)
> > 
> > Additional information about what's happening in all those 500k connections might be helpful, as well as the relevant configuration section :)
> > 
> > Hope this helps
> > 
> > 
> > -- 
> > AA @ nginx
> > http://nginx.com/support.html
> > 
> > _______________________________________________
> > nginx mailing list
> > nginx at nginx.org
> > http://mailman.nginx.org/mailman/listinfo/nginx


-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 834 bytes
Desc: not available
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20121016/ce7fc25b/attachment-0001.bin>