Handling 500k concurrent connections on Linux
john at disqus.com
Wed Oct 10 20:11:42 UTC 2012
1) Error logs are clean (except for some 404s)
2) nginx.conf and sysctl.conf: https://gist.github.com/0b3b52050254e273ff11
Set TX/RX descriptors to 4096/4096 (maximum):
ethtool -G eth1 tx 4096 rx 4096
Disabled irqbalanced and pinned IRQs to CPU0-7 for NIC
Don't know exact amount, but a good majority of the connections are
sitting idle for 90s before being closed.
Some graphs on the network interface for past couple days:
On Wed, Oct 10, 2012 at 01:05:05PM +0400, Andrew Alexeev <andrew at nginx.com> wrote:
> On Oct 9, 2012, at 11:10 PM, John Watson wrote:
> > I was wondering if anyone had some tips/guidelines for scaling Nginx on
> > Linux to >500k concurrent connections. Playing with the
> > nginx_http_push_stream module in streaming mode. Noticing periodic slow
> > accept and/or response headers. I've scoured the Internet
> > looking/learning ways to tune Nginx/Linux but I think I've exhausted my
> > abilities.
> > Any help would be appreciated.
> > Hardware
> > Dual Nehalem 5520
> > 24G RAM
> > Intel 82576 (igb)
> > Ubuntu 12.04.1 (3.2.0-31-generic x86_64)
> > Thank You,
> > John W
> I'd assume you've already checked/fixed the following, right?
> 1) Error logs - anything wrong seen in there?
> 2) http://nginx.org/en/docs/ngx_core_module.html#multi_accept and http://nginx.org/en/docs/ngx_core_module.html#accept_mutex - did you try it on/off?
> 3) file descriptors limits (cat /proc/sys/fs/file-max, sudo - nginx && ulimit, worker_rlimit_nofile)
> 4) sysctl net.ipv4.ip_local_port_range (if you're aiming at proxying all those connections to upstreams)
> Additional information about what's happening in all those 500k connections might be helpful, as well as the relevant configuration section :)
> Hope this helps
> AA @ nginx
> nginx mailing list
> nginx at nginx.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Size: 834 bytes
Desc: not available
More information about the nginx