Nginx setting up >25.000 concurrent connections per second

[atadmin]

Hi,

I a preparing a new web environment with high requirements: 100.000
concurrents connections per second (sometimes). Every server will
execute a php script through php5-fpm.
I am testing where are the limits of nginx (without any php) and how to
setup the machine for optimize it. I will explain my tests and results:

Test:

10 servers 4 CPUs, 4 Gb ram, 16Gb HD.
Local Network: 1Gb (Datacenter network)

1 Server has a debian squeeze with basic installation (from netinstall
iso) and nginx from debian repositories (0.7.67-3)

I changed only 2 options for nginx config (i tested with others):

worker_processes  4;
worker_connections  10240;

I add this lines to /etc/security/limits.conf (restart nginx)

www-data        soft    nproc           100000
www-data        soft    nofile          100000

and for discard I/O issues i mounted /var/log/nginx in ram:

mount -t tmpfs -o nodev,nosuid,noexec,nodiratime,size=2500M none
/var/log/nginx/

Created static file:
echo "HOLA">/var/www/a.txt

>From the rest of 9 servers with the same basic installation i installed
apache2-utils and changed: ulimit -n 100000. After just try this
command:

ab -n 500000 -c 200 http://192.168.1.11/a.txt


Really i tested with few server and more with a lot of diferents values
for ab tool, but i can not get better results:

# awk '{ print $4 }' /var/log/nginx/localhost.access.log |awk -F: '{
print $2 ":" $3 ":" $4 }'|sort|uniq -c
[...]
  22345 19:57:58
  21088 19:57:59
  19010 19:58:00
  20211 19:58:01
  22469 19:58:02
  23121 19:58:03
  22682 19:58:04
  23105 19:58:05
  24537 19:58:06
  22313 19:58:07
  22406 19:58:08
  22804 19:58:09
  23823 19:58:10
  22280 19:58:11
  24634 19:58:12
  22722 19:58:13
  22429 19:58:14
  24271 19:58:15
  20265 19:58:16
  20678 19:58:17
  23136 19:58:18
  22203 19:58:19
  22521 19:58:20
  24254 19:58:21
  23216 19:58:22
  22587 19:58:23
  18365 19:58:24
  22221 19:58:25
  22123 19:58:26
  24464 19:58:27
[...]

Also i tried changing a lot of things in /etc/sysctl.conf (sysctl -p and
restart nginx) but i didn't see better results.

For example:

net.ipv4.tcp_keepalive_time = 300
# Avoid a smurf attack
net.ipv4.icmp_echo_ignore_broadcasts = 1
 
# Turn on protection for bad icmp error messages
net.ipv4.icmp_ignore_bogus_error_responses = 1
 
# Turn on syncookies for SYN flood attack protection
net.ipv4.tcp_syncookies = 0
 
# Turn on and log spoofed, source routed, and redirect packets
net.ipv4.conf.all.log_martians = 1
net.ipv4.conf.default.log_martians = 1
 
# No source routed packets here
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
 
# Turn on reverse path filtering
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
 
# Make sure no one can alter the routing tables
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.default.secure_redirects = 0
 
# Don't act as a router
net.ipv4.ip_forward = 1
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
 
# Turn on execshild
kernel.exec-shield = 1
kernel.randomize_va_space = 1
 
# Tuen IPv6
net.ipv6.conf.default.router_solicitations = 0
net.ipv6.conf.default.accept_ra_rtr_pref = 0
net.ipv6.conf.default.accept_ra_pinfo = 0
net.ipv6.conf.default.accept_ra_defrtr = 0
net.ipv6.conf.default.autoconf = 0
net.ipv6.conf.default.dad_transmits = 0
net.ipv6.conf.default.max_addresses = 1
 
# Optimization for port usefor LBs
# Increase system file descriptor limit
fs.file-max = 655350
 
# Allow for more PIDs (to reduce rollover problems); may break some
programs 32768
kernel.pid_max = 65536
 
# Increase system IP port limits
net.ipv4.ip_local_port_range = 1500 65000
 
# Increase TCP max buffer size setable using setsockopt()
net.ipv4.tcp_rmem = 4096 87380 33554432 
net.ipv4.tcp_wmem = 4096 65536 33554432
 
# Increase Linux auto tuning TCP buffer limits
# min, default, and max number of bytes to use
# set max to at least 4MB, or higher if you use very high BDP paths
# Tcp Windows etc
net.core.rmem_max = 33554432
net.core.wmem_max = 33554432
net.core.rmem_default=65536 
net.core.wmem_default=65536 
net.core.netdev_max_backlog = 5000
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_no_metrics_save = 1

With last kernels and autoptimize is not necessary change anything about
tcp buffers (but i think for this requirements yes).

I was monitoring the machine while tests, CPU usage by nginx is around
30%, RAM nothing important, and few I/O traffic, Load <0.50.

Could somebody help me for find where is the bottleneck?

Thanks.

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,216332,216332#msg-216332