Nginx setting up >25.000 concurrent connections per second

[ktm at rice.edu]

On Thu, Oct 06, 2011 at 02:30:41PM -0400, atadmin wrote:
> Hi,
> 
> I a preparing a new web environment with high requirements: 100.000
> concurrents connections per second (sometimes). Every server will
> execute a php script through php5-fpm.
> I am testing where are the limits of nginx (without any php) and how to
> setup the machine for optimize it. I will explain my tests and results:
> 
> Test:
> 
> 10 servers 4 CPUs, 4 Gb ram, 16Gb HD.
> Local Network: 1Gb (Datacenter network)
> 
> 1 Server has a debian squeeze with basic installation (from netinstall
> iso) and nginx from debian repositories (0.7.67-3)
> 
> I changed only 2 options for nginx config (i tested with others):
> 
> worker_processes  4;
> worker_connections  10240;
> 
> I add this lines to /etc/security/limits.conf (restart nginx)
> 
> www-data        soft    nproc           100000
> www-data        soft    nofile          100000
> 
> and for discard I/O issues i mounted /var/log/nginx in ram:
> 
> mount -t tmpfs -o nodev,nosuid,noexec,nodiratime,size=2500M none
> /var/log/nginx/
> 
> Created static file:
> echo "HOLA">/var/www/a.txt
> 
> From the rest of 9 servers with the same basic installation i installed
> apache2-utils and changed: ulimit -n 100000. After just try this
> command:
> 
> ab -n 500000 -c 200 http://192.168.1.11/a.txt
> 
> 
> Really i tested with few server and more with a lot of diferents values
> for ab tool, but i can not get better results:
> 
> # awk '{ print $4 }' /var/log/nginx/localhost.access.log |awk -F: '{
> print $2 ":" $3 ":" $4 }'|sort|uniq -c
> [...]
>   22345 19:57:58
>   21088 19:57:59
>   19010 19:58:00
>   20211 19:58:01
>   22469 19:58:02
>   23121 19:58:03
>   22682 19:58:04
>   23105 19:58:05
>   24537 19:58:06
>   22313 19:58:07
>   22406 19:58:08
>   22804 19:58:09
>   23823 19:58:10
>   22280 19:58:11
>   24634 19:58:12
>   22722 19:58:13
>   22429 19:58:14
>   24271 19:58:15
>   20265 19:58:16
>   20678 19:58:17
>   23136 19:58:18
>   22203 19:58:19
>   22521 19:58:20
>   24254 19:58:21
>   23216 19:58:22
>   22587 19:58:23
>   18365 19:58:24
>   22221 19:58:25
>   22123 19:58:26
>   24464 19:58:27
> [...]
> 
> Also i tried changing a lot of things in /etc/sysctl.conf (sysctl -p and
> restart nginx) but i didn't see better results.
> 
> For example:
> 
> net.ipv4.tcp_keepalive_time = 300
> # Avoid a smurf attack
> net.ipv4.icmp_echo_ignore_broadcasts = 1
>  
> # Turn on protection for bad icmp error messages
> net.ipv4.icmp_ignore_bogus_error_responses = 1
>  
> # Turn on syncookies for SYN flood attack protection
> net.ipv4.tcp_syncookies = 0
>  
> # Turn on and log spoofed, source routed, and redirect packets
> net.ipv4.conf.all.log_martians = 1
> net.ipv4.conf.default.log_martians = 1
>  
> # No source routed packets here
> net.ipv4.conf.all.accept_source_route = 0
> net.ipv4.conf.default.accept_source_route = 0
>  
> # Turn on reverse path filtering
> net.ipv4.conf.all.rp_filter = 1
> net.ipv4.conf.default.rp_filter = 1
>  
> # Make sure no one can alter the routing tables
> net.ipv4.conf.all.accept_redirects = 0
> net.ipv4.conf.default.accept_redirects = 0
> net.ipv4.conf.all.secure_redirects = 0
> net.ipv4.conf.default.secure_redirects = 0
>  
> # Don't act as a router
> net.ipv4.ip_forward = 1
> net.ipv4.conf.all.send_redirects = 0
> net.ipv4.conf.default.send_redirects = 0
>  
> # Turn on execshild
> kernel.exec-shield = 1
> kernel.randomize_va_space = 1
>  
> # Tuen IPv6
> net.ipv6.conf.default.router_solicitations = 0
> net.ipv6.conf.default.accept_ra_rtr_pref = 0
> net.ipv6.conf.default.accept_ra_pinfo = 0
> net.ipv6.conf.default.accept_ra_defrtr = 0
> net.ipv6.conf.default.autoconf = 0
> net.ipv6.conf.default.dad_transmits = 0
> net.ipv6.conf.default.max_addresses = 1
>  
> # Optimization for port usefor LBs
> # Increase system file descriptor limit
> fs.file-max = 655350
>  
> # Allow for more PIDs (to reduce rollover problems); may break some
> programs 32768
> kernel.pid_max = 65536
>  
> # Increase system IP port limits
> net.ipv4.ip_local_port_range = 1500 65000
>  
> # Increase TCP max buffer size setable using setsockopt()
> net.ipv4.tcp_rmem = 4096 87380 33554432 
> net.ipv4.tcp_wmem = 4096 65536 33554432
>  
> # Increase Linux auto tuning TCP buffer limits
> # min, default, and max number of bytes to use
> # set max to at least 4MB, or higher if you use very high BDP paths
> # Tcp Windows etc
> net.core.rmem_max = 33554432
> net.core.wmem_max = 33554432
> net.core.rmem_default=65536 
> net.core.wmem_default=65536 
> net.core.netdev_max_backlog = 5000
> net.ipv4.tcp_window_scaling = 1
> net.ipv4.tcp_timestamps = 1
> net.ipv4.tcp_sack = 1
> net.ipv4.tcp_no_metrics_save = 1
> 
> With last kernels and autoptimize is not necessary change anything about
> tcp buffers (but i think for this requirements yes).
> 
> I was monitoring the machine while tests, CPU usage by nginx is around
> 30%, RAM nothing important, and few I/O traffic, Load <0.50.
> 
> Could somebody help me for find where is the bottleneck?
> 
> Thanks.
> 

Could you be bottle-necked in your testing tool "ab"?

Ken