Quick performance deterioration when No. of clients increases

Nikolaos Milas nmilas at noa.gr
Fri Oct 11 08:07:22 UTC 2013


Hello,

I am trying to migrate a Joomla 2.5.8 website from Apache to NGINX 1.4.2 
with php-fpm 5.3.3 (and MySQL 5.5.34) on a CentOS 6.4 x86_64 Virtual 
Machine (running under KMS).

The goal is to achieve better peak performance: This site has occasional 
high peaks; while the normal traffic is ~10 req/sec, it may reach > 3000 
req/sec for periods of a few hours (due to the type of services the site 
provides - it is a non-profit, real-time seismicity-related site - so 
php caching should not be more than 10 seconds).

The new VM (using Nginx) currently is in testing mode and it only has 
1-core CPU / 3 GB of RAM. We tested performance with loadimpact and the 
results are attached.

You can see at the load graph that as the load approaches 250 clients, 
the response time increases very much and is already unacceptable (this 
happens consistently). I expected better performance, esp. since caching 
is enabled. Despite many efforts, I cannot find the cause of the 
bottleneck, and how to deal with it. We would like to achieve better 
scaling, esp. since NGINX is famous for its scaling capabilities. Having 
very little experience with Nginx, I would like to ask for your 
assistance for a better configuration.

When this performance deterioration occurs, we don't see very high CPU 
load (Unix load peaks 2.5), neither RAM exhaustion (System RAM usage 
appears to be below 30%). [Monitoring is through Nagios.]

Can you please guide me on how to correct this issue? Any and all 
suggestions will be appreciated.

Current configuration, based on info available on the Internet, is as 
follows (replaced true domain/website name and public IP address(es)):

=================== Nginx.conf ===================

user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

worker_rlimit_nofile 200000;

events {
     worker_connections 8192;
     multi_accept on;
     use epoll;
}

http {
     include       /etc/nginx/mime.types;
     default_type  application/octet-stream;
     server_names_hash_bucket_size 64;

     log_format  main  '$remote_addr - $remote_user [$time_local] 
"$request" '
                       '$status $body_bytes_sent "$http_referer" '
                       '"$http_user_agent" "$http_x_forwarded_for"';

     log_format cache  '$remote_addr - $remote_user [$time_local] 
"$request" '
                       '$status $upstream_cache_status $body_bytes_sent 
"$http_referer" '
                       '"$http_user_agent" "$http_x_forwarded_for"';

     fastcgi_cache_path /var/cache/nginx levels=1:2 
keys_zone=microcache:5m max_size=1000m;

     access_log  /var/log/nginx/access.log  main;

     sendfile           on;

     tcp_nopush         on;
     tcp_nodelay        on;
     keepalive_timeout  2;

     types_hash_max_size 2048;
     server_tokens off;

     keepalive_requests 30;

     open_file_cache max=5000 inactive=20s;
     open_file_cache_valid 30s;
     open_file_cache_min_uses 2;
     open_file_cache_errors on;

     gzip on;
     gzip_static on;
     gzip_disable "msie6";
     gzip_http_version 1.1;
     gzip_vary on;
     gzip_comp_level 6;
     gzip_proxied any;
     gzip_types text/plain text/css application/json 
application/x-javascript text/xml application/xml application/xml+rss 
text/javascript application/javascript text/x-js;
     gzip_buffers 16 8k;

     include /etc/nginx/conf.d/*.conf;
}

==================================================

================ website config ==================

server {
     listen       80;
     server_name  www.example.com;
     access_log  /var/webs/wwwexample/log/access_log main;
     error_log /var/webs/wwwexample/log/error_log warn;
     root   /var/webs/wwwexample/www/;

     index  index.php index.html index.htm index.cgi default.html 
default.htm default.php;
     location / {
         try_files $uri $uri/ /index.php?$args;
     }

     location /nginx_status {
        stub_status on;
        access_log   off;
        allow 10.10.10.0/24;
        deny all;
     }

     location ~* 
/(images|cache|media|logs|tmp)/.*\.(php|pl|py|jsp|asp|sh|cgi)$ {
         return 403;
         error_page 403 /403_error.html;
     }

     location ~ /\.ht {
         deny  all;
     }

     location /administrator {
         allow 10.10.10.0/24;
         deny all;
     }

     location ~ \.php$ {

         # Setup var defaults
         set $no_cache "";
         # If non GET/HEAD, don't cache & mark user as uncacheable for 1 
second via cookie
if ($request_method !~ ^(GET|HEAD)$) {
             set $no_cache "1";
}
         # Drop no cache cookie if need be
         # (for some reason, add_header fails if included in prior if-block)
         if ($no_cache = "1") {
             add_header Set-Cookie "_mcnc=1; Max-Age=2; Path=/";
             add_header X-Microcachable "0";
         }
         # Bypass cache if no-cache cookie is set
         if ($http_cookie ~* "_mcnc") {
             set $no_cache "1";
         }
         # Bypass cache if flag is set
         fastcgi_no_cache $no_cache;
         fastcgi_cache_bypass $no_cache;
         fastcgi_cache microcache;
         fastcgi_cache_key $scheme$host$request_uri$request_method;
         fastcgi_cache_valid 200 301 302 10s;
         fastcgi_cache_use_stale updating error timeout invalid_header 
http_500;
         fastcgi_pass_header Set-Cookie;
         fastcgi_pass_header Cookie;
         fastcgi_ignore_headers Cache-Control Expires Set-Cookie;

         try_files $uri =404;
         include /etc/nginx/fastcgi_params;
         fastcgi_param PATH_INFO $fastcgi_script_name;
         fastcgi_intercept_errors on;

         fastcgi_buffer_size 128k;
         fastcgi_buffers 256 16k;
         fastcgi_busy_buffers_size 256k;
         fastcgi_temp_file_write_size 256k;
         fastcgi_read_timeout 240;

         fastcgi_pass unix:/tmp/php-fpm.sock;

         fastcgi_index index.php;
         include /etc/nginx/fastcgi_params;
         fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;

     }

     location ~* \.(ico|pdf|flv)$ {
         expires 1d;
     }

     location ~* \.(js|css|png|jpg|jpeg|gif|swf|xml|txt)$ {
         expires 1d;
     }

}
==================================================

================= php-fpm.conf ===================
include=/etc/php-fpm.d/*.conf
[global]
pid = /var/run/php-fpm/php-fpm.pid
error_log = /var/log/php-fpm/error.log

daemonize = no
==================================================

============== php-fpm.d/www.conf ================

[www]
listen = /tmp/php-fpm.sock
listen.allowed_clients = 127.0.0.1
user = nginx
group = nginx

pm = dynamic
pm.max_children = 1024
pm.start_servers = 5
pm.min_spare_servers = 5
pm.max_spare_servers = 35

slowlog = /var/log/php-fpm/www-slow.log

php_flag[display_errors] = off
php_admin_value[error_log] = /var/log/php-fpm/www-error.log
php_admin_flag[log_errors] = on
php_admin_value[memory_limit] = 128M

php_value[session.save_handler] = files
php_value[session.save_path] = /var/lib/php/session

==================================================

================ mysql my.cnf ====================

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
symbolic-links=0
user=mysql

query_cache_limit = 2M
query_cache_size = 200M
query_cache_type=1
thread_cache_size=128
key_buffer = 100M
join_buffer = 2M
table_cache= 150M
sort_buffer= 2M
read_rnd_buffer_size=10M
tmp_table_size=200M
max_heap_table_size=200M

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

==================================================

=============== mysqltuner report ================

  >>  MySQLTuner 1.2.0 - Major Hayden <major at mhtx.net>

-------- General Statistics 
--------------------------------------------------
[--] Skipped version check for MySQLTuner script
[OK] Currently running supported MySQL version 5.5.34
[OK] Operating on 64-bit architecture

-------- Storage Engine Statistics 
-------------------------------------------
[--] Status: +Archive -BDB -Federated +InnoDB -ISAM -NDBCluster
[--] Data in MyISAM tables: 9M (Tables: 80)
[--] Data in InnoDB tables: 1M (Tables: 65)
[--] Data in PERFORMANCE_SCHEMA tables: 0B (Tables: 17)
[--] Data in MEMORY tables: 0B (Tables: 4)
[!!] Total fragmented tables: 66

-------- Security Recommendations 
-------------------------------------------
[OK] All database users have passwords assigned

-------- Performance Metrics 
-------------------------------------------------
[--] Up for: 12h 51m 16s (21K q [0.471 qps], 1K conn, TX: 10M, RX: 1M)
[--] Reads / Writes: 55% / 45%
[--] Total buffers: 694.0M global + 21.4M per thread (151 max threads)
[!!] Maximum possible memory usage: 3.8G (135% of installed RAM)
[OK] Slow queries: 0% (0/21K)
[OK] Highest usage of available connections: 23% (36/151)
[OK] Key buffer size / total MyISAM indexes: 150.0M/5.1M
[OK] Key buffer hit rate: 99.3% (51K cached / 358 reads)
[OK] Query cache efficiency: 80.9% (10K cached / 13K selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 55 sorts)
[OK] Temporary tables created on disk: 8% (5 on disk / 60 total)
[OK] Thread cache hit rate: 98% (36 created / 1K connections)
[OK] Table cache hit rate: 20% (192 open / 937 opened)
[OK] Open file limit used: 0% (210/200K)
[OK] Table locks acquired immediately: 99% (4K immediate / 4K locks)
[!!] Connections aborted: 8%
[OK] InnoDB data size / buffer pool: 1.1M/128.0M

==================================================

Please advise.

Thanks and Regards,
Nick

-------------- next part --------------
A non-text attachment was scrubbed...
Name: load_impact_1.png
Type: image/png
Size: 56882 bytes
Desc: not available
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20131011/c8d6445e/attachment-0001.png>


More information about the nginx mailing list