Quick performance deterioration when No. of clients increases
Nikolaos Milas
nmilas at noa.gr
Fri Oct 11 08:07:22 UTC 2013
Hello,
I am trying to migrate a Joomla 2.5.8 website from Apache to NGINX 1.4.2
with php-fpm 5.3.3 (and MySQL 5.5.34) on a CentOS 6.4 x86_64 Virtual
Machine (running under KMS).
The goal is to achieve better peak performance: This site has occasional
high peaks; while the normal traffic is ~10 req/sec, it may reach > 3000
req/sec for periods of a few hours (due to the type of services the site
provides - it is a non-profit, real-time seismicity-related site - so
php caching should not be more than 10 seconds).
The new VM (using Nginx) currently is in testing mode and it only has
1-core CPU / 3 GB of RAM. We tested performance with loadimpact and the
results are attached.
You can see at the load graph that as the load approaches 250 clients,
the response time increases very much and is already unacceptable (this
happens consistently). I expected better performance, esp. since caching
is enabled. Despite many efforts, I cannot find the cause of the
bottleneck, and how to deal with it. We would like to achieve better
scaling, esp. since NGINX is famous for its scaling capabilities. Having
very little experience with Nginx, I would like to ask for your
assistance for a better configuration.
When this performance deterioration occurs, we don't see very high CPU
load (Unix load peaks 2.5), neither RAM exhaustion (System RAM usage
appears to be below 30%). [Monitoring is through Nagios.]
Can you please guide me on how to correct this issue? Any and all
suggestions will be appreciated.
Current configuration, based on info available on the Internet, is as
follows (replaced true domain/website name and public IP address(es)):
=================== Nginx.conf ===================
user nginx;
worker_processes 1;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
worker_rlimit_nofile 200000;
events {
worker_connections 8192;
multi_accept on;
use epoll;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
server_names_hash_bucket_size 64;
log_format main '$remote_addr - $remote_user [$time_local]
"$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
log_format cache '$remote_addr - $remote_user [$time_local]
"$request" '
'$status $upstream_cache_status $body_bytes_sent
"$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
fastcgi_cache_path /var/cache/nginx levels=1:2
keys_zone=microcache:5m max_size=1000m;
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 2;
types_hash_max_size 2048;
server_tokens off;
keepalive_requests 30;
open_file_cache max=5000 inactive=20s;
open_file_cache_valid 30s;
open_file_cache_min_uses 2;
open_file_cache_errors on;
gzip on;
gzip_static on;
gzip_disable "msie6";
gzip_http_version 1.1;
gzip_vary on;
gzip_comp_level 6;
gzip_proxied any;
gzip_types text/plain text/css application/json
application/x-javascript text/xml application/xml application/xml+rss
text/javascript application/javascript text/x-js;
gzip_buffers 16 8k;
include /etc/nginx/conf.d/*.conf;
}
==================================================
================ website config ==================
server {
listen 80;
server_name www.example.com;
access_log /var/webs/wwwexample/log/access_log main;
error_log /var/webs/wwwexample/log/error_log warn;
root /var/webs/wwwexample/www/;
index index.php index.html index.htm index.cgi default.html
default.htm default.php;
location / {
try_files $uri $uri/ /index.php?$args;
}
location /nginx_status {
stub_status on;
access_log off;
allow 10.10.10.0/24;
deny all;
}
location ~*
/(images|cache|media|logs|tmp)/.*\.(php|pl|py|jsp|asp|sh|cgi)$ {
return 403;
error_page 403 /403_error.html;
}
location ~ /\.ht {
deny all;
}
location /administrator {
allow 10.10.10.0/24;
deny all;
}
location ~ \.php$ {
# Setup var defaults
set $no_cache "";
# If non GET/HEAD, don't cache & mark user as uncacheable for 1
second via cookie
if ($request_method !~ ^(GET|HEAD)$) {
set $no_cache "1";
}
# Drop no cache cookie if need be
# (for some reason, add_header fails if included in prior if-block)
if ($no_cache = "1") {
add_header Set-Cookie "_mcnc=1; Max-Age=2; Path=/";
add_header X-Microcachable "0";
}
# Bypass cache if no-cache cookie is set
if ($http_cookie ~* "_mcnc") {
set $no_cache "1";
}
# Bypass cache if flag is set
fastcgi_no_cache $no_cache;
fastcgi_cache_bypass $no_cache;
fastcgi_cache microcache;
fastcgi_cache_key $scheme$host$request_uri$request_method;
fastcgi_cache_valid 200 301 302 10s;
fastcgi_cache_use_stale updating error timeout invalid_header
http_500;
fastcgi_pass_header Set-Cookie;
fastcgi_pass_header Cookie;
fastcgi_ignore_headers Cache-Control Expires Set-Cookie;
try_files $uri =404;
include /etc/nginx/fastcgi_params;
fastcgi_param PATH_INFO $fastcgi_script_name;
fastcgi_intercept_errors on;
fastcgi_buffer_size 128k;
fastcgi_buffers 256 16k;
fastcgi_busy_buffers_size 256k;
fastcgi_temp_file_write_size 256k;
fastcgi_read_timeout 240;
fastcgi_pass unix:/tmp/php-fpm.sock;
fastcgi_index index.php;
include /etc/nginx/fastcgi_params;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
}
location ~* \.(ico|pdf|flv)$ {
expires 1d;
}
location ~* \.(js|css|png|jpg|jpeg|gif|swf|xml|txt)$ {
expires 1d;
}
}
==================================================
================= php-fpm.conf ===================
include=/etc/php-fpm.d/*.conf
[global]
pid = /var/run/php-fpm/php-fpm.pid
error_log = /var/log/php-fpm/error.log
daemonize = no
==================================================
============== php-fpm.d/www.conf ================
[www]
listen = /tmp/php-fpm.sock
listen.allowed_clients = 127.0.0.1
user = nginx
group = nginx
pm = dynamic
pm.max_children = 1024
pm.start_servers = 5
pm.min_spare_servers = 5
pm.max_spare_servers = 35
slowlog = /var/log/php-fpm/www-slow.log
php_flag[display_errors] = off
php_admin_value[error_log] = /var/log/php-fpm/www-error.log
php_admin_flag[log_errors] = on
php_admin_value[memory_limit] = 128M
php_value[session.save_handler] = files
php_value[session.save_path] = /var/lib/php/session
==================================================
================ mysql my.cnf ====================
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
symbolic-links=0
user=mysql
query_cache_limit = 2M
query_cache_size = 200M
query_cache_type=1
thread_cache_size=128
key_buffer = 100M
join_buffer = 2M
table_cache= 150M
sort_buffer= 2M
read_rnd_buffer_size=10M
tmp_table_size=200M
max_heap_table_size=200M
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
==================================================
=============== mysqltuner report ================
>> MySQLTuner 1.2.0 - Major Hayden <major at mhtx.net>
-------- General Statistics
--------------------------------------------------
[--] Skipped version check for MySQLTuner script
[OK] Currently running supported MySQL version 5.5.34
[OK] Operating on 64-bit architecture
-------- Storage Engine Statistics
-------------------------------------------
[--] Status: +Archive -BDB -Federated +InnoDB -ISAM -NDBCluster
[--] Data in MyISAM tables: 9M (Tables: 80)
[--] Data in InnoDB tables: 1M (Tables: 65)
[--] Data in PERFORMANCE_SCHEMA tables: 0B (Tables: 17)
[--] Data in MEMORY tables: 0B (Tables: 4)
[!!] Total fragmented tables: 66
-------- Security Recommendations
-------------------------------------------
[OK] All database users have passwords assigned
-------- Performance Metrics
-------------------------------------------------
[--] Up for: 12h 51m 16s (21K q [0.471 qps], 1K conn, TX: 10M, RX: 1M)
[--] Reads / Writes: 55% / 45%
[--] Total buffers: 694.0M global + 21.4M per thread (151 max threads)
[!!] Maximum possible memory usage: 3.8G (135% of installed RAM)
[OK] Slow queries: 0% (0/21K)
[OK] Highest usage of available connections: 23% (36/151)
[OK] Key buffer size / total MyISAM indexes: 150.0M/5.1M
[OK] Key buffer hit rate: 99.3% (51K cached / 358 reads)
[OK] Query cache efficiency: 80.9% (10K cached / 13K selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 55 sorts)
[OK] Temporary tables created on disk: 8% (5 on disk / 60 total)
[OK] Thread cache hit rate: 98% (36 created / 1K connections)
[OK] Table cache hit rate: 20% (192 open / 937 opened)
[OK] Open file limit used: 0% (210/200K)
[OK] Table locks acquired immediately: 99% (4K immediate / 4K locks)
[!!] Connections aborted: 8%
[OK] InnoDB data size / buffer pool: 1.1M/128.0M
==================================================
Please advise.
Thanks and Regards,
Nick
-------------- next part --------------
A non-text attachment was scrubbed...
Name: load_impact_1.png
Type: image/png
Size: 56882 bytes
Desc: not available
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20131011/c8d6445e/attachment-0001.png>
More information about the nginx
mailing list