Quick performance deterioration when No. of clients increases
Steve Holdoway
steve at greengecko.co.nz
Fri Oct 11 08:18:39 UTC 2013
The ultimate bottleneck in any setup like this is usually raw cpu
power. A single virtual core doesn't look like it'll hack it. You've
got 35 php processes serving 250 users, and I think it's just spread a
bit thin.
Apart from adding cores, there are 2 things I'd suggest looking at
- are you using an opcode cacher? APC ( install via pecl to get the
latest ) works really well with php in fpm... allocate plenty of memory
to it too
- check the bandwidth at the network interface. The usual 100Mbit
connection can easily get swamped by a graphics rich site - especially
with 250 concurrent users. If this is a problem, then look at using a
CDN to ease things.
hth,
Steve
On 11/10/13 21:07, Nikolaos Milas wrote:
> Hello,
>
> I am trying to migrate a Joomla 2.5.8 website from Apache to NGINX
> 1.4.2 with php-fpm 5.3.3 (and MySQL 5.5.34) on a CentOS 6.4 x86_64
> Virtual Machine (running under KMS).
>
> The goal is to achieve better peak performance: This site has
> occasional high peaks; while the normal traffic is ~10 req/sec, it may
> reach > 3000 req/sec for periods of a few hours (due to the type of
> services the site provides - it is a non-profit, real-time
> seismicity-related site - so php caching should not be more than 10
> seconds).
>
> The new VM (using Nginx) currently is in testing mode and it only has
> 1-core CPU / 3 GB of RAM. We tested performance with loadimpact and
> the results are attached.
>
> You can see at the load graph that as the load approaches 250 clients,
> the response time increases very much and is already unacceptable
> (this happens consistently). I expected better performance, esp. since
> caching is enabled. Despite many efforts, I cannot find the cause of
> the bottleneck, and how to deal with it. We would like to achieve
> better scaling, esp. since NGINX is famous for its scaling
> capabilities. Having very little experience with Nginx, I would like
> to ask for your assistance for a better configuration.
>
> When this performance deterioration occurs, we don't see very high CPU
> load (Unix load peaks 2.5), neither RAM exhaustion (System RAM usage
> appears to be below 30%). [Monitoring is through Nagios.]
>
> Can you please guide me on how to correct this issue? Any and all
> suggestions will be appreciated.
>
> Current configuration, based on info available on the Internet, is as
> follows (replaced true domain/website name and public IP address(es)):
>
> =================== Nginx.conf ===================
>
> user nginx;
> worker_processes 1;
>
> error_log /var/log/nginx/error.log warn;
> pid /var/run/nginx.pid;
>
> worker_rlimit_nofile 200000;
>
> events {
> worker_connections 8192;
> multi_accept on;
> use epoll;
> }
>
> http {
> include /etc/nginx/mime.types;
> default_type application/octet-stream;
> server_names_hash_bucket_size 64;
>
> log_format main '$remote_addr - $remote_user [$time_local]
> "$request" '
> '$status $body_bytes_sent "$http_referer" '
> '"$http_user_agent" "$http_x_forwarded_for"';
>
> log_format cache '$remote_addr - $remote_user [$time_local]
> "$request" '
> '$status $upstream_cache_status $body_bytes_sent
> "$http_referer" '
> '"$http_user_agent" "$http_x_forwarded_for"';
>
> fastcgi_cache_path /var/cache/nginx levels=1:2
> keys_zone=microcache:5m max_size=1000m;
>
> access_log /var/log/nginx/access.log main;
>
> sendfile on;
>
> tcp_nopush on;
> tcp_nodelay on;
> keepalive_timeout 2;
>
> types_hash_max_size 2048;
> server_tokens off;
>
> keepalive_requests 30;
>
> open_file_cache max=5000 inactive=20s;
> open_file_cache_valid 30s;
> open_file_cache_min_uses 2;
> open_file_cache_errors on;
>
> gzip on;
> gzip_static on;
> gzip_disable "msie6";
> gzip_http_version 1.1;
> gzip_vary on;
> gzip_comp_level 6;
> gzip_proxied any;
> gzip_types text/plain text/css application/json
> application/x-javascript text/xml application/xml application/xml+rss
> text/javascript application/javascript text/x-js;
> gzip_buffers 16 8k;
>
> include /etc/nginx/conf.d/*.conf;
> }
>
> ==================================================
>
> ================ website config ==================
>
> server {
> listen 80;
> server_name www.example.com;
> access_log /var/webs/wwwexample/log/access_log main;
> error_log /var/webs/wwwexample/log/error_log warn;
> root /var/webs/wwwexample/www/;
>
> index index.php index.html index.htm index.cgi default.html
> default.htm default.php;
> location / {
> try_files $uri $uri/ /index.php?$args;
> }
>
> location /nginx_status {
> stub_status on;
> access_log off;
> allow 10.10.10.0/24;
> deny all;
> }
>
> location ~*
> /(images|cache|media|logs|tmp)/.*\.(php|pl|py|jsp|asp|sh|cgi)$ {
> return 403;
> error_page 403 /403_error.html;
> }
>
> location ~ /\.ht {
> deny all;
> }
>
> location /administrator {
> allow 10.10.10.0/24;
> deny all;
> }
>
> location ~ \.php$ {
>
> # Setup var defaults
> set $no_cache "";
> # If non GET/HEAD, don't cache & mark user as uncacheable for
> 1 second via cookie
> if ($request_method !~ ^(GET|HEAD)$) {
> set $no_cache "1";
> }
> # Drop no cache cookie if need be
> # (for some reason, add_header fails if included in prior
> if-block)
> if ($no_cache = "1") {
> add_header Set-Cookie "_mcnc=1; Max-Age=2; Path=/";
> add_header X-Microcachable "0";
> }
> # Bypass cache if no-cache cookie is set
> if ($http_cookie ~* "_mcnc") {
> set $no_cache "1";
> }
> # Bypass cache if flag is set
> fastcgi_no_cache $no_cache;
> fastcgi_cache_bypass $no_cache;
> fastcgi_cache microcache;
> fastcgi_cache_key $scheme$host$request_uri$request_method;
> fastcgi_cache_valid 200 301 302 10s;
> fastcgi_cache_use_stale updating error timeout invalid_header
> http_500;
> fastcgi_pass_header Set-Cookie;
> fastcgi_pass_header Cookie;
> fastcgi_ignore_headers Cache-Control Expires Set-Cookie;
>
> try_files $uri =404;
> include /etc/nginx/fastcgi_params;
> fastcgi_param PATH_INFO $fastcgi_script_name;
> fastcgi_intercept_errors on;
>
> fastcgi_buffer_size 128k;
> fastcgi_buffers 256 16k;
> fastcgi_busy_buffers_size 256k;
> fastcgi_temp_file_write_size 256k;
> fastcgi_read_timeout 240;
>
> fastcgi_pass unix:/tmp/php-fpm.sock;
>
> fastcgi_index index.php;
> include /etc/nginx/fastcgi_params;
> fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
>
> }
>
> location ~* \.(ico|pdf|flv)$ {
> expires 1d;
> }
>
> location ~* \.(js|css|png|jpg|jpeg|gif|swf|xml|txt)$ {
> expires 1d;
> }
>
> }
> ==================================================
>
> ================= php-fpm.conf ===================
> include=/etc/php-fpm.d/*.conf
> [global]
> pid = /var/run/php-fpm/php-fpm.pid
> error_log = /var/log/php-fpm/error.log
>
> daemonize = no
> ==================================================
>
> ============== php-fpm.d/www.conf ================
>
> [www]
> listen = /tmp/php-fpm.sock
> listen.allowed_clients = 127.0.0.1
> user = nginx
> group = nginx
>
> pm = dynamic
> pm.max_children = 1024
> pm.start_servers = 5
> pm.min_spare_servers = 5
> pm.max_spare_servers = 35
>
> slowlog = /var/log/php-fpm/www-slow.log
>
> php_flag[display_errors] = off
> php_admin_value[error_log] = /var/log/php-fpm/www-error.log
> php_admin_flag[log_errors] = on
> php_admin_value[memory_limit] = 128M
>
> php_value[session.save_handler] = files
> php_value[session.save_path] = /var/lib/php/session
>
> ==================================================
>
> ================ mysql my.cnf ====================
>
> [mysqld]
> datadir=/var/lib/mysql
> socket=/var/lib/mysql/mysql.sock
> symbolic-links=0
> user=mysql
>
> query_cache_limit = 2M
> query_cache_size = 200M
> query_cache_type=1
> thread_cache_size=128
> key_buffer = 100M
> join_buffer = 2M
> table_cache= 150M
> sort_buffer= 2M
> read_rnd_buffer_size=10M
> tmp_table_size=200M
> max_heap_table_size=200M
>
> [mysqld_safe]
> log-error=/var/log/mysqld.log
> pid-file=/var/run/mysqld/mysqld.pid
>
> ==================================================
>
> =============== mysqltuner report ================
>
> >> MySQLTuner 1.2.0 - Major Hayden <major at mhtx.net>
>
> -------- General Statistics
> --------------------------------------------------
> [--] Skipped version check for MySQLTuner script
> [OK] Currently running supported MySQL version 5.5.34
> [OK] Operating on 64-bit architecture
>
> -------- Storage Engine Statistics
> -------------------------------------------
> [--] Status: +Archive -BDB -Federated +InnoDB -ISAM -NDBCluster
> [--] Data in MyISAM tables: 9M (Tables: 80)
> [--] Data in InnoDB tables: 1M (Tables: 65)
> [--] Data in PERFORMANCE_SCHEMA tables: 0B (Tables: 17)
> [--] Data in MEMORY tables: 0B (Tables: 4)
> [!!] Total fragmented tables: 66
>
> -------- Security Recommendations
> -------------------------------------------
> [OK] All database users have passwords assigned
>
> -------- Performance Metrics
> -------------------------------------------------
> [--] Up for: 12h 51m 16s (21K q [0.471 qps], 1K conn, TX: 10M, RX: 1M)
> [--] Reads / Writes: 55% / 45%
> [--] Total buffers: 694.0M global + 21.4M per thread (151 max threads)
> [!!] Maximum possible memory usage: 3.8G (135% of installed RAM)
> [OK] Slow queries: 0% (0/21K)
> [OK] Highest usage of available connections: 23% (36/151)
> [OK] Key buffer size / total MyISAM indexes: 150.0M/5.1M
> [OK] Key buffer hit rate: 99.3% (51K cached / 358 reads)
> [OK] Query cache efficiency: 80.9% (10K cached / 13K selects)
> [OK] Query cache prunes per day: 0
> [OK] Sorts requiring temporary tables: 0% (0 temp sorts / 55 sorts)
> [OK] Temporary tables created on disk: 8% (5 on disk / 60 total)
> [OK] Thread cache hit rate: 98% (36 created / 1K connections)
> [OK] Table cache hit rate: 20% (192 open / 937 opened)
> [OK] Open file limit used: 0% (210/200K)
> [OK] Table locks acquired immediately: 99% (4K immediate / 4K locks)
> [!!] Connections aborted: 8%
> [OK] InnoDB data size / buffer pool: 1.1M/128.0M
>
> ==================================================
>
> Please advise.
>
> Thanks and Regards,
> Nick
>
>
>
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20131011/bbbe5124/attachment-0001.html>
More information about the nginx
mailing list