Strange nginx issue

Steve Wilson lists-nginx at swsystem.co.uk
Mon Apr 7 15:45:36 UTC 2014


 

A quick read at
http://dev.mysql.com/doc/refman/4.1/en/innodb-parameters.html#sysvar_innodb_flush_log_at_trx_commit
[2]
suggests there's a possibility of losing 1s worth of data. I'm not sure
if we'd still have a problem with this now we've moved page caching to
memcache as that was causing a lot of updates. 

Unfortunately I'm at work so can't investigate other variables easily at
the moment, I'll hopefully have time this evening though. 

Steve.

On 07/04/2014 16:18, Geo P.C. wrote: 

> Thanks Steve for your update. We are using separate mysql server and in this innodb_flush_log_at_trx_commit = 1. This site is running money transactions applications so is it safe to change this option. 
> 
> Also to this mysql server other servers with default nginx and PHP5-fpm configuration is connecting it and they don't have such issues. For optimization we done the above changes and causing this issue. 
> 
> So can you please help me. 
> 
> On Mon, Apr 7, 2014 at 8:33 PM, Steve Wilson <lists-nginx at swsystem.co.uk> wrote:
> 
> I've just done a drupal7 site under nginx+php-fpm on debian. 
> 
> One thing I noticed was that the php process wasn't closing fast enough, this was tracked down to an issue with mysql. Connections were sitting idle for a long time which basically exhausted the fpm workers on both the web servers. 
> 
> We have 2 mysql nodes doing replication between them and I think the binlog commit was holding this up, adding "innodb_flush_log_at_trx_commit = 0" to the my.cnf stopped this problem from occuring. 
> 
> Page caching is stored in mysql too, moving this to memcached helped massively and reduced the daily binlogs from 5G/day down to a few hundred meg. 
> 
> I'm not sure if this is a strange setup but we have nginx terminating ssl which proxies to varnish which then has 2 additional nginx nodes serving drupal7, these use a 2 node mysql cluster and 2 memcached nodes for caching pages etc. 
> 
> Steve. 
> 
> On 07/04/2014 15:34, Geo P.C. wrote: 
> 
> We are facing a strange issue on our servers. We have servers with 1GB RAM and some drupal sites are running on it. 
> 
> Generally all sites are loading fine but sometimes we are unable to access any sites. After waiting for 10mts we are getting a 502 gateway timeout error. In middle when we restart either nginx or php5-fpm it will load. 
> 
> Our configurations are as follows: 
> 
> /etc/nginx/nginx.conf: 
> 
> user www-data; 
> 
> worker_processes 1; 
> 
> pid /run/nginx.pid; 
> 
> worker_rlimit_nofile 400000; 
> 
> events { 
> 
> worker_connections 10000; 
> 
> multi_accept on; 
> 
> use epoll; 
> 
> } 
> 
> http { 
> 
> access_log off; 
> 
> sendfile on; 
> 
> tcp_nopush on; 
> 
> tcp_nodelay on; 
> 
> keepalive_timeout 2; 
> 
> types_hash_max_size 2048; 
> 
> server_tokens off; 
> 
> keepalive_requests 100000; 
> 
> reset_timedout_connection on; 
> 
> port_in_redirect off; 
> 
> client_max_body_size 10m; 
> 
> proxy_connect_timeout 600s; 
> 
> proxy_send_timeout 600s; 
> 
> proxy_read_timeout 600s; 
> 
> fastcgi_send_timeout 600s; 
> 
> fastcgi_read_timeout 600s; 
> 
> open_file_cache max=200000 inactive=20s; 
> 
> open_file_cache_valid 30s; 
> 
> open_file_cache_min_uses 2; 
> 
> open_file_cache_errors on; 
> 
> /etc/php5/fpm/pool.d/www.conf 
> 
> pm.max_children = 5 
> 
> pm.start_servers = 2 
> 
> pm.min_spare_servers = 1 
> 
> pm.max_spare_servers = 3 
> 
> ;pm.process_idle_timeout = 10s; 
> 
> ;pm.max_requests = 200 
> 
> request_terminate_timeout = 300s 
> 
> And please see the added contents in /etc/sysctl.conf 
> 
> ########################## 
> 
> fs.file-max = 150000 
> 
> net.core.netdev_max_backlog=32768 
> 
> net.core.optmem_max=20480 
> 
> #net.core.rmem_default=65536 
> 
> #net.core.rmem_max=16777216 
> 
> net.core.somaxconn=50000 
> 
> #net.core.wmem_default=65536 
> 
> #net.core.wmem_max=16777216 
> 
> net.ipv4.tcp_fin_timeout=120 
> 
> #net.ipv4.tcp_keepalive_intvl=30 
> 
> #net.ipv4.tcp_keepalive_probes=3 
> 
> #net.ipv4.tcp_keepalive_time=120 
> 
> net.ipv4.tcp_max_orphans=262144 
> 
> net.ipv4.tcp_max_syn_backlog=524288 
> 
> net.ipv4.tcp_max_tw_buckets=524288 
> 
> #net.ipv4.tcp_mem=1048576 1048576 2097152 
> 
> #net.ipv4.tcp_no_metrics_save=1 
> 
> net.ipv4.tcp_orphan_retries=0 
> 
> #net.ipv4.tcp_rmem=4096 16384 16777216 
> 
> #net.ipv4.tcp_synack_retries=2 
> 
> net.ipv4.tcp_syncookies=1 
> 
> #net.ipv4.tcp_syn_retries=2 
> 
> #net.ipv4.tcp_wmem=4096 32768 16777216 
> 
> ########################## 
> 
> Can anyone please help us on it. 
> 
> Thanks 
> 
> Geo 
> 
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx [1]
> 
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx [1]

_______________________________________________
nginx mailing list
nginx at nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx [1]

-- 
Steve Wilson
IT Team Leader - Pirate Party UK
OpenDNS 2012 Sysadmin Awards: Flying Solo - Winner
+44.7751874508

Pirate Party UK is a political party registered with the Electoral
Commission.
 

Links:
------
[1] http://mailman.nginx.org/mailman/listinfo/nginx
[2]
http://dev.mysql.com/doc/refman/4.1/en/innodb-parameters.html#sysvar_innodb_flush_log_at_trx_commit
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20140407/946f18c6/attachment-0001.html>


More information about the nginx mailing list