<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto">Your web server logs should have the key to solving this.<div><br></div><div>Do you know what url was being requested? Do the URLs look valid?</div><div>Are there requests all for the same resource? </div><div>Are the requests coming from a single IP range?</div><div>Are the requests all coming with the same user-agent?</div><div>Does the time this started mean anything?</div><div>Is it static content? </div><div>Was the user-agent the same or different?</div><div><br></div><div>You say that this is proxied content. Is it something you can microcache?</div><div><br><div><br></div><div>I once saw a DDOS attack on a retail site that did business in most countries but not China, for an expensive dynamic page. We were using nginx to cache the page but the bots were requesting a URL that had an additional cache-busting parameter. The source IPs were all from Shanghai so we knew they were problematic.</div><div><br></div><div>I’ve seen problems like the one you describe caused by many different things- redirect loops, CDN configuration problems, website misconfigurations that confuse google or bing, toxic test agents, badly designed webpages, - but the bottom line is that if you understand the patterns in your web server logs then you will know what’s happening</div><div><br></div><div>If the requests<br><br><div id="AppleMailSignature" dir="ltr">Sent from my iPhone</div><div dir="ltr"><br>On Jan 10, 2019, at 1:04 PM, Anoop Alias <<a href="mailto:anoopalias01@gmail.com">anoopalias01@gmail.com</a>> wrote:<br><br></div><blockquote type="cite"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">The issue was identified to be an enormous number of http request ( attack) to one of the hosted domains that was using cloudflare. The traffic is coming in from cloudflare and this was causing nginx to be exhausted in terms of the TCP stack<div><br></div><div>#########################################</div><div><div># netstat -tn|awk '{print $6}'|sort|uniq -c</div><div> 1 </div><div> 19922 CLOSE_WAIT</div><div> 2 CLOSING</div><div> 23528 ESTABLISHED</div><div> 17785 FIN_WAIT1</div><div> 4 FIN_WAIT2</div><div> 1 Foreign</div><div> 17 LAST_ACK</div><div> 904 SYN_RECV</div><div> 14 SYN_SENT</div><div> 142 TIME_WAIT</div></div><div>############################################</div><div><br></div><div>Interestingly with the same attack, removing Nginx from the picture and exposing httpd cause the connections to be fine</div><div><br></div><div>############################################</div><div><div>]# netstat -tn|awk '{print $6}'|sort|uniq -c</div><div> 1 </div><div> 39 CLOSE_WAIT</div><div> 9 CLOSING</div><div> 664 ESTABLISHED</div><div> 13 FIN_WAIT1</div><div> 48 FIN_WAIT2</div><div> 1 Foreign</div><div> 24 LAST_ACK</div><div> 8 SYN_RECV</div><div> 12 SYN_SENT</div><div> 1137 TIME_WAIT</div></div><div>##############################################</div><div><br></div><div>Although the load is a bit high than usual.</div><div><br></div><div>It looks like the TCP connections in the established state is somehow piling up with Nginx </div><div><br></div><div>Number of established connections over time with nginx</div><div>##############</div><div><div>535 ESTABLISHED</div><div>1195 ESTABLISHED</div><div>23437 ESTABLISHED</div><div>23490 ESTABLISHED</div><div>23482 ESTABLISHED</div><div>389 ESTABLISHED</div></div><div>##############</div><div><br></div><div>I think this could be a misconfiguration in Nginx?. Would be great if someone points out what is wrong with the config </div><div><br></div><div>Thanks,</div><div><br></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr">On Thu, Jan 10, 2019 at 8:27 AM Anoop Alias <<a href="mailto:anoopalias01@gmail.com">anoopalias01@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi,<div><br></div><div>Have had a really strange issue on a Nginx server configured as a reverse proxy wherein the server stops responding when the network connections in ESTABLISHED state and FIN_WAIT state in very high compared to normal working </div><div><br></div><div>If you see the below network graph, at around 00:30 hours there is a big spike in network connections in FIN_WAIT state, to around 12000 from the normal value of ~20</div><div><br></div><div><a href="https://i.imgur.com/wb6VMWo.png" target="_blank">https://i.imgur.com/wb6VMWo.png</a><br clear="all"><div><br></div><div>At this state, Nginx stops responding fully and does not work even after a full restart of the service.</div><div><br></div><div>Switching off Nginx and bring Apache service to the frontend (removing the reverse proxy) fix this and the connections drop</div><div><br></div><div>Nginx config & build setting</div><div>##################################</div><div><div> nginx -V</div><div>nginx version: nginx/1.15.8</div><div>built by gcc 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC) </div><div>built with LibreSSL 2.8.3</div><div>TLS SNI support enabled</div><div>configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/etc/nginx/modules --with-pcre=./pcre-8.42 --with-pcre-jit --with-zlib=./zlib-1.2.11 --with-openssl=./libressl-2.8.3 --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error_log --http-log-path=/var/log/nginx/access_log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/dev/shm/client_temp --http-proxy-temp-path=/dev/shm/proxy_temp --http-fastcgi-temp-path=/dev/shm/fastcgi_temp --http-uwsgi-temp-path=/dev/shm/uwsgi_temp --http-scgi-temp-path=/dev/shm/scgi_temp --user=nobody --group=nobody --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-http_auth_request_module --with-file-aio --with-threads --with-stream --with-stream_ssl_module --with-http_slice_module --with-compat --with-http_v2_module --add-dynamic-module=incubator-pagespeed-ngx-1.13.35.2-stable --add-dynamic-module=/usr/local/rvm/gems/ruby-2.5.3/gems/passenger-6.0.0/src/nginx_module --add-dynamic-module=ngx_brotli --add-dynamic-module=echo-nginx-module-0.61 --add-dynamic-module=headers-more-nginx-module-0.32 --add-dynamic-module=ngx_http_redis-0.3.8 --add-dynamic-module=redis2-nginx-module --add-dynamic-module=srcache-nginx-module-0.31 --add-dynamic-module=ngx_devel_kit-0.3.0 --add-dynamic-module=set-misc-nginx-module-0.31 --add-dynamic-module=ngx_http_geoip2_module --add-dynamic-module=testcookie-nginx-module --add-dynamic-module=ModSecurity-nginx --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic' --with-ld-opt=-Wl,-E</div></div><div><br></div><div>#####################################</div><div><div># worker_processes auto; #Set to auto for a powerful server</div><div>worker_processes 1;</div><div>worker_rlimit_nofile 69152;</div><div>worker_shutdown_timeout 10s;</div><div># worker_cpu_affinity auto;</div><div>timer_resolution 1s;</div><div>thread_pool iopool threads=32 max_queue=65536;</div><div>pcre_jit on;</div><div>pid /var/run/nginx.pid;</div><div>error_log /var/log/nginx/error_log;</div><div><br></div><div>#Load Dynamic Modules</div><div>include /etc/nginx/modules.d/*.load;</div><div><br></div><div><br></div><div>events {</div><div>worker_connections 20480;</div><div>use epoll;</div><div>multi_accept on;</div><div>accept_mutex off;</div><div>}</div></div><div><br></div><div><div>lingering_close off;</div><div>limit_req zone=FLOODVHOST burst=200;</div><div>limit_req zone=FLOODPROTECT burst=200;</div><div>limit_conn PERSERVER 60;</div><div>client_header_timeout 5s;</div><div>client_body_timeout 5s;</div><div>send_timeout 5s;</div><div>keepalive_timeout 0;</div><div>http2_idle_timeout 20s;</div><div>http2_recv_timeout 20s;</div></div><div><br></div><div><br></div><div><div>aio threads=iopool;</div><div>aio_write on;</div><div>directio 64m;</div><div>output_buffers 2 512k;</div><div><br></div><div>tcp_nodelay on;</div><div><br></div><div>types_hash_max_size 4096;</div><div>server_tokens off;</div><div>client_max_body_size 2048m;</div><div>reset_timedout_connection on;</div><div><br></div><div>#Proxy</div><div>proxy_read_timeout 300;</div><div>proxy_send_timeout 300;</div><div>proxy_connect_timeout 30s;</div></div><div><br></div><div><div>#FastCGI</div><div>fastcgi_read_timeout 300;</div><div>fastcgi_send_timeout 300;</div><div>fastcgi_connect_timeout 30s;</div></div><div><br></div><div><div>#Proxy Buffer</div><div>proxy_buffering on;</div><div>proxy_buffer_size 128k;</div><div>proxy_buffers 8 128k;</div><div>proxy_busy_buffers_size 256k;</div><div><br></div><div>#FastCGI Buffer</div><div>fastcgi_buffer_size 128k;</div><div>fastcgi_buffers 8 128k;</div><div>fastcgi_busy_buffers_size 256k;</div></div><div><br></div><div><div>server_names_hash_max_size 2097152;</div><div>server_names_hash_bucket_size 128;</div></div><div>######################################################</div><div><br></div><div><br></div><div><br></div>-- <br><div dir="ltr" class="gmail-m_-3316508141525942457gmail_signature"><div dir="ltr"><div><b>Anoop P Alias</b> <div><br></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><b>Anoop P Alias</b> <div><br></div></div></div></div>
</div></blockquote><blockquote type="cite"><div dir="ltr"><span>_______________________________________________</span><br><span>nginx mailing list</span><br><span><a href="mailto:nginx@nginx.org">nginx@nginx.org</a></span><br><span><a href="http://mailman.nginx.org/mailman/listinfo/nginx">http://mailman.nginx.org/mailman/listinfo/nginx</a></span></div></blockquote></div></div></body></html>