<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">We have recently migrated across from HAProxy to Nginx because it supports true zero-downtime configuration reloads. However, we are occasionally getting 502 and 504 errors from our monitoring systems during deployments. Looking into this, I have been able to consistently replicate the 502 and 504 errors as follows. I believe this is an error in how Nginx handles in-flight requests, but wanted to ask the community in case I am missing something obvious.<div class=""><br class=""></div><div class="">Note the set up of Nginx is as follows:</div><div class="">* Ubuntu 14.04</div><div class="">* Nginx version 1.9.1</div><div class="">* Configuration for an HTTP listener:</div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""> map $http_upgrade $connection_upgrade {</div><div class=""> default upgrade;</div><div class=""> '' close;</div><div class=""> }</div><div class=""> server {</div><div class=""> listen 8080;</div><div class=""> # pass on real client's IP</div><div class=""> proxy_set_header X-Real-IP $remote_addr;</div><div class=""> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;</div><div class=""> access_log /var/log/nginx/access.ws-8080.log combined;</div><div class=""><br class=""></div><div class=""> location / {</div><div class=""> proxy_pass <a href="http://server-ws-8080" class="">http://server-ws-8080</a>;</div><div class=""> proxy_http_version 1.1;</div><div class=""> proxy_set_header Upgrade $http_upgrade;</div><div class=""> proxy_set_header Connection $connection_upgrade;</div><div class=""> }</div><div class=""> }</div><div class=""><br class=""></div><div class=""> upstream server-ws-8080 {</div><div class=""> least_conn;</div><div class=""> server 172.17.0.51:8080 max_fails=0;</div><div class=""> }</div></blockquote><div class=""><div class=""><br class=""></div><div class="">1. Telnet to the Nginx server on the HTTP port it is listening on.</div><div class=""><br class=""></div><div class="">2. Send a HTTP/1.1 request to the upstream server (172.17.0.51):</div></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><div class=""><div class="">GET /health HTTP/1.1</div><div class="">Host: localhost</div><div class="">Connection: Keep-Alive</div></div></div><div class=""><br class=""></div></blockquote>This request succeeds and the response is valid<br class=""><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><br class=""></div></blockquote>3. Start a new HTTP/1.1 request but don’t finish the request i.e. send the following line using telnet:<blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><div class=""><div class="">GET /health HTTP/1.1</div></div></div><div class=""><br class=""></div></blockquote>4. Whilst that request is now effectively in-flight because it’s not finished and Nginx is waiting for the request to be completed, reconfigure Nginx with a SIGHUP signal. The only difference in the config preceding the SIGHUP signal is that the upstream server has changed i.e. we intentionally want all new requests to go to the new upstream server.<div class=""><br class=""></div><div class="">5. Terminate the old upstream server 172.17.0.51<br class=""><div class=""><br class=""></div><div class="">6. Complete the in-flight HTTP/1.1 request started in point 3 above with:</div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><div class=""><div class="">Host: localhost</div><div class="">Connection: Keep-Alive</div></div></div><div class=""><br class=""></div></blockquote>7. Nginx will consistently respond with a 502 if the old upstream server rejects the request, or a 504 if there is no response on that IP and port. </div><div class=""><br class=""></div><div class="">I believe this behaviour is incorrect as Nginx, once it receives the complete request, should direct the request to the current available upstream server. However, it seems that that Nginx is instead deciding which upstream server to send the request to before the request is completed and as such is directing the request to a server that no longer exists.</div><div class=""><br class=""></div><div class="">Any advice appreciated.</div><div class=""><br class=""></div><div class="">BTW. I tried to raise an issue on <a href="http://trac.nginx.com/" class="">http://trac.nginx.com/</a>, however it seems that the authentication system is completely broken. I tried logging in with Google, My Open Id, Wordpress and Yahoo, and all of those OpenID providers no longer work. </div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Matt<br class=""><div class=""><div class=""><div class=""><div class=""><br class=""></div></div></div></div></div></body></html>