<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:9.0pt;
font-family:"Verdana","sans-serif";
mso-fareast-language:EN-US;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Verdana","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="DA" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoPlainText"><span lang="EN-GB">On Friday, September 30, 2016 12:55 AM Francis Daly wrote,<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">>> No, I have an "error_page 503" and a similar one for 404 that points to two named locations, but that's it.<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">> That might matter.<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">> I can now get a 503, 404, or 405 result from nginx, when upstream sends a 503.<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">[...]<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">> Now make /tmp/x exist, and /tmp/y not exist.<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">> A GET request for /x is proxied, gets a 503, and returns the content of /tmp/x with a 503 status.<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">> A GET request for /y is proxied, gets a 503, and returns a 404 status.<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">> A POST request for /x is proxied, gets a 503, and returns a 405 status.<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">> A POST request for /y is proxied, gets a 503, and returns a 404 status.<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">> Since you also have an error_page for 404, perhaps that does something that leads to the output that you see.<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">> I suspect that when you show your error_page config and the relevant<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">> locations, it may become clearer what you want to end up with.<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">My local test config looks like this (log specifications and other stuff left out):<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> server {<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> listen 80;<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> server_name localhost;<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> location / {<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> root html;<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> try_files /offline.html @xact;<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> add_header Cache-Control "no-cache, max-age=0, no-store, must-revalidate";<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> }<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> location @xact {<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> proxy_pass http://127.0.0.1:4431;<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> proxy_redirect default;<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> proxy_read_timeout 2s;<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> proxy_send_timeout 2s;<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> proxy_connect_timeout 2s;<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> proxy_intercept_errors on;<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> }<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> error_page 404 @error_404;<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> error_page 503 @error_503;<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> location @error_404 {<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> root error;<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> rewrite (logo.png)$ /$1 break;<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> rewrite ^(.*)$ /error404.html break;<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> }<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> location @error_503 {<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> root error;<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> rewrite (logo.png)$ /$1 break;<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> rewrite ^(.*)$ /error503.html break;<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> }<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">> A test system which talks to a local HAProxy which has no "up" backends<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">> would probably be quicker to build.<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">Yes, thats what I had listening on 127.0.0.1:4431, and it did give me the same behaviour as I'm seeing in our production environment.<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">I got the following captures via pcap and wireshark:<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">Conditions are: HAProxy has a backend with no available servers, so every request results in a 503 to upstream client (nginx).<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">A POST request to some resource from a browser:<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> POST /2 HTTP/1.1 <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> Host: localhost <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:47.0) Gecko/20100101 Firefox/47.0
<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> Accept-Language: en <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> Accept-Encoding: gzip, deflate <o:p>
</o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> DNT: 1 <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> Content-Type: application/x-www-form-urlencoded
<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> Content-Length: 0 <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> Cookie: new-feature=1; Language_In_Use=
<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> Connection: keep-alive<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">This makes nginx send this request to HAProxy:<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> POST /2 HTTP/1.0 <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> Host: 127.0.0.1:4431 <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> Connection: close <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> Content-Length: 0 <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:47.0) Gecko/20100101 Firefox/47.0
<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> Accept-Language: en <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> Accept-Encoding: gzip, deflate <o:p>
</o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> DNT: 1 <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> Content-Type: application/x-www-form-urlencoded
<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> Cookie: new-feature=1; Language_In_Use=
<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">HAProxy returns this:<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> HTTP/1.0 503 Service Unavailable <o:p>
</o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> Cache-Control: no-cache <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> Connection: close <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> Content-Type: text/html <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> <html><body><h1>503 Service Unavailable</h1><o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> No server is available to handle this request.<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> </body></html><o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">HAProxy also logs this (raw syslog packet):<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> <134>Oct 15 13:17:33 jedc-local haproxy[10104]: 127.0.0.1:64746 [15/Oct/2016:13:17:33.800] xact_in-DK xact_admin/<NOSRV> 0/-1/-1/-1/0 503 212 - - SC-- 0/0/0/0/0 0/0 "POST /2 HTTP/1.0"<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">This makes nginx return this back to the browser:<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> HTTP/1.1 405 Not Allowed <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> Server: nginx/1.8.0 <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> Date: Sat, 15 Oct 2016 11:17:33 GMT <o:p>
</o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> Content-Type: text/html <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> Content-Length: 172 <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> Connection: keep-alive<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">nginx also logs this:<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"> localhost 127.0.0.1 "-" [15/Oct/2016:13:17:33 +0200] "POST /2 HTTP/1.1" 405 172 503 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:47.0) Gecko/20100101 Firefox/47.0" http "-" "-" "-" "-" -/-<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">There is no mention of the error_page 503 location or any of the resources they specify (logo.png or error503.html) in any of nginx’ logs, so I assume that they are not really connected to the problems I see.<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">Any ideas?<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">Regards,<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">Jens Dueholm Christensen<o:p></o:p></span></p>
</div>
</body>
</html>