Upstream Backup Issue

[Nathan Folkman]

Hi,

We have the following configuration, running nginx 0.6.34:

...
upstream frontend {
	server host_a;
	server host_b;
	server host_c;
	server host_d;
	server failover_host backup;
}

server {
         listen 80;
         server_name _;
         error_page 502 503 504 /error.png;

         location / {
             proxy_set_header Host $host;
             proxy_set_header X-Real-IP $remote_addr;
             proxy_pass http://frontend;
             proxy_connect_timeout 5;
             proxy_send_timeout 30;
             proxy_read_timeout 30;
             charset utf-8;
         }
}
...

Everything has been working great for weeks, and then today we started  
seeing the following up stream errors:

[error] 28016#0: *742426639 upstream sent invalid header while reading  
response header from upstream, client: CLIENT, server: _, request:  
"GET REQUEST HTTP/1.1", upstream: "ADDRESS", host: "HOST"

The errors were caused by a malformed HTTP redirect message, that we  
were constructing upstream, that contained embedded newline characters  
in the "Location" HTTP header portion of the response. It seems like  
what was happening was that once I'd get 4 of those errors, the backup  
would kick in for a few seconds and then everything would recover.

Is this correct behavior? The upstream hosts were technically not  
down, and were all responding so there weren't any timeout issues. Is  
there a way to prevent this from happening unless all of the upstream  
hosts are truly down or non-responsive, as opposed to simply returning  
invalid responses? Could this be a bug?

Thanks in advance - love nginx! :)

- n