"upstream sent too big header while reading response header from upstream"?

[Rafał Radecki]

Hi All.

I am currently trying to find the source of "upstream sent too big header
while reading response header from upstream" in my logfiles because nginx
as a consequence returns "502 Bad Gateway".

Basically I used tcpdump to get the traffic and I compared two cases:

1) the communication is ok:

12167 07:03:51.466937 IP 10.10.3.7.80 > 10.10.2.121.43817: Flags [P.], seq
1779345520:1779348777, ack 1001934214, win 219, options [nop,nop,TS val
2527240108 ecr 619023542], length 3257
12168 E...7. at .@...
12169
12170 ..
12171
12172 .y.P.)j..p;.M............
12173 ....$...HTTP/1.1 302 Found^M
12174 Content-Length: 58^M
12175 Content-Type: text/html; charset=utf-8^M
12176 Date: Wed, 03 Feb 2016 12:03:51 GMT^M
12177 Location: /^M
12178 Set-Cookie: rfid-mgt-console=...
12179 Set-Cookie: rfid-mgt-console.sig=xxx; path=/; expires=Wed, 03 Feb
2016 13:03:51 GMT; httponly^M
12180 Vary: Accept, Accept-Encoding^M
12181 X-Cf-Requestid: yyy^M
12182 X-Powered-By: Express^M
12183 ^M
12184 <p>Moved Temporarily. Redirecting to <a href="/">/</a></p>

In this case everything up to "Moved T..." is in one response from upstream
(10.10.3.7).

2) nginx throws mentioned error to error.log and returns 502 code
56874 06:31:45.307207 IP 10.10.3.7.80 > 10.10.2.121.58073: Flags [P.], seq
953075345:953079441, ack 3028520986, win 219, options [nop,nop,TS val
2526758567 ecr 617097381], length 4096
56875 E..4.. at .@...
56876
56877 ..
56878
56879 .y.P..8...........h......
56880 ..F.$.(.HTTP/1.1 302 Found^M
56881 Content-Length: 58^M
56882 Content-Type: text/html; charset=utf-8^M
56883 Date: Wed, 03 Feb 2016 11:31:45 GMT^M
56884 Location: /^M
56885 Set-Cookie: rfid-mgt-console=...
56886 Set-Cookie: rfid-mgt-console.sig=xxx; path=/; expires=Wed, 03 Feb
2016 12:31:45 GMT; httponly^M
56887 Vary: Accept, Accept-Encoding^M
56888 X-Cf-Requestid: yyy

56889 06:31:45.307213 IP 10.10.2.121.58073 > 10.10.3.7.80: Flags [.], ack
953079441, win 280, options [nop,nop,TS val 617097490 ecr 2526758567],
length 0
56890 E..4.. at .@.zw
56891
56892 .y
56893
56894 .....P....8..............
56895 $.)...F.

56896 06:31:45.307218 IP 10.10.3.7.80 > 10.10.2.121.58073: Flags [P.], seq
953079441:953079542, ack 3028520986, win 219, options [nop,nop,TS val
2526758567 ecr 617097381], length 101
56897 E..... at .@...
56898
56899 ..
56900
56901 .y.P..8...........JU.....
56902 ..F.$.(.503-564da79aecb5^M
56903 X-Powered-By: Express^M
56904 ^M
56905 <p>Moved Temporarily. Redirecting to <a href="/">/</a></p>

In this case response from upstream (10.10.3.7) is much larger because of
larger "Set-Cookie: rfid-mgt-console=..." returned and is divided into two
parts. "upstream sent too big header while reading response header from
upstream" is written to error.log.

For clarity I only pasted the part of the traffic which differs.

Initially in "http" section in nginx.conf:

...
http {
    proxy_max_temp_file_size 0;

    proxy_buffering off;
...

I tried to change it to:

...
http {
    proxy_max_temp_file_size 0;

    proxy_buffering on;

    proxy_buffers 8 256k;
...

And I also added:

...
    proxy_buffering on;

    proxy_buffers 8 256k;
...

to my "server" sections. I performed a restart but the error did not change.

Can someone help me with this one?

BR,
Rafal.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20160203/9934577e/attachment.html>