Nginx reload leading to ELB 502 on AWS Elastic Load Balancer

[Maxim Dounin]

Hello!

On Wed, Jan 19, 2022 at 07:24:20AM -0500, gab wrote:

> # Issue Summary
> 
> * After executing Nginx soft reload with "service nginx reload", nginx is
> able to close a lot of connections gracefully, but some connections aren't
> closed gracefully and Nginx is sending an RST packet. For these connections,
> Nginx didn't send FIN packet, and it didn't send "Connection: Close" header.
> The connections are HTTP/1.1 keep alive connections.
> 
> # Expected Behaviour
> 
> * After executing Nginx soft reload, Nginx is gracefully closing all
> connections by sending a "Connection: close" header in the response or a FIN
> packet.
> 
> # Supporting Data
> 
> I have tcpdump of 5 such connections where nginx didn't close the connection
> gracefully after nginx reload.
> Here's the tcpdump -
> https://drive.google.com/file/d/1UquhmJET9i8ShEizu8453iUKpprutILV/view?usp=sharing
> 
> If we analyse one such connection, we see that nginx didn't send FIN packet
> on this connection, refer this image - https://i.imgur.com/zqyLOLc.png
> 
> If we see the response of second last request, we see nginx didn't send
> "Connection: close" header either, refer this image -
> https://i.imgur.com/P2uu722.png
> 
> In this image I have plotted FIN packets sent by nginx over time -
> https://i.imgur.com/5lNAmnk.png
> 
> Nginx was reloaded on 2022-01-12 13:57:44 UTC.
> 
> FIN packet graph (https://i.imgur.com/5lNAmnk.png) shows that Nginx was able
> to close a lot of connections gracefully at the time of reload, but it
> wasn't able to close the 5 connections. TCP dump of which I've shared above
> (https://drive.google.com/file/d/1UquhmJET9i8ShEizu8453iUKpprutILV/view?usp=sharing).

You may want to check if the following change improves things:

https://mailman.nginx.org/pipermail/nginx-devel/2022-January/014728.html

Note though that it is generally the client's responsibility (AWS 
ELB's) to retry requests in such cases.  See the this thread for 
details:

https://mailman.nginx.org/pipermail/nginx-devel/2021-December/014681.html

-- 
Maxim Dounin
http://mdounin.ru/