Nginx reload leading to ELB 502 on AWS Elastic Load Balancer

[gab]

Maxim Dounin Wrote:
-------------------------------------------------------
> Hello!
> 
> On Wed, Jan 19, 2022 at 07:24:20AM -0500, gab wrote:
> 
> > # Issue Summary
> > 
> > * After executing Nginx soft reload with "service nginx reload",
> nginx is
> > able to close a lot of connections gracefully, but some connections
> aren't
> > closed gracefully and Nginx is sending an RST packet. For these
> connections,
> > Nginx didn't send FIN packet, and it didn't send "Connection: Close"
> header.
> > The connections are HTTP/1.1 keep alive connections.
> > 
> > # Expected Behaviour
> > 
> > * After executing Nginx soft reload, Nginx is gracefully closing all
> > connections by sending a "Connection: close" header in the response
> or a FIN
> > packet.
> > 
> > # Supporting Data
> > 
> > I have tcpdump of 5 such connections where nginx didn't close the
> connection
> > gracefully after nginx reload.
> > Here's the tcpdump -
> >
> https://drive.google.com/file/d/1UquhmJET9i8ShEizu8453iUKpprutILV/view
> ?usp=sharing
> > 
> > If we analyse one such connection, we see that nginx didn't send FIN
> packet
> > on this connection, refer this image -
> https://i.imgur.com/zqyLOLc.png
> > 
> > If we see the response of second last request, we see nginx didn't
> send
> > "Connection: close" header either, refer this image -
> > https://i.imgur.com/P2uu722.png
> > 
> > In this image I have plotted FIN packets sent by nginx over time -
> > https://i.imgur.com/5lNAmnk.png
> > 
> > Nginx was reloaded on 2022-01-12 13:57:44 UTC.
> > 
> > FIN packet graph (https://i.imgur.com/5lNAmnk.png) shows that Nginx
> was able
> > to close a lot of connections gracefully at the time of reload, but
> it
> > wasn't able to close the 5 connections. TCP dump of which I've
> shared above
> >
> (https://drive.google.com/file/d/1UquhmJET9i8ShEizu8453iUKpprutILV/vie
> w?usp=sharing).
> 
> You may want to check if the following change improves things:
> 
> https://mailman.nginx.org/pipermail/nginx-devel/2022-January/014728.ht
> ml
> 
> Note though that it is generally the client's responsibility (AWS 
> ELB's) to retry requests in such cases.  See the this thread for 
> details:
> 
> https://mailman.nginx.org/pipermail/nginx-devel/2021-December/014681.h
> tml
> 
> -- 
> Maxim Dounin
> http://mdounin.ru/
> _______________________________________________
> nginx mailing list -- nginx at nginx.org
> To unsubscribe send an email to nginx-leave at nginx.org


Your commit will help a lot - https://hg.nginx.org/nginx/rev/96ae8e57b3dd

I see from this commit that the change will be a part of 1.21.6 release -
https://hg.nginx.org/nginx/rev/57581198e51e

Thanks Maxim!

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,293375,293396#msg-293396