[PATCH] HTTP: keepalive_graceful_close support

[Maxim Dounin]

Hello!

On Tue, Dec 28, 2021 at 03:36:22PM +0000, 封 幼麟 wrote:

> In my scenario, nginx is used with consul-template as a load 
> balancer that implements service discovery. Each time a service 
> instance is registered to or unregistered from 
> consul，consul-template will trigger all nginx instances to 
> reload. In a busy system, this happens very frequently, and each 
> reload will cause many "keepalived" connections to be closed. 
> The client based on golang got a lot of "server closed idle 
> connection" and "EOF" errors.

The "server closed idle connection" doesn't look like an error at 
all.  Not sure what "EOF" means, but either way it looks more like 
an issue in the client than something to address in nginx.

> Of course, the client should implement the "auto retry" logic, 
> but I think we have a better way to completely solve this 
> problem in nginx. My idea is, don't treat the keepalived http 
> connections as idle connections, and send a "Connection: close" 
> header in the next response. If no more requests come, the 
> worker process will wait until "keepalive_timeout", then exit, 
> and won't wait long. So, if we give the client a smaller 
> "IdleConnTimeout", there is no race now. If someone doesn't want 
> to use this patch, just leave "keepalive_graceful_close" unset.

To re-iterate: even with your patch (which basically returns the 
nginx behaviour before 0.5.15), the race is still here, since 
there are cases when keepalive connections are closed (notably, 
when keepalive_timeout expires, or when there are no free 
connections).  That is, the patch basically tries to hide client 
issues in some common case at the cost of keeping old worker 
processes alive till keepalive_timeout expiration.  Still, these 
issues can re-appear in other cases, and the only complete 
solution can be implemented on the client.

-- 
Maxim Dounin
http://mdounin.ru/