limit_conn is dropping valid connections and causing memory leaks on nginx reload

[Valentin V. Bartenev]

On Saturday 30 September 2017 06:05:15 Dejan Grofelnik Pelzel wrote:
> Hello,
> 
> We are running the nginx 1.13.5  with HTTP/2 in a proxy_pass proxy_cache
> configuration with clients having relatively long open connections. Our
> system does automatic reloads for any new configuration and we recently
> introduced a limit_conn to some of the config files. After that, I've
> started noticing a rapid drop in connections and outgoing network every-time
> the system would perform a configuration reload. Even stranger, on every
> reload the memory usage would go up for about 1-2GB until ultimately
> everything crashed if the reloads were too frequent. The memory usage did go
> down after old workers were released, but that could take up to 30 minutes,
> while the configuration could get reloaded up to twice per minute.
> 
> We used the following configuration as recommended by pretty much any
> example:
> limit_conn_zone $binary_remote_addr zone=1234con:10m;
> limit_conn zone1234con 10;
> 
> I was able to verify the connection drop by doing a simple ab test, for
> example, I would run ab -c 100 -n -k 1000 https://127.0.0.1/file.bin
> 990 of the connections went through, however, 10 would still be active.
> Immediately after the reload, those would get dropped as well. Adding -r
> option would help the problem, but that doesn't fix our problem.
> 
> Finally, after I tried to create a workaround, I've configured the limit
> zone to:
> limit_conn_zone "v$binary_remote_addr" zone=1234con:10m;
> 
> Suddenly everything magically started to work. The connections were not
> being dropped, the limit worked as expected and even more surprisingly the
> memory usage was not going up anymore. I've been tearing my hair out almost
> all day yesterday trying to figure this out. While I was very happy to see
> this resolved, I am now confused as to why nginx behaves in such a way.
> 
> I'm thinking this might likely be a bug, so I'm just wondering if anyone
> could explain why it is happening or has a similar problem.
>  
> Thank you!
> 

Have you checked error log?

  wbr, Valentin V. Bartenev