limit_conn is dropping valid connections and causing memory leaks on nginx reload

[Dejan Grofelnik Pelzel]

Hello,

We are running the nginx 1.13.5  with HTTP/2 in a proxy_pass proxy_cache
configuration with clients having relatively long open connections. Our
system does automatic reloads for any new configuration and we recently
introduced a limit_conn to some of the config files. After that, I've
started noticing a rapid drop in connections and outgoing network every-time
the system would perform a configuration reload. Even stranger, on every
reload the memory usage would go up for about 1-2GB until ultimately
everything crashed if the reloads were too frequent. The memory usage did go
down after old workers were released, but that could take up to 30 minutes,
while the configuration could get reloaded up to twice per minute.

We used the following configuration as recommended by pretty much any
example:
limit_conn_zone $binary_remote_addr zone=1234con:10m;
limit_conn zone1234con 10;

I was able to verify the connection drop by doing a simple ab test, for
example, I would run ab -c 100 -n -k 1000 https://127.0.0.1/file.bin
990 of the connections went through, however, 10 would still be active.
Immediately after the reload, those would get dropped as well. Adding -r
option would help the problem, but that doesn't fix our problem.

Finally, after I tried to create a workaround, I've configured the limit
zone to:
limit_conn_zone "v$binary_remote_addr" zone=1234con:10m;

Suddenly everything magically started to work. The connections were not
being dropped, the limit worked as expected and even more surprisingly the
memory usage was not going up anymore. I've been tearing my hair out almost
all day yesterday trying to figure this out. While I was very happy to see
this resolved, I am now confused as to why nginx behaves in such a way.

I'm thinking this might likely be a bug, so I'm just wondering if anyone
could explain why it is happening or has a similar problem.
 
Thank you!

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,276633,276633#msg-276633