Possible memory leak?

[wkbrad]

Hi all,

I just wanted to share the details of what I've found about this issue. 
Also thanks to Maxim Dounin and Reinis Rozitis who gave some really great
answers!

The more I look into this the more I'm convinced this is an issue with Nginx
itself.  I've tested this with 3 different builds now and all have the exact
same issue.

The first 2 types of servers I tested were both running Nginx 1.15.8 on
Centos 7 ( with 1 of them being on 6 ).  I tested about 10 of our over 100
servers.  This time I tested in a default install of Debian 9 with Nginix
version 1.10.3 and the issue exists there too.  I just wanted to test on
something completely different.

For the test, I created 50k very simple vhosts which used about 1G of RAM. 
Here is the ps_mem output.
 94.3 MiB +   1.0 GiB =   1.1 GiB	nginx (3)

After a normal reload it then uses 2x the ram:
186.3 MiB +   1.9 GiB =   2.1 GiB	nginx (3)

And if I reload it again it briefly jumps up to about 4G during the reload
and then goes back down to 2G.

If I instead use the "upgrade" option.  In the case of Debian, service nginx
upgrade, then it reloads gracefully and goes back to using 1G again.
100.8 MiB +   1.0 GiB =   1.1 GiB	nginx (3)

The difference between the "reload" and "upgrade" process is basically only
that reload sends a HUP signal to Nginx and upgrade sends a USR2 and then
QUIT signal.  What happens with all of those signals is entirely up to
Nginx.  It could even ignore them if chose too.

Additionally, I ran the same test with Apache.  Not because I want to
compare Nginx to Apache, they are different for a reason.  I just wanted to
test if this was a system issue.  So I did the same thing on Debian 9,
installed Apache and created 50k simple vhosts.  It used about 800M of ram
and reloading did not cause that to increase at all.

All of that leads me to these questions.

Why would anyone want to use the normal reload process to reload the Nginx
configuration?
Shouldn't we always be using the upgrade process instead?
Are there any downsides to doing that?
Has anyone else noticed these issues and have you found another fix?

Look forward to hearing back and thanks in advance!

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,283216,283309#msg-283309