Possible memory leak?

[wkbrad]

Hey Maxim,

Thanks so much for the very thoughtful and detailed response!  Sorry for the
delay in getting back to you.  I wanted to test all of this and then I got
busy at work and haven't been able to get back to it.

I've run a lot of tests on this and I'm seeing some success.  :)

The first test I ran was in FreeBSD just because I was curious.  Lol.  But I
actually saw the exact same problem on it.  I can send you some tests if you
like but they look the same as the others do.

Now on to the partial success!  But first, I'll include the results without
any modifications to malloc.
--------------------------------------------------------------------------------
[root at localhost ~]# systemctl restart nginx; sleep 10; ps_mem |egrep
'RAM|nginx'
 Private  +   Shared  =  RAM used	Program 
 13.8 MiB + 750.7 MiB = 764.5 MiB	nginx (3)
[root at localhost ~]# systemctl reload nginx; sleep 10; ps_mem |egrep
'RAM|nginx'
 Private  +   Shared  =  RAM used	Program 
 27.2 MiB +   1.4 GiB =   1.5 GiB	nginx (3)
--------------------------------------------------------------------------------

Instead of passing the environment variables in the command line I'm going
to run the tests by modifying the systemd script to use the following.
--------------------------------------------------------------------------------
Environment=MALLOC_MMAP_THRESHOLD_=16384
Environment=MALLOC_MMAP_MAX_=524288
--------------------------------------------------------------------------------

Now for the test:
--------------------------------------------------------------------------------
[root at localhost ~]# systemctl restart nginx; sleep 10; ps_mem |egrep
'RAM|nginx'
 Private  +   Shared  =  RAM used	Program 
  7.6 MiB + 905.8 MiB = 913.4 MiB	nginx (3)
[root at localhost ~]# systemctl reload nginx; sleep 10; ps_mem |egrep
'RAM|nginx'
 Private  +   Shared  =  RAM used	Program 
  9.7 MiB + 905.4 MiB = 915.1 MiB	nginx (3)
--------------------------------------------------------------------------------

Notice it starts up using more memory than before but it now does not double
it's ram usage after a reload.  I can deal with that but I'm still curious
if you know why it's now using more ram.  Any thoughts on that?

But when I bring this setup to one of my live servers the effect isn't quite
the same.  Here is a test from it with the same settings in systemd.
--------------------------------------------------------------------------------
[root at live ~]# systemctl restart nginx; sleep 60; ps_mem |egrep 'RAM|nginx'
 Private  +   Shared  =  RAM used	Program 
 21.6 MiB +   1.4 GiB =   1.5 GiB	nginx (3)
[root at live ~]# systemctl reload nginx; sleep 60; ps_mem |egrep 'RAM|nginx'
 Private  +   Shared  =  RAM used	Program 
 21.9 MiB +   2.3 GiB =   2.4 GiB	nginx (3)
--------------------------------------------------------------------------------

So that's definitely better but still not like the test environment.  I'm
sure part of the difference is that this server has real traffic and is
somewhat busy.  And it's using caching and probably has different buffer
settings and such.  So that may explain why there is an increase.  But it's
still very perplexing.

What are your thoughts on all of that?

And thank you again for all of the help!

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,283216,283448#msg-283448