Possible memory leak?

[Maxim Dounin]

Hello!

On Thu, Mar 21, 2019 at 11:29:04AM -0400, wkbrad wrote:

> Thanks again Maxim!  You're really providing some valuable insights for me.
> 
> > This is because mmap()-based individual allocations imply additional
> > overhead. Using mmapAnd this is are costly.
> 
> That's what I figured might be going on.  I assume there are also some
> negative impacts on performance with accessing memory that is allocated via
> mmap.  Is that right?

Accessing the memory shouldn't depend on the way how it is 
allocated.  But there may be negative effects on the memory 
allocator performance due to the number of individual mmap() 
allocations it has to maintain, and there may be negative effects 
on the whole system performance with such a big number of mmap()s.

> > On FreeBSD you'll see that _virtual_ memory usage grows on reload,
> > but the actual memory is returned to the system and can be used by
> > other processes. 
> 
> Actually, I see an increase in virtual and resident.  Here are my tests from
> FreeBSD.
> --------------------------------------------------------------------------------
> service nginx restart; sleep 10; echo; ps aux|grep -v grep|egrep
> 'RSS|nginx'
> 
> USER  PID  %CPU %MEM    VSZ    RSS TT  STAT STARTED     TIME COMMAND
> root  923   0.0  5.4 121840 110368  -  Ss   08:10    0:00.00 nginx: master
> process /usr/local/
> www   924   0.0  5.4 121840 110372  -  S    08:10    0:00.01 nginx: worker
> process (nginx)
> --------------------------------------------------------------------------------
> service nginx reload; sleep 10; echo; ps aux|grep -v grep|egrep 'RSS|nginx'
> 
> USER  PID  %CPU %MEM    VSZ    RSS TT  STAT STARTED     TIME COMMAND
> root  923   0.0 10.2 219828 209388  -  Ss   08:10    0:00.44 nginx: master
> process /usr/local/
> www   940   0.0 10.2 219828 209384  -  S    08:10    0:00.01 nginx: worker
> process (nginx)
> --------------------------------------------------------------------------------
> 
> Those tests look exactly like the Linux tests and it's definitely not
> released back to the system.

The problem is that "ps" doesn't know if memory counted in RSS 
can be reclaimed by the system or not.  And reclaiming the memory 
might not happen till the system needs this memory for something 
else.  You may have a better picture by looking at memory stats 
reported by top:

$ ps aux | grep nginx | grep -v grep; top | grep Mem
mdounin 55612   0.0 18.0 55192 41868  0  S+   18:48       0:00.29 nginx: master
mdounin 55613   0.0 18.0 55192 41868  0  S+   18:48       0:00.00 nginx: worker
Mem: 44M Active, 8208K Inact, 63M Wired, 29M Buf, 108M Free
$ kill -HUP 55612
$ ps aux | grep nginx | grep -v grep; top | grep Mem
mdounin 55612   1.0 32.6 87960 75808  0  S+   18:48       0:00.68 nginx: master
mdounin 55619   1.0 32.6 87960 75808  0  S+   18:49       0:00.00 nginx: worker
Mem: 52M Active, 34M Inact, 63M Wired, 29M Buf, 74M Free

Note that most of the additional memory is now counted as "Inact", 
i.e., it can be reclaimed by the system at any time.

The other factor is that jemalloc maintains various allocation 
caches, and some of the allocations are preserved there and not 
returned to the system.

[...]

> You may be asking why I'm even looking into this and I've had a lot of push
> back and questions as to why I'm doing this.  My point is this.  Even if
> Nginx on my server is only using 50M of ram, wouldn't it better for it to
> use only 25M when that's all it needs?  That's clearly a problem in my mind.
>  Can you address why that is not a problem?

There are two things to keep in mind here:

- Every task needs memory.  Depending on the amount of memory used 
  and how common the task is, it may worth optimizing or not.  In 
  most setups, nginx uses far less memory for the configuration 
  data than for other things.  And optimizing configuration 
  storage hardly worth the effort, as it doesn't matter if your 
  configuration uses 25M or 50M compared to a 2G shared memory 
  zone used for cache keys, or several gigabytes of proxy buffers.

- The particular effect of "doubling" memory usage on reload you 
  are trying to understand in this thread is not really about 
  nginx, but rather about how your system allocator works.  You can 
  tune or improve your system allocator to handle this better, but 
  see above about if it worth the effort.

If you still think this is a problem for you, and you want to save 
25M of memory, you were already suggested several ways to improve 
things - including re-writing your configuration, re-compiling 
nginx without unused modules, or tuning/replacing your system 
allocator.

-- 
Maxim Dounin
http://mdounin.ru/