Memory use flares up sharply, how to troubleshoot?

[gthb]

> How do you track "nginx memory"?

What I was tracking was memory use per process name as reported by New Relic
nrsysmond, which I'm pretty sure is RSS from ps output, summed over all
nginx processes.

> From what you describe I suspect that disk buffering occurs (see
> http://nginx.org/r/uwsgi_max_temp_file_size), and the number you
> are looking at includes the size of files on disk.

I wish : ) because that's what I want to happen for these large responses.
But that's definitely not it, because we see a spike of swap when this
occurs, with most other processes on the machine being paged out ... and in
the worst spikes swap has filled up and an OOM kill has occurred, which
conveniently records in syslog the RSS for an nginx process being killed:

Jul 21 03:54:16 ren2 kernel: [3929562.712779] uwsgi invoked oom-killer:
gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
...
Jul 21 03:54:16 ren2 kernel: [3929562.737340] Out of memory: Kill process
5248 (nginx) score 328 or sacrifice child
Jul 21 03:54:16 ren2 kernel: [3929562.737352] Killed process 5248 (nginx)
total-vm:3662860kB, anon-rss:3383776kB, file-rss:16kB

So that catches Nginx holding a 3.2GB resident set, matching what New Relic
says about the same time.

> The debuging log includes full information about all memory
> allocations, see http://nginx.org/en/docs/debugging_log.html.

Thank you. I haven't been able to reproduce this outside of production (or
even in production) so I might have to leave debug logging enabled in
production and hope to catch this next time it happens. Am I right to assume
that enabling debug is going to weigh quite heavily on production usage, and
eat up disk very fast? (Traffic peaks at around 100 req/sec and 2 MB/sec.)

Cheers,

Gulli

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,251964,251967#msg-251967