Memory use flares up sharply, how to troubleshoot?

[gthb]

Hi,

Several times recently, we have seen our production nginx memory usage flare
up a hundred-fold, from its normal ~42 MB to 3-4 GB, for 20 minutes to an
hour or so, and then recover. There is not a spike in number of connections,
just memory use, so whatever causes this, it does not seem to be an increase
in concurrency.

The obvious thing to suspect for this is our app's newest change, which
involves streaming responses proxied from an upstream (via uwsgi_pass);
these responses can get very large and run for many minutes, pumping
hundreds of megabytes each. But I expect nginx memory use for these requests
to be bounded by uwsgi_buffers (shouldn't it be?) -- and indeed I cannot
reproduce the problem by making such requests, even copying the exact ones
that are being made when the memory spike occurs. In my tests, the responses
get buffered as they should be, and delivered normally, without memory
growth.

So, what is a good way to investigate what causes all this memory to be
suddenly allocated? Is there a way of introspecting/logging nginx memory
allocation edge cases like this? (Is there documentation on this which I
didn't find?)

During the memory spike (and before and after), connections are around 270,
of which around 20-25 are writing and the rest are waiting.

Our nginx config has:

worker_processes 2;
events {
    worker_connections  1024;
}
http {
    uwsgi_buffers 64 8k;
    proxy_buffers 64 8k;
}

All other *_buffers settings are at their defaults, on a 64-bit machine.

Accounting for the above buffers, other buffer defaults, and some
uwsgi_cache and proxy_cache zones, and estimating on the high side (e.g. all
buffer budgets fully used for each active connection) for 270 connections,
here is my rough sketch of the memory uses I am aware of, probably with some
misunderstandings along the way:

22 * keys_zone=10m for uwsgi_cache_path + proxy_cache_path = 220   MB
uwsgi_buffers: 64 * 8kB * 270                              = 135   MB
proxy_buffers: 64 * 8kB * 270                              = 135   MB
gzip_buffers:  16 * 8kB * 270                              =  34   MB
output_buffers: 1 * 32kB * 270                             =   8.6 MB
large_client_header_buffers: 4 * 8kB * 270                 =   8.6 MB
ssl_buffer_size: 1 * 16k * 270                             =   4.3 MB
client_body_buffer_size: 1 * 16kB * 270                    =   4.3 MB
client_header_buffer_size: 1 * 1kB * 270                   =   0.3 MB 
access_log buffers: 24 * 8kB + 1 * 32k                     =   0.2 MB

This comes out to 546 MB tops ... what big memory use cases are missing in
this picture?

(And are these buffer size upper-bound estimates significantly incorrect?)

This is in Nginx 1.6.0 on Linux 3.2.57 64-bit ... specifically:

$ nginx -V
nginx version: nginx/1.6.0
built by gcc 4.7.2 (Debian 4.7.2-5) 
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx
--conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log
--http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid
--lock-path=/var/run/nginx.lock
--http-client-body-temp-path=/var/cache/nginx/client_temp
--http-proxy-temp-path=/var/cache/nginx/proxy_temp
--http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp
--http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp
--http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx
--with-http_ssl_module --with-http_realip_module --with-http_addition_module
--with-http_sub_module --with-http_dav_module --with-http_flv_module
--with-http_mp4_module --with-http_gunzip_module
--with-http_gzip_static_module --with-http_random_index_module
--with-http_secure_link_module --with-http_stub_status_module
--with-http_auth_request_module --with-mail --with-mail_ssl_module
--with-file-aio --with-http_spdy_module --with-cc-opt='-g -O2
-fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security
-Wp,-D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-z,relro -Wl,--as-needed'
--with-ipv6

$ uname -a
Linux ren2 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3+deb7u2 x86_64 GNU/Linux

Thanks, best regards,

Gulli

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,251964,251964#msg-251964