Memory use flares up sharply, how to troubleshoot?

[Maxim Dounin]

Hello!

On Tue, Jul 22, 2014 at 10:51:43AM -0400, gthb wrote:

> Hi,
> 
> here's a minimal configuration where I can reproduce this:
> 
> error_log  debug.log debug;
> 
> events {
>     worker_connections  1024;
> }
> 
> http {
>     uwsgi_buffers 64 8k;
> 
>     upstream nginx-test.uwsgi {
>         server 10.0.0.7:13003;
>         least_conn;
>     }
> 
>     server {
>         listen 8080;
>         server_name  nginx-test.com;
> 
>         location /api/ {
>             include uwsgi_params;
>             uwsgi_pass nginx-test.uwsgi;
>         }
>     }
> }
> 
> Here's a debug log covering server start, a single request that exhibits the
> problem, and server shutdown:
> 
> http://filebin.ca/1UClE4zzhfZe/debug.log.gz
> 
> Everything goes OK for a while, just a few stray mallocs, and then maybe
> half a minute into the request (the time varies), after maybe 20-25MB have
> been transferred, the flood of mallocs starts:

Ok, I see what goes on here.  It is a combination of multiple 
factors:

- there are more than 16 buffers and hence stack-based buffer for 
  iovs isn't big enough and nginx have to allocate memory in 
  ngx_readv_chain();

- your backend app returns data in very small chunks, thus there 
  are many ngx_readv_chain() calls;

- response is big, and hence the effect of the above is 
  noticeable.

Trivial workaround is to use "uwsgi_buffers 8 64k" instead.
Or you may try the following patch:

# HG changeset patch
# User Maxim Dounin <mdounin at mdounin.ru>
# Date 1406044801 -14400
#      Tue Jul 22 20:00:01 2014 +0400
# Node ID 129a91bfb0565ab21a0f399688be148fe5e76a1e
# Parent  0896d5cb6b3d9ba7d229863ac65cd1559b2c439a
Avoid memory allocations in ngx_readv_chain().

diff --git a/src/os/unix/ngx_readv_chain.c b/src/os/unix/ngx_readv_chain.c
--- a/src/os/unix/ngx_readv_chain.c
+++ b/src/os/unix/ngx_readv_chain.c
@@ -10,7 +10,11 @@
 #include <ngx_event.h>
 
 
-#define NGX_IOVS  16
+#if (IOV_MAX > 64)
+#define NGX_IOVS  64
+#else
+#define NGX_IOVS  IOV_MAX
+#endif
 
 
 #if (NGX_HAVE_KQUEUE)
@@ -71,7 +75,7 @@ ngx_readv_chain(ngx_connection_t *c, ngx
             iov->iov_len += chain->buf->end - chain->buf->last;
 
         } else {
-            if (vec.nelts >= IOV_MAX) {
+            if (vec.nelts >= NGX_IOVS) {
                 break;
             }
 
@@ -200,7 +204,7 @@ ngx_readv_chain(ngx_connection_t *c, ngx
             iov->iov_len += chain->buf->end - chain->buf->last;
 
         } else {
-            if (vec.nelts >= IOV_MAX) {
+            if (vec.nelts >= NGX_IOVS) {
                 break;
             }
 


-- 
Maxim Dounin
http://nginx.org/