High memory consumption when proxying to a Comet server

[Maxim Dounin]

Hello!

On Sun, Apr 11, 2010 at 10:08:26PM -0300, Rogério Schneider wrote:

> Maxim, thanks.
> 
> I have tried to reduce memory usage by reducing the default values for:
>   proxy_buffers
> from "8 4k" to "2 4k".
> 
> With this one would expect the memory usage to be 4 times lower, but
> that was not what I saw.
> I saw the memory usage stay in the same growth step.

No, changing 8 buffers to 2 buffers doesn't mean memory usage will 
be 4 times less.  1) this isn't the only allocations made, and 2) 
not all requests allocate all proxy buffers, only ones with big 
responses and slow clients.  But see below.

> May I mention that for the fact I am using Comet, I turned proxy buffering off?
>   proxy_buffering         off;
> I think this is important to note and it can explain why tuning the
> proxy buffers down did not lower the memory consumption.

With proxy_buffering off; setting proxy_buffers indeed doesn't 
matter at all.

Next candidates is gzip_buffers (only matters if you are using 
gzip, up to 32 * 4k by default; and keep in mind that about 200k 
or so will be allocated per request anyway with gzip enabled).

Remaining ones are output_buffers and fastcgi_buffers, these 
shouldn't matter in your case anyway.

> Could I debug Nginx procs to see where the memory is being allocated
> and for what purpose?
> How could I debug it? Do you have some guideline?

You may switch on debug log, it will be possible to trace big 
allocations there.  See here:

http://nginx.org/en/docs/debugging_log.html

Maxim Dounin

p.s. Please do not top-post.  Thank you.