High memory consumption when proxying to a Comet server

[Rogério Schneider]

> No, changing 8 buffers to 2 buffers doesn't mean memory usage will
> be 4 times less.  1) this isn't the only allocations made, and 2)
> not all requests allocate all proxy buffers, only ones with big
> responses and slow clients.  But see below.

Great, I didn't know that.

> Next candidates is gzip_buffers (only matters if you are using
> gzip, up to 32 * 4k by default; and keep in mind that about 200k
> or so will be allocated per request anyway with gzip enabled).

I am not using gzip. Nor the request come with Accept-Encoding,
neither I am turning "gzip on;", so I think this is not the case too.

Another thing to notice is about the "per request" usage. I have
only one request per client, but this request lasts for many minutes,
still, when monitoring Nginx memory usage, it increases every minute,
even with no new requests made from the client side. In this period
of monitoring only the Comet server was writing data, up to the
Nginx proxy.

> Remaining ones are output_buffers and fastcgi_buffers, these
> shouldn't matter in your case anyway.

I could not find where are the docs about output_buffers. I am
interested on reading about it.

Also, I am using this version of Nginx:

nginx version: nginx/0.7.65
built by gcc 4.3.0 20080428 (Red Hat 4.3.0-8) (GCC)
configure arguments: --prefix=/opt/tr/nginx --without-select_module
--without-poll_module --without-http_charset_module
--without-http_ssi_module --without-http_auth_basic_module
--without-http_autoindex_module --without-http_geo_module
--without-http_map_module --without-http_referer_module
--without-http_rewrite_module --without-http_fastcgi_module
--without-http_memcached_module --without-http_limit_zone_module
--without-http_limit_req_module --without-http_empty_gif_module
--without-http_browser_module --without-http_upstream_ip_hash_module
--without-mail_pop3_module --without-mail_imap_module
--without-mail_smtp_module --with-http_stub_status_module --with-debug

> You may switch on debug log, it will be possible to trace big
> allocations there.  See here:
>
> http://nginx.org/en/docs/debugging_log.html

I posted this log of some tests, and if you can please take a
look at it, I would appreciate.

It was separated in sections, the start, the small msg, the big
msg, the close.
Then another run, the small msg after many big msgs, and the
close (with lots of "strange" free()s):

http://pastebin.com/Fggn4Ui7

Regards,
-- 
Rogério Schneider
http://stockrt.github.com