Memory use flares up sharply, how to troubleshoot?
mdounin at mdounin.ru
Tue Jul 22 11:53:11 UTC 2014
On Mon, Jul 21, 2014 at 05:44:45PM -0400, gthb wrote:
> I finally reproduced this, with debug logging enabled --- I found the
> problematic request in the error log preceding the kill signal, saying it
> was being buffered to a temporary file:
> 2014/07/21 11:39:39 [warn] 21182#0: *32332838 an upstream response is
> buffered to a temporary file /var/cache/nginx/uwsgi_temp/9/90/0000186909
> while reading upstream, client: x.x.x.x, server: foo.com, request: "GET
> /api/nasty/troublemaker.csv?el=xyzzy!a:b&dates_as_dates=1 HTTP/1.1",
> upstream: "uwsgi://184.108.40.206:3003", host: "foo.com"
> 2014/07/21 11:41:18 [alert] 16885#0: worker process 21182 exited on
> signal 9
> and retrying that request reproduces the problem, nginx growing in size
> without bound. (The request never made it to the access log because of the
> OOM kill, which is why my previous testing didn't reproduce it)
> These extra lines *never* appear in the healthy requests, so I imagine they
> point to the problem (but I am not at all familiar with Nginx debug output);
> in particular those "write new buf" lines look relevant; they are output
> right after ngx_alloc_chain_link is called.
The lines in question are just sending the response to the client
via the response body filter chain.
> All the possibly relevant Nginx config:
I don't see anything obviously wrong in the config, but again - I
strongly recommend you to post _full_ configuration. Or, better
yet, if you are able to reproduce the problem with a reduced
config, post it instead (and corresponding debug log).
> Does this narrow down the problem? Can I provide anything further?
No. Please show the debug log.
More information about the nginx