100% CPU use in ngx_http_finalize_connection
r1ch+nginx at teamliquid.net
Fri Mar 24 13:11:12 UTC 2017
Thanks for the quick patch! I've applied it to our server and will
monitor the results. Usually the problem starts to occur within 1-2
hours of a restart, so I'll post again later today with an update.
On Fri, Mar 24, 2017 at 2:03 PM, Maxim Dounin <mdounin at mdounin.ru> wrote:
> On Fri, Mar 24, 2017 at 01:31:35PM +0100, Richard Stanway wrote:
>> I recently moved our site to a new server running Linux 4.9, Debian
>> 8.7 64 bit with nginx 1.11.11 from the nginx repository. Our config is
>> straightforward - epoll, a few proxy backends and a few fastcgi
>> backends, a handful of vhosts, some with HTTP2, geoip module loaded.
>> No AIO, no threads, no timer_resolution.
>> After some time, nginx worker processes are getting stuck at 100% CPU
>> use in what seems to be ngx_http_finalize_connection. New requests
>> hitting the worker are completely stalled. Eventually all nginx
>> workers will become stuck and the sites become unreachable.
>> I'm running older versions of nginx on the same versions of Debian and
>> Linux at other sites without a problem, but the server giving me
>> problems also receives a much larger amount of traffic than the
>> others. Due to the traffic, the debug log gets incredibly large which
>> makes it difficult to isolate the error. I've posted a 1 second
>> excerpt of the core debug log at http://pastebin.com/hqzGzjTV during
>> the time that some of the workers were at 100%, however I'm not sure
>> this contains enough information. I'll look into enabling HTTP level
>> logging if necessary.
>> Has anyone experienced anything similar to this or have any ideas
>> where to start looking to debug this?
>> nginx version: nginx/1.11.11
>> built by gcc 4.9.2 (Debian 4.9.2-10)
>> #0 0x000055d533ab87e8 in ngx_pfree (pool=0x55d536202fe0,
>> p=0x55d5361636c0) at src/core/ngx_palloc.c:282
>> #1 0x000055d533af54d9 in ngx_http_set_keepalive (r=<optimized out>)
>> at src/http/ngx_http_request.c:3000
>> #2 ngx_http_finalize_connection (r=<optimized out>) at
>> #3 0x000055d533af0d8b in ngx_http_core_content_phase
>> (r=0x55d536136f10, ph=0x55d537cbf210) at
> I think I see the problem.
> Please try the following patch:
> diff --git a/src/http/ngx_http_request.c b/src/http/ngx_http_request.c
> --- a/src/http/ngx_http_request.c
> +++ b/src/http/ngx_http_request.c
> @@ -2904,6 +2904,7 @@ ngx_http_set_keepalive(ngx_http_request_
> cl->buf = b;
> + cl->next = NULL;
> hc->busy = cl;
> hc->nbusy = 1;
> Maxim Dounin
> nginx mailing list
> nginx at nginx.org
More information about the nginx