100% CPU use in ngx_http_finalize_connection

[Richard Stanway]

Hi Maxim,
Thanks for the quick patch! I've applied it to our server and will
monitor the results. Usually the problem starts to occur within 1-2
hours of a restart, so I'll post again later today with an update.

On Fri, Mar 24, 2017 at 2:03 PM, Maxim Dounin <mdounin at mdounin.ru> wrote:
> Hello!
>
> On Fri, Mar 24, 2017 at 01:31:35PM +0100, Richard Stanway wrote:
>
>> Hello,
>> I recently moved our site to a new server running Linux 4.9, Debian
>> 8.7 64 bit with nginx 1.11.11 from the nginx repository. Our config is
>> straightforward - epoll, a few proxy backends and a few fastcgi
>> backends, a handful of vhosts, some with HTTP2, geoip module loaded.
>> No AIO, no threads, no timer_resolution.
>>
>> After some time, nginx worker processes are getting stuck at 100% CPU
>> use in what seems to be ngx_http_finalize_connection. New requests
>> hitting the worker are completely stalled. Eventually all nginx
>> workers will become stuck and the sites become unreachable.
>>
>> I'm running older versions of nginx on the same versions of Debian and
>> Linux at other sites without a problem, but the server giving me
>> problems also receives a much larger amount of traffic than the
>> others. Due to the traffic, the debug log gets incredibly large which
>> makes it difficult to isolate the error. I've posted a 1 second
>> excerpt of the core debug log at http://pastebin.com/hqzGzjTV during
>> the time that some of the workers were at 100%, however I'm not sure
>> this contains enough information. I'll look into enabling HTTP level
>> logging if necessary.
>>
>> Has anyone experienced anything similar to this or have any ideas
>> where to start looking to debug this?
>>
>> Thanks.
>>
>> nginx version: nginx/1.11.11
>> built by gcc 4.9.2 (Debian 4.9.2-10)
>
> [...]
>
>> #0  0x000055d533ab87e8 in ngx_pfree (pool=0x55d536202fe0,
>> p=0x55d5361636c0) at src/core/ngx_palloc.c:282
>> #1  0x000055d533af54d9 in ngx_http_set_keepalive (r=<optimized out>)
>> at src/http/ngx_http_request.c:3000
>> #2  ngx_http_finalize_connection (r=<optimized out>) at
>> src/http/ngx_http_request.c:2556
>> #3  0x000055d533af0d8b in ngx_http_core_content_phase
>> (r=0x55d536136f10, ph=0x55d537cbf210) at
>> src/http/ngx_http_core_module.c:1391
>
> I think I see the problem.
> Please try the following patch:
>
> diff --git a/src/http/ngx_http_request.c b/src/http/ngx_http_request.c
> --- a/src/http/ngx_http_request.c
> +++ b/src/http/ngx_http_request.c
> @@ -2904,6 +2904,7 @@ ngx_http_set_keepalive(ngx_http_request_
>              }
>
>              cl->buf = b;
> +            cl->next = NULL;
>
>              hc->busy = cl;
>              hc->nbusy = 1;
>
> --
> Maxim Dounin
> http://nginx.org/
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx