Nginx eats 100% cpu in ngx_event_pipe_write_to_downstream

[Breno Silva]

Looks like we never see the chain->buf->last_buf, so never get the buffer
entirely ?

Breno


On Sun, May 12, 2013 at 11:09 AM, Breno Silva <breno.silva at gmail.com> wrote:

> Hello,
>
> Yes removing he ngx_free_chain fix the cpu issue. Looks like ther modules
> needs the chain.
> I detected another issue.. sometimes when loading big response bodies
> (like images.. +32K) it does not load it entirely or does it slowly.
>
> Do you have any idea ?
> Thanks
>
> Breno
>
>
> On Sat, May 11, 2013 at 9:06 PM, Maxim Dounin <mdounin at mdounin.ru> wrote:
>
>> Hello!
>>
>> On Sat, May 11, 2013 at 08:09:59PM -0300, Breno Silva wrote:
>>
>> > Hello Maxim,
>> >
>> > We are sending chains data into a brigade/bucket structure (APR). It
>> > sometimes works fine.. sometimes trigger the cpu issue
>> >
>> > If i call:
>> >  rc = move_chain_to_brigade(in, ctx->brigade, r->pool, 1);   <--- change
>> > from 0 to 1
>> > Then i will get only the first X bytes of the response body. This works
>> > great... but i need the entire buffer :)
>> >
>> > Could you give us more details how we could patch this function to fix
>> the
>> > issue ?
>>
>> Remove the
>>
>>         ngx_free_chain(pool, cl);
>>
>> call from the move_chain_to_brigade() function, it should fix you
>> CPU hog issues.
>>
>>
>> >
>> > Thanks
>> >
>> >
>> > On Sat, May 11, 2013 at 7:56 PM, Maxim Dounin <mdounin at mdounin.ru>
>> wrote:
>> >
>> > > Hello!
>> > >
>> > > On Sat, May 11, 2013 at 04:23:00PM -0300, Breno Silva wrote:
>> > >
>> > > > Hello list,
>> > > >
>> > > > We are porting ModSecurity to NGINX. However we are seeing
>> sometimes an
>> > > > issue. Nginx eats 100% of cpu and when i use gdb i see:
>> > > >
>> > > > gdb -p 8645
>> > > >
>> > > > ngx_event_pipe_write_to_downstream (p=0x9bbc720, do_write=0) at
>> > > > src/event/ngx_event_pipe.c:551
>> > > >
>> > > > 551 if (cl->buf->recycled) {
>> > > > (gdb)
>> > > >
>> > > > Looks like it is happening when we call
>> > > ngx_http_modsecurity_body_filter()
>> > > > then go to this conditions;
>> > > >
>> > > >     rc = move_chain_to_brigade(in, ctx->brigade, r->pool, 0);
>> > > >
>> > > >     if (rc != NGX_OK)  {
>> > > >
>> > > >         r->buffered |= NGX_HTTP_SSI_BUFFERED;
>> > > >
>> > > >         return rc;
>> > > >
>> > > >      }
>> > > >
>> > > > move_chainto_brigade is defined as:
>> > > >
>> > > >
>> > > > ngx_int_t
>> > > >
>> > > > move_chain_to_brigade(ngx_chain_t *chain, apr_bucket_brigade *bb,
>> > > > ngx_pool_t *pool, ngx_int_t last_buf) {
>> > > >
>> > > >     apr_bucket         *e;
>> > > >
>> > > >     ngx_chain_t        *cl;
>> > > >
>> > > >
>> > > >     while (chain) {
>> > > >
>> > > >         e = ngx_buf_to_apr_bucket(chain->buf, bb->p,
>> bb->bucket_alloc);
>> > > >
>> > > >         if (e == NULL) {
>> > > >
>> > > >             return NGX_ERROR;
>> > > >
>> > > >         }
>> > > >
>> > > >
>> > > >         APR_BRIGADE_INSERT_TAIL(bb, e);
>> > > >
>> > > >         if (chain->buf->last_buf) {
>> > > >
>> > > >             e = apr_bucket_eos_create(bb->bucket_alloc);
>> > > >
>> > > >             APR_BRIGADE_INSERT_TAIL(bb, e);
>> > > >
>> > > >             chain->buf->last_buf = 0;
>> > > >
>> > > >             return NGX_OK;
>> > > >
>> > > >         }
>> > > >
>> > > >         cl = chain;
>> > > >
>> > > >         chain = chain->next;
>> > > >
>> > > >         ngx_free_chain(pool, cl);
>> > > >
>> > > >     }
>> > > >
>> > > >
>> > > >     if (last_buf) {
>> > > >
>> > > >         e = apr_bucket_eos_create(bb->bucket_alloc);
>> > > >
>> > > >         APR_BRIGADE_INSERT_TAIL(bb, e);
>> > > >
>> > > >         return NGX_OK;
>> > > >
>> > > >     }
>> > > >
>> > > >     return NGX_AGAIN;
>> > > >
>> > > > }
>> > > > Let me know if you guys can help us understanding why sometimes we
>> > > trigger
>> > > > this issue
>> > >
>> > > It looks like your code modify chain links (ngx_chain_t
>> > > structures) as got by your response body filter.  This is not
>> > > something filters are allowed to do, and results are undefined.
>> > > If a filter needs to modify chain, it should allocate it's own
>> > > chain links.
>> > >
>> > > In the code quoted you probably don't want to touch chain links at
>> > > all.
>> > >
>> > > --
>> > > Maxim Dounin
>> > > http://nginx.org/en/donation.html
>> > >
>> > > _______________________________________________
>> > > nginx-devel mailing list
>> > > nginx-devel at nginx.org
>> > > http://mailman.nginx.org/mailman/listinfo/nginx-devel
>> > >
>>
>> > _______________________________________________
>> > nginx-devel mailing list
>> > nginx-devel at nginx.org
>> > http://mailman.nginx.org/mailman/listinfo/nginx-devel
>>
>>
>> --
>> Maxim Dounin
>> http://nginx.org/en/donation.html
>>
>> _______________________________________________
>> nginx-devel mailing list
>> nginx-devel at nginx.org
>> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20130512/cf3c300f/attachment.html>