Nginx eats 100% cpu in ngx_event_pipe_write_to_downstream
Breno Silva
breno.silva at gmail.com
Mon May 13 12:17:23 UTC 2013
Hello Maxim,
Yes, my module cannot work with streams, we need full response body before
it starts working.
So what you suggest it something like this in my move_chain_to_brigado() ?
while (chain) {
buf = ngx_calloc_buf(pool);
if (buf == NULL) {
return NGX_ERROR;
}
len = chain->buf->end - chain->buf->start;
buf->start = ngx_palloc(pool, len);
ngx_memcpy(buf->start, chain->buf->start, len);
buf->pos = buf->start;
buf->end = buf->last = buf->pos + len;
e = ngx_buf_to_apr_bucket(buf, bb->p, bb->bucket_alloc);
if (e == NULL) {
return NGX_ERROR;
}
APR_BRIGADE_INSERT_TAIL(bb, e);
if (chain->buf->last_buf) {
e = apr_bucket_eos_create(bb->bucket_alloc);
APR_BRIGADE_INSERT_TAIL(bb, e);
chain->buf->last_buf = 0;
chain->buf->pos = chain->buf->last;
return NGX_OK;
}
chain->buf->pos = chain->buf->last;
chain = chain->next;
Thanks
Breno
On Mon, May 13, 2013 at 4:01 AM, Maxim Dounin <mdounin at mdounin.ru> wrote:
> Hello!
>
> On Sun, May 12, 2013 at 11:09:26AM -0300, Breno Silva wrote:
>
> > Hello,
> >
> > Yes removing he ngx_free_chain fix the cpu issue. Looks like ther modules
> > needs the chain.
> > I detected another issue.. sometimes when loading big response bodies
> (like
> > images.. +32K) it does not load it entirely or does it slowly.
> >
> > Do you have any idea ?
> > Thanks
>
> It looks like you don't release buffers you got in your filter,
> and this results buffers exhaustion (output_buffers,
> proxy_buffers) and transfer stall.
>
> If your code can't work with a data stream and needs full response
> before it can start it's work, you have to copy needed data from
> buffers and release buffers got (by setting b->pos = b->last, like
> if a buffer was actually sent).
>
> To test things you may try the following in config:
>
> output_buffers 1 1k;
>
> Then request a static file larger than 1k.
>
>
> >
> > Breno
> >
> >
> > On Sat, May 11, 2013 at 9:06 PM, Maxim Dounin <mdounin at mdounin.ru>
> wrote:
> >
> > > Hello!
> > >
> > > On Sat, May 11, 2013 at 08:09:59PM -0300, Breno Silva wrote:
> > >
> > > > Hello Maxim,
> > > >
> > > > We are sending chains data into a brigade/bucket structure (APR). It
> > > > sometimes works fine.. sometimes trigger the cpu issue
> > > >
> > > > If i call:
> > > > rc = move_chain_to_brigade(in, ctx->brigade, r->pool, 1); <---
> change
> > > > from 0 to 1
> > > > Then i will get only the first X bytes of the response body. This
> works
> > > > great... but i need the entire buffer :)
> > > >
> > > > Could you give us more details how we could patch this function to
> fix
> > > the
> > > > issue ?
> > >
> > > Remove the
> > >
> > > ngx_free_chain(pool, cl);
> > >
> > > call from the move_chain_to_brigade() function, it should fix you
> > > CPU hog issues.
> > >
> > >
> > > >
> > > > Thanks
> > > >
> > > >
> > > > On Sat, May 11, 2013 at 7:56 PM, Maxim Dounin <mdounin at mdounin.ru>
> > > wrote:
> > > >
> > > > > Hello!
> > > > >
> > > > > On Sat, May 11, 2013 at 04:23:00PM -0300, Breno Silva wrote:
> > > > >
> > > > > > Hello list,
> > > > > >
> > > > > > We are porting ModSecurity to NGINX. However we are seeing
> sometimes
> > > an
> > > > > > issue. Nginx eats 100% of cpu and when i use gdb i see:
> > > > > >
> > > > > > gdb -p 8645
> > > > > >
> > > > > > ngx_event_pipe_write_to_downstream (p=0x9bbc720, do_write=0) at
> > > > > > src/event/ngx_event_pipe.c:551
> > > > > >
> > > > > > 551 if (cl->buf->recycled) {
> > > > > > (gdb)
> > > > > >
> > > > > > Looks like it is happening when we call
> > > > > ngx_http_modsecurity_body_filter()
> > > > > > then go to this conditions;
> > > > > >
> > > > > > rc = move_chain_to_brigade(in, ctx->brigade, r->pool, 0);
> > > > > >
> > > > > > if (rc != NGX_OK) {
> > > > > >
> > > > > > r->buffered |= NGX_HTTP_SSI_BUFFERED;
> > > > > >
> > > > > > return rc;
> > > > > >
> > > > > > }
> > > > > >
> > > > > > move_chainto_brigade is defined as:
> > > > > >
> > > > > >
> > > > > > ngx_int_t
> > > > > >
> > > > > > move_chain_to_brigade(ngx_chain_t *chain, apr_bucket_brigade *bb,
> > > > > > ngx_pool_t *pool, ngx_int_t last_buf) {
> > > > > >
> > > > > > apr_bucket *e;
> > > > > >
> > > > > > ngx_chain_t *cl;
> > > > > >
> > > > > >
> > > > > > while (chain) {
> > > > > >
> > > > > > e = ngx_buf_to_apr_bucket(chain->buf, bb->p,
> > > bb->bucket_alloc);
> > > > > >
> > > > > > if (e == NULL) {
> > > > > >
> > > > > > return NGX_ERROR;
> > > > > >
> > > > > > }
> > > > > >
> > > > > >
> > > > > > APR_BRIGADE_INSERT_TAIL(bb, e);
> > > > > >
> > > > > > if (chain->buf->last_buf) {
> > > > > >
> > > > > > e = apr_bucket_eos_create(bb->bucket_alloc);
> > > > > >
> > > > > > APR_BRIGADE_INSERT_TAIL(bb, e);
> > > > > >
> > > > > > chain->buf->last_buf = 0;
> > > > > >
> > > > > > return NGX_OK;
> > > > > >
> > > > > > }
> > > > > >
> > > > > > cl = chain;
> > > > > >
> > > > > > chain = chain->next;
> > > > > >
> > > > > > ngx_free_chain(pool, cl);
> > > > > >
> > > > > > }
> > > > > >
> > > > > >
> > > > > > if (last_buf) {
> > > > > >
> > > > > > e = apr_bucket_eos_create(bb->bucket_alloc);
> > > > > >
> > > > > > APR_BRIGADE_INSERT_TAIL(bb, e);
> > > > > >
> > > > > > return NGX_OK;
> > > > > >
> > > > > > }
> > > > > >
> > > > > > return NGX_AGAIN;
> > > > > >
> > > > > > }
> > > > > > Let me know if you guys can help us understanding why sometimes
> we
> > > > > trigger
> > > > > > this issue
> > > > >
> > > > > It looks like your code modify chain links (ngx_chain_t
> > > > > structures) as got by your response body filter. This is not
> > > > > something filters are allowed to do, and results are undefined.
> > > > > If a filter needs to modify chain, it should allocate it's own
> > > > > chain links.
> > > > >
> > > > > In the code quoted you probably don't want to touch chain links at
> > > > > all.
> > > > >
> > > > > --
> > > > > Maxim Dounin
> > > > > http://nginx.org/en/donation.html
> > > > >
> > > > > _______________________________________________
> > > > > nginx-devel mailing list
> > > > > nginx-devel at nginx.org
> > > > > http://mailman.nginx.org/mailman/listinfo/nginx-devel
> > > > >
> > >
> > > > _______________________________________________
> > > > nginx-devel mailing list
> > > > nginx-devel at nginx.org
> > > > http://mailman.nginx.org/mailman/listinfo/nginx-devel
> > >
> > >
> > > --
> > > Maxim Dounin
> > > http://nginx.org/en/donation.html
> > >
> > > _______________________________________________
> > > nginx-devel mailing list
> > > nginx-devel at nginx.org
> > > http://mailman.nginx.org/mailman/listinfo/nginx-devel
> > >
>
> > _______________________________________________
> > nginx-devel mailing list
> > nginx-devel at nginx.org
> > http://mailman.nginx.org/mailman/listinfo/nginx-devel
>
>
> --
> Maxim Dounin
> http://nginx.org/en/donation.html
>
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20130513/8e1d5170/attachment.html>
More information about the nginx-devel
mailing list