Nginx eats 100% cpu in ngx_event_pipe_write_to_downstream
Maxim Dounin
mdounin at mdounin.ru
Mon May 13 13:05:50 UTC 2013
Hello!
On Mon, May 13, 2013 at 09:17:23AM -0300, Breno Silva wrote:
> Hello Maxim,
>
> Yes, my module cannot work with streams, we need full response body before
> it starts working.
> So what you suggest it something like this in my move_chain_to_brigado() ?
>
> while (chain) {
>
> buf = ngx_calloc_buf(pool);
> if (buf == NULL) {
> return NGX_ERROR;
> }
>
> len = chain->buf->end - chain->buf->start;
> buf->start = ngx_palloc(pool, len);
> ngx_memcpy(buf->start, chain->buf->start, len);
>
> buf->pos = buf->start;
> buf->end = buf->last = buf->pos + len;
>
> e = ngx_buf_to_apr_bucket(buf, bb->p, bb->bucket_alloc);
> if (e == NULL) {
> return NGX_ERROR;
> }
>
> APR_BRIGADE_INSERT_TAIL(bb, e);
> if (chain->buf->last_buf) {
> e = apr_bucket_eos_create(bb->bucket_alloc);
> APR_BRIGADE_INSERT_TAIL(bb, e);
> chain->buf->last_buf = 0;
Just a side note: resetting last_buf looks useless.
> chain->buf->pos = chain->buf->last;
> return NGX_OK;
> }
>
> chain->buf->pos = chain->buf->last;
> chain = chain->next;
This looks suboptimal, but should work.
>
> Thanks
>
> Breno
>
>
> On Mon, May 13, 2013 at 4:01 AM, Maxim Dounin <mdounin at mdounin.ru> wrote:
>
> > Hello!
> >
> > On Sun, May 12, 2013 at 11:09:26AM -0300, Breno Silva wrote:
> >
> > > Hello,
> > >
> > > Yes removing he ngx_free_chain fix the cpu issue. Looks like ther modules
> > > needs the chain.
> > > I detected another issue.. sometimes when loading big response bodies
> > (like
> > > images.. +32K) it does not load it entirely or does it slowly.
> > >
> > > Do you have any idea ?
> > > Thanks
> >
> > It looks like you don't release buffers you got in your filter,
> > and this results buffers exhaustion (output_buffers,
> > proxy_buffers) and transfer stall.
> >
> > If your code can't work with a data stream and needs full response
> > before it can start it's work, you have to copy needed data from
> > buffers and release buffers got (by setting b->pos = b->last, like
> > if a buffer was actually sent).
> >
> > To test things you may try the following in config:
> >
> > output_buffers 1 1k;
> >
> > Then request a static file larger than 1k.
> >
> >
> > >
> > > Breno
> > >
> > >
> > > On Sat, May 11, 2013 at 9:06 PM, Maxim Dounin <mdounin at mdounin.ru>
> > wrote:
> > >
> > > > Hello!
> > > >
> > > > On Sat, May 11, 2013 at 08:09:59PM -0300, Breno Silva wrote:
> > > >
> > > > > Hello Maxim,
> > > > >
> > > > > We are sending chains data into a brigade/bucket structure (APR). It
> > > > > sometimes works fine.. sometimes trigger the cpu issue
> > > > >
> > > > > If i call:
> > > > > rc = move_chain_to_brigade(in, ctx->brigade, r->pool, 1); <---
> > change
> > > > > from 0 to 1
> > > > > Then i will get only the first X bytes of the response body. This
> > works
> > > > > great... but i need the entire buffer :)
> > > > >
> > > > > Could you give us more details how we could patch this function to
> > fix
> > > > the
> > > > > issue ?
> > > >
> > > > Remove the
> > > >
> > > > ngx_free_chain(pool, cl);
> > > >
> > > > call from the move_chain_to_brigade() function, it should fix you
> > > > CPU hog issues.
> > > >
> > > >
> > > > >
> > > > > Thanks
> > > > >
> > > > >
> > > > > On Sat, May 11, 2013 at 7:56 PM, Maxim Dounin <mdounin at mdounin.ru>
> > > > wrote:
> > > > >
> > > > > > Hello!
> > > > > >
> > > > > > On Sat, May 11, 2013 at 04:23:00PM -0300, Breno Silva wrote:
> > > > > >
> > > > > > > Hello list,
> > > > > > >
> > > > > > > We are porting ModSecurity to NGINX. However we are seeing
> > sometimes
> > > > an
> > > > > > > issue. Nginx eats 100% of cpu and when i use gdb i see:
> > > > > > >
> > > > > > > gdb -p 8645
> > > > > > >
> > > > > > > ngx_event_pipe_write_to_downstream (p=0x9bbc720, do_write=0) at
> > > > > > > src/event/ngx_event_pipe.c:551
> > > > > > >
> > > > > > > 551 if (cl->buf->recycled) {
> > > > > > > (gdb)
> > > > > > >
> > > > > > > Looks like it is happening when we call
> > > > > > ngx_http_modsecurity_body_filter()
> > > > > > > then go to this conditions;
> > > > > > >
> > > > > > > rc = move_chain_to_brigade(in, ctx->brigade, r->pool, 0);
> > > > > > >
> > > > > > > if (rc != NGX_OK) {
> > > > > > >
> > > > > > > r->buffered |= NGX_HTTP_SSI_BUFFERED;
> > > > > > >
> > > > > > > return rc;
> > > > > > >
> > > > > > > }
> > > > > > >
> > > > > > > move_chainto_brigade is defined as:
> > > > > > >
> > > > > > >
> > > > > > > ngx_int_t
> > > > > > >
> > > > > > > move_chain_to_brigade(ngx_chain_t *chain, apr_bucket_brigade *bb,
> > > > > > > ngx_pool_t *pool, ngx_int_t last_buf) {
> > > > > > >
> > > > > > > apr_bucket *e;
> > > > > > >
> > > > > > > ngx_chain_t *cl;
> > > > > > >
> > > > > > >
> > > > > > > while (chain) {
> > > > > > >
> > > > > > > e = ngx_buf_to_apr_bucket(chain->buf, bb->p,
> > > > bb->bucket_alloc);
> > > > > > >
> > > > > > > if (e == NULL) {
> > > > > > >
> > > > > > > return NGX_ERROR;
> > > > > > >
> > > > > > > }
> > > > > > >
> > > > > > >
> > > > > > > APR_BRIGADE_INSERT_TAIL(bb, e);
> > > > > > >
> > > > > > > if (chain->buf->last_buf) {
> > > > > > >
> > > > > > > e = apr_bucket_eos_create(bb->bucket_alloc);
> > > > > > >
> > > > > > > APR_BRIGADE_INSERT_TAIL(bb, e);
> > > > > > >
> > > > > > > chain->buf->last_buf = 0;
> > > > > > >
> > > > > > > return NGX_OK;
> > > > > > >
> > > > > > > }
> > > > > > >
> > > > > > > cl = chain;
> > > > > > >
> > > > > > > chain = chain->next;
> > > > > > >
> > > > > > > ngx_free_chain(pool, cl);
> > > > > > >
> > > > > > > }
> > > > > > >
> > > > > > >
> > > > > > > if (last_buf) {
> > > > > > >
> > > > > > > e = apr_bucket_eos_create(bb->bucket_alloc);
> > > > > > >
> > > > > > > APR_BRIGADE_INSERT_TAIL(bb, e);
> > > > > > >
> > > > > > > return NGX_OK;
> > > > > > >
> > > > > > > }
> > > > > > >
> > > > > > > return NGX_AGAIN;
> > > > > > >
> > > > > > > }
> > > > > > > Let me know if you guys can help us understanding why sometimes
> > we
> > > > > > trigger
> > > > > > > this issue
> > > > > >
> > > > > > It looks like your code modify chain links (ngx_chain_t
> > > > > > structures) as got by your response body filter. This is not
> > > > > > something filters are allowed to do, and results are undefined.
> > > > > > If a filter needs to modify chain, it should allocate it's own
> > > > > > chain links.
> > > > > >
> > > > > > In the code quoted you probably don't want to touch chain links at
> > > > > > all.
> > > > > >
> > > > > > --
> > > > > > Maxim Dounin
> > > > > > http://nginx.org/en/donation.html
> > > > > >
> > > > > > _______________________________________________
> > > > > > nginx-devel mailing list
> > > > > > nginx-devel at nginx.org
> > > > > > http://mailman.nginx.org/mailman/listinfo/nginx-devel
> > > > > >
> > > >
> > > > > _______________________________________________
> > > > > nginx-devel mailing list
> > > > > nginx-devel at nginx.org
> > > > > http://mailman.nginx.org/mailman/listinfo/nginx-devel
> > > >
> > > >
> > > > --
> > > > Maxim Dounin
> > > > http://nginx.org/en/donation.html
> > > >
> > > > _______________________________________________
> > > > nginx-devel mailing list
> > > > nginx-devel at nginx.org
> > > > http://mailman.nginx.org/mailman/listinfo/nginx-devel
> > > >
> >
> > > _______________________________________________
> > > nginx-devel mailing list
> > > nginx-devel at nginx.org
> > > http://mailman.nginx.org/mailman/listinfo/nginx-devel
> >
> >
> > --
> > Maxim Dounin
> > http://nginx.org/en/donation.html
> >
> > _______________________________________________
> > nginx-devel mailing list
> > nginx-devel at nginx.org
> > http://mailman.nginx.org/mailman/listinfo/nginx-devel
> >
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel
--
Maxim Dounin
http://nginx.org/en/donation.html
More information about the nginx-devel
mailing list