Usage of the shadow field in ngx_buf_t?

[Brian Pane]

On Thu, Dec 3, 2009 at 5:35 PM, agentzh <agentzh at gmail.com> wrote:

> On Fri, Dec 4, 2009 at 7:13 AM, Brian Pane <brianp at brianp.net> wrote:
> > A follow-up question: When a body filter is processing buffers that
> > were produced by the proxy module (i.e., buffers containing an HTTP
> response
> > from a back-end server), is it safe to discard those buffers?
>
> I think so :) My ngx_chunkin module's output body filter does
> something like that to simply disgard the contents of the 411 error
> page and I don't think an upstream module like ngx_proxy will have any
> difference here :)
>
> > Or is it necessary to
> > pass the buffers through to the standard filters?
>
> Then your client will probably see those stuffs sooner or later, which
> is certainly not what you want :)
>
> >  It's not clear to me
> > whether anything in the standard filter chain needs to see the buffers
> > containing the upstream response in order to do cleanup or flow control.
>
> No, at least according to my knowledge :) Cleanup and flow control
> usually happens in places like ngx_http_finalize_request, or a
> read/write event handler. A notable exception might be bufs with the
> temporary field set to 1. These bufs might need to be freed sooner
> rather than later, but they're usually allocated in the current
> request's pool anyway, it may not be a very big issue.
>

After an afternoon with gdb, I think I've found one dependency: before
throwing away the buffers, the body filter needs to set buf->pos=buf->last
for each one.  This is due to the code at line ~201 of
core/ngx_output_chain.c:

        last = ctx->output_filter(ctx->filter_ctx, out);

        if (last == NGX_ERROR || last == NGX_DONE) {
            return last;
        }

        ngx_chain_update_chains(&ctx->free, &ctx->busy, &out, ctx->tag);
        last_out = &out;

After the filter returns, that code still references the filter chain that
was
passed into the filter (the variable "out").  It looks like the event loop
in ngx_event_pipe_write_to_downstream counts up the unsent bytes in
this chain, where unsent for each buf is computed as buf->last - buf->pos.
I don't yet understand all the logic that follows, but the end result was
that
for large responses, I'd eventually cross a threshold where my filter kept
getting called over and over with no new data.

-Brian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://nginx.org/pipermail/nginx-devel/attachments/20091203/09a1c7fc/attachment-0001.html>