Usage of the shadow field in ngx_buf_t?

[Dave Bailey]

On Thu, Dec 3, 2009 at 6:29 PM, Brian Pane <brianp at brianp.net> wrote:

> After an afternoon with gdb, I think I've found one dependency: before
> throwing away the buffers, the body filter needs to set buf->pos=buf->last
> for each one.  This is due to the code at line ~201 of
> core/ngx_output_chain.c:
>         last = ctx->output_filter(ctx->filter_ctx, out);
>         if (last == NGX_ERROR || last == NGX_DONE) {
>             return last;
>         }
>         ngx_chain_update_chains(&ctx->free, &ctx->busy, &out, ctx->tag);
>         last_out = &out;
> After the filter returns, that code still references the filter chain that
> was
> passed into the filter (the variable "out").  It looks like the event loop
> in ngx_event_pipe_write_to_downstream counts up the unsent bytes in
> this chain, where unsent for each buf is computed as buf->last - buf->pos.
> I don't yet understand all the logic that follows, but the end result was
> that
> for large responses, I'd eventually cross a threshold where my filter kept
> getting called over and over with no new data.

I think that's right.  I've written an nginx body filter that uses
ngx_chain_update_chains, and the input buffers need to be marked as
fully consumed (buf->pos = buf->last) in order for
ngx_chain_update_chains to recycle them properly.  In many cases, this
permits the same one or two buffers to get recycled over and over
again as the entire request body is passed through the filter.  I
think it's one of the reasons nginx body filters are so incredibly
high performance.

I used the gzip body filter source code to figure out how to consume
input buffers in their entirety, and generate completely new output
buffers (or no output buffers, as desired).

-dave

-dave