Buffer reuse like gzip filter module, with pre-configured number of buffers

[hanzhai]

Hi,

I am writing my own filter module based on the gzip filter module. My filter
module would first insert a long text (200 to 1024 KB based on the
situation) at the beginning of the original response and then do some other
manipulations to the original response. The pre-configured number of buffers
that can be allocated per request (like the gzip_buffers directive in gzip
filter module) will reduce the risk of OOM. 

Here is the example code:


ngx_int_t rc;
ngx_chain_t *ch = NULL;
size_t len = 64 * 4096;

ngx_buf_t *b = ngx_create_temp_buf(r->pool, len);
if (b == NULL) {
    return NGX_ERROR;
}
b->recycled = 1;
b->tag = (ngx_buf_tag_t) &ngx_http_my_filter_module;

// operation to copy 64 kb data to the ngx_buf_t b;
b->last = b->pos + len;

ch = ngx_alloc_chain_link(r->pool);
if (ch == NULL) {
    goto failed;
}
ch->buf = b;
ch->next = NULL;

rc = ngx_http_next_body_filter(r, ch); // rc got NGX_OK
if (rc != NGX_OK && rc != NGX_AGAIN) {
    goto failed;
}

ngx_chain_update_chains(r->pool, &ctx->free, &ctx->busy, &ch,
                        (ngx_buf_tag_t) &ngx_http_my_filter_module);

ngx_chain_t *cl = ch;
while (cl) {
    ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, "buf sz: %z",
ngx_buf_size(cl->buf)); // output: buf sz: 65536
    cl = cl->next;
}

// reuse the buffer b if possible


Explanation:

I created a buffer b with 64k, then I copied 64k data, then I called
ngx_http_next_body_filter to send the first 64k data to the client, then I
want to reuse the buffer b to send the next 64k data and so on.

The question is after ngx_http_next_body_filter, ctx->free is still NULL,
because ngx_buf_size(b) got 65536 which means that the buffer b cannot be
recycled. My guess even if the next_body_filter returns an NGX_OK, it
doesn't guarantee any of the buffers inside the ngx_chain_t can be reused.

What should I do if I want to reuse the 64k buffer?

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,294374,294374#msg-294374