Chunked request body and HTTP header parser

[agentzh]

On Tue, Nov 24, 2009 at 10:30 AM, agentzh <agentzh at gmail.com> wrote:
> I've tested my "chunkin" module with the standard proxy module and it
> works flawlessly.

"Flawlessly" seems to be too optimistic :P I've just fixed a bunch of
serious bugs in ngx_chunkin, prompted by the guy simply known as "J"
in the last few days. It's finally looking much better now and
still...considered highly experimental :)

> Because ngx_http_read_client_request_body does not
> do anything when r->request_body already exists.
>

I must add that ngx_chunkin does not affect how the ngx_proxy module
decodes an upstream http server's response. It's a in very different
context and that's not the problem that ngx_chunkin originally aimed
to solve.

> Regarding the pipelined requests in a keep-alive connection, yes, it's
> a TODO :) I'll have to reuse the bytes left in a previous request's
> buffer.
>

Yesterday I quickly hacked in a preliminary support for keepalive and
pipelining for ngx_chunkin and my simplest test cases (driven by the
non-blocking Perl socket API) are already passing.

The basic idea is to copy the bytes left by my chunked parser in
r->request_body->buf over into r->header_in so that nginx's
ngx_http_set_keepalive and ngx_http_init_request functions will pick
it up for the subsequent pipelined requests. When the request body is
small enough to be completely preread into the r->header_in buffer,
then no data copy is needed here -- just setting r->header_in->pos
correctly will suffice.

The only issue that remains is how to enlarge r->header_in when the
data left in r->request_body->buf is just too large to be hold in the
remaining room between r->header_in->pos and r->header_in->end. For
now, ngx_chunkin will just give up and simply turn off r->keepalive.

I know we can always use exactly the remaining room in r->header_in as
the buffer size when reading data from c->recv, but's suboptimal when
the remaining room in r->header_in happens to be very small while
r->request_body->buf is quite large.

I haven't fully grokked all the details among r->header_in, c->buffer,
busy/free lists and those so-called "large header buffers". Is there a
clean a safe way to reallocate or extend the r->header_in buffer?

> It seems that the existing ngx_http_read_client_request_body is having
> similar problems there? Because the leading bytes of the next request
> is likely to be read into r->header_in when the previous request's
> body and headers are small enough. But I'm not sure :)

This statement is totally wrong :) Nginx does a great job in
supporting HTTP 1.1 pipelining, as far as I've tested and checked. I'd
really see ngx_chunkin has full support for this HTTP 1.1 feature,
thus we'd see "streams of streams" flowing into our dear nginx engine.
Wow.

Cheers!
-agentzh