CPS-chained subrequests with I/O interceptions no longer work in nginx 0.8.21 (Was Re: Custom event + timer regressions caused by the new release)

[agentzh]

On Wed, Oct 28, 2009 at 12:26 PM, agentzh <agentzh at gmail.com> wrote:
> I'm guessing that nobody has done such things before, but I do believe
> this is really useful for doing component-style programming in nginx,
> because we don't have to rely on output filters (that are both costly
> and *very* verbose to write) to hack in sequential subrequests.
>

Okay, folks, I've just created a git branch for my "echo" module and
quickly hacked in the "classical" model of sequential subrequests by
means of output filters.

But again, 0.8.21 hangs the tests and passing every test if commenting
out that line. 0.8.20 is still passing as before.

The "classical" model looks like this:

1. The content handler issues a subrequest A, and registers a specific
context object to the subrequest A. This context object could be
recognized by a special output filter, say, F. And the current request
object (ngx_http_request_t instance) associated with the main content
handler is put into this context object.

2. The subrequest A generates outputs, chain by chain. And the F
filter runs over them, chain by chain as well. The F filter detects if
there's a context object attached to the current request object, if
not, just calls the next body filter; if yes, walks through the
current chain link to find out the last buf indicating the end of the
currrent subrequest's response body stream.

3. If the last buf is found, the filter F takes the original request
object that is associated with the original content handler from its
context object, and initiates subrequests or do other things (actually
this part is implemented as resuming the continuation of the content
handler in the "echo" module).

The tricky part is to search the "last buf" in the filter F which runs
over the response body of a subrequest. It seems that buf->last_buf is
already cleared by something else (more investigation needed) and
buf->sync is set instead. So currently I'm cheating a bit and just
checking the buf->sync flag and thinking it's the right way to do in
at least late 0.8.x's. Well, it does not work in 0.7.x anyway.
Thoughts?

You can find the relevant source code here:

   http://github.com/agentzh/echo-nginx-module/blob/master/src/location.c

and here

    http://github.com/agentzh/echo-nginx-module/blob/cps-filter/src/cps-filter.c

where "cps_filter" is the codename for the filter F in the previous explanation.

Even though the "last buf" searching in subrequest output filter is
still a hack, this experiment has convinced me that it looks more like
a regression in nginx itself rather than misuse of the
"post_subrequest" thingy or something else. In my cps-filter branch,
I'm not using "post_subrequest" anyway.

What do *you* think? ;)

Cheers,
-agentzh

Cheers,
-agentzh