Parallel subrequests for multi-source long polling?

[agentzh]

On Thu, Nov 19, 2009 at 5:54 PM, agentzh <agentzh at gmail.com> wrote:
>
> It's weird that the client sees the response header and body after the
> slowest subrequest finishes. It seems that the response has been
> buffered in the last few output filters somehow. Please ensure that
> you have set b->flush and b->last_buf in your output chain link. These
> flags should defeat buffering in most cases.
>

Okay, I was wrong here :) The output body of the current subrequest
and its parent request  will be permanently buffered if the request
under question is not at the current head of the postponed chain. For
example

    location /main {
        echo hello;
        echo_flush;
        echo_location_async '/foo';
        echo_location_async '/bar';
        echo_location_async '/baz';
        echo world;
        echo_flush;
    }

    location /foo {
        echo_sleep 1;
        echo foo;
        echo_flush;
    }

    location /bar {
        echo_sleep 2;
        echo bar;
        echo_flush;
    }

    location /baz {
        echo_sleep 1;
        echo baz;
        echo_flush;
    }

Accessing /main using curl will show "hello" immediately, and "foo" 1
sec later, and finally "bar", "baz", and "world" together after
another 1 sec. So if the slowest subrequest is issued first, like the
location /foo here, then there's little hope to get the outputs of
later subrequests like /baz properly flushed without using hacks.

The "world" output of the main request is buffered because it's at the
end of the postponed chain while "hello" is at the head. When "hello"
gets out, "foo" becomes the head of the postponed chain.

> And still, we have to cancel the pending subrequests, but close the
> connection seems a bit overkill especially in the context of HTTP
> keepalive.
>

Forcibly cancle subrequests can be dangerous because we have to ensure
all those timers and event handlers get properly cleared. And I have
never done such things myself. Sorry about that. chaoslawful and I
will take a closer look at this issue but with no promise.

>
>> I tried calling ngx_http_finalize_request with various rc values on the pending subrequests, hoping to find something that would force the completion, but nothing seemed to work
>

Yeah, it won't work. I've tested within my "echo" module by
introducing a "echo_abort_parent" directive. I wonder if we'll have to
arrange the postponed chain ourselves or bypass the postpone filter
completely. I'm not sure. It's getting evil already :)

Maybe other people on the list can give some advice on canceling a
pending subrequest?

Cheers,
-agentzh