Parallel subrequests for multi-source long polling?

[shaun]

agentzh Wrote:
-------------------------------------------------------
> On Wed, Nov 18, 2009 at 2:30 PM, shaun  wrote:
> > Here's a description of what I'm hoping to
> accomplish:
> 
> Here's a proposal for your consideration:
> 
> 1. You need a content handler and an output filter
> to work together.
> 2. Use a content handler to issue all the
> subrequests (to all your
> upstream backends). We assume the content handler
> lives in the main
> request.
> 3. When starting each subrequest, the content
> handler feeds a
> "post_subrequest" handle and the main request's
> ctx object into the
> ngx_http_subrequest call. Also, the content
> handler registers a ctx
> object to the subrequest object to inform the
> subrequest's output
> filter to buffer the response.
> 4. In your output filter, first check if there's a
> ctx object
> associated with the current request. If yes, then
> you're filtering one
> of the subrequests that your content handler
> starts. So now simply
> buffer the response chain link in this ctx object
> (in meomory or on
> disk, it's up to you).
> 5. In your "post_subrequest" handle, check the
> main request's ctx
> object (it should the the last argument passed
> in). We assume there's
> a "success" flag and a "failures" counter in that
> ctx object (both are
> zeroed initially), as well as the total number of
> subrequests. If
> "success" flag is not set and if the current
> subrequest succeeds,
> simply return the buffered response contents in
> your subrequest's ctx
> object, and set a "success" flag in your main
> request's ctx object. If
> the current subrequest fails, then increment the
> "failures" counter.
> If the "failures" counter reaches the total number
> of the subrequests
> (which means all tests are failed), then returns
> the final failure
> (and set the main request's status code if
> appropriate and send
> headers for the main request.)
> 
> Your main request, i.e., the content handler
> should not send headers
> or any contents by itself and should return
> NGX_DONE.
> 
> I think this model should work after my various
> experiments in the
> "echo" module, but I could be wrong ;) Feel free
> to tell us your
> findings on your way :)
> 
> Good luck!
> -agentzh

This seems like a perfectly reasonable way to tackle the problem.

I was trying to do this without using an output filter, which, in retrospect, doesn't make much sense.  Thanks for the advice!  I'll see if I can get something working tomorrow.

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,23767,23794#msg-23794