Parallel subrequests for multi-source long polling?

[Piotr Sikora]

> So, as another option, I wonder if it's possible to do this using 
> completely separate requests.  I have no idea about the feasibility of 
> this, but it seems like it could work.  I could create new requests where 
> I was using subrequests before and then configure the upstream manually, 
> attach a context and then collate the results in the output filter. 
> Everything would work nearly the same except the finalizing of the main 
> request would have to happen in the child request filter instead of the 
> subrequest's postreq handler.
>
> I'll take a stab at hacking the postpone chains and if that doesn't seem 
> safe or sane, I'll give the second option a try.

If you want to go that route then I believe that you could re-use a lot of 
code from ngx_supervisord[1]. Most of the code below line 993 ("nginx <> 
supervisord communication" comment) does exacly what you need (creating 
"fake request", connecting to upstream outside of "upstream {}" confing, 
etc).

One thing you should know upfront is that I needed requests which would be 
totally independent from any real connections / requests (ngx_supervisord 
communicates with supervisord even when there are no requests - to shut down 
idle backends, for example), and that's why there is a lot of code to create 
"fake" connections, events and configs in ngx_supervisord_init. I'm pretty 
sure you could get away with pointing most of them to data from original 
request...

Also, responses from supervisord are quite small, so I'm forcing nginx to 
read them into memory. You might need to use temp files for bigger things.

[1] http://labs.frickle.com/nginx_ngx_supervisord/

Best regards,
Piotr Sikora < piotr.sikora at frickle.com >