One last try - large long-running worker tasks

[Jeff Heisz]

Hi all, I've asked this before with no response, trying one last time
before I just make something work.

I'm making a custom module for nginx that does a number of things but
one of the actions is a long-running (in the nginx sense) task that
could produce a large response.  I've already got proper processing
around using worker tasks for the other long-running operations that
have small datasets, but worry about accumulating a large amount of
memory in a buffer chain for the response.  Ideally it would drain as
fast as the client can consume it and throttle appropriately, there
could conceivably be gigabytes of content.

My choices (besides blowing all of the memory in the system) are:

- write to a temporary file and attach a file buffer as the response,
less than ideal as it's essentially translating a file to begin with,
so it's a lot of disk I/O and performance will be less than stellar.
>From what I can tell, this is one of the models for the various CGI
systems for caching, although in my case caching is not of use

- somehow hook into the eventing system of nginx to detect the write
transitions and implement flow control directly using threading
conditionals.  I've tried this for a few weeks but can't figure out
the 'right' thing to make the hooks work in a separate module without
changing the core nginx code, which I'm loathe to do (unless you are
looking for someone to contribute such a solution, but I'd probably
need some initial guidance)

- attach a kernel pipe object (yah yah, won't work on Windows, don't
care) to each of my worker instances and somehow 'connect' that as an
upstream-like resource, so that the nginx event loop handles the
read/write consumption and the thread automatically blocks when full
on the kernel pipe.  Would need some jiggery to handle reuse and
start/end markers.  Also not clear if I can override the connection
model for the upstream without again changing core nginx server code

Any thoughts?  Not looking for code here (although telling me to look
at the blah-blah-blah that does exactly this would be awesome), but if
someone who is more familiar with the inner workings of the nginx data
flow could just say which solution is a non-starter (so I don't waste
time trying to make it work) or even which would be a suitable
solution would be awesome!

jmh