proxying from pipes

[Maxim Dounin]

Hello!

On Tue, Oct 23, 2012 at 11:14:52AM -0400, Jeff Kaufman wrote:

> My module wants to sit in the filter chain passing buffers to an
> asynchronous optimization thread and then send them out to the user
> when they finish.  When a request comes in I have my module roughly
> doing:
> 
> body filter:
>  - if first set of buffers
>    - create pipe, pass pipe_write_fd to optimization thread
>  - pass all input data to optimization thread
>  - don't call ngx_http_next_body_filter
> 
> Is there a way I can ask nginx to watch this pipe and treat any data
> appearing on the pipe as if it is output from my body filter?  Passing
> it through ngx_http_next_body_filter, etc?  And finalizing the request
> if there's a problem with the pipe?

You may try to reuse ngx_event_pipe() code for this, much like 
it's used in the upstream module.  It's not designed to be used in 
a filter chain though, and anyway I suspect it would be an 
overkill unless you need some request buffering as well.  In any 
case you'll need to handle events and filter chain invocations 
yourself anyway.

> What I'm doing now is:
>  - when creating the pipe
>      c = ngx_get_connection(pipe_read_fd, r->connection->log);
>      c.read->handler = my_read_handler;
>      ngx_add_event(c->read, NGX_READ_EVENT, 0);
>  - when my_read_handler is called
>      read() from pipe_read_fd
>      create buffer, chain link
>      call ngx_http_next_body_filter
> 
> Is this the right way to go about this?  It seems very low level and I
> would expect proxying from a pipe would be something nginx already
> supported.

This looks reasonable, though may need more details.  In 
particular you may need to look into ngx_event_connect() to see 
how to properly set up event notification for a connection, as 
e.g. ngx_add_conn() is needed for rtsig.

> Another option would be to simply use the pipe for notification and
> load the data from the optimization thread through shared memory.

Depending on use case this aproach may be more efficient.

-- 
Maxim Dounin
http://nginx.com/support.html