custom handler module - dynamic response with unknown content length

[Maxim Dounin]

Hello!

On Sat, Mar 01, 2014 at 12:48:11PM +0330, Yasser Zamani wrote:

> Thanks for your response....
> 
> On Sat 01 Mar 2014 03:41:24 AM IRST, Maxim Dounin wrote:
> >Hello!
> >
> >You've tried to send the same chain with the same buffer multiple
> >times.  After a buffer is sent for the first time, its pointers
> >are adjusted to indicate it was sent - b->pos moved to b->last, and
> >buffer's size become zero.  Second attempt to send the same buffer
> >will expectedly trigger the "zero size buf" check.
> >
> Great! I tried:
> 
> 	for(i=1;i<10000000;i++){b->flush =
> (0==(i%100));rc=ngx_http_output_filter(r,
> &out);if(rc!=NGX_OK){ngx_log_error(NGX_LOG_ALERT,r->connection->log,0,"bad
> rc, rc:%d", rc);return rc;}b->pos = ngx_hello_string;b->last =
> ngx_hello_string + sizeof(ngx_hello_string) - 1;b->memory = 1;b->last_buf =
> 0;}
> 
> which now fails with:
> 
> 2014/03/01 12:23:39 [alert] 5022#0: *1 bad rc, rc:-2, client: 127.0.0.1,
> server: localhost, request: "GET / HTTP/1.1", host: "localhost:8080"
> 2014/03/01 12:23:39 [alert] 5022#0: *1 zero size buf in writer t:0 r:0 f:0
> 00000000 080C7431-080C7431 00000000 0-0, client: 127.0.0.1, server:
> localhost, request: "GET / HTTP/1.1", host: "localhost:8080"
> 
> And to resolve this I know I should follow the solution at [3].
> [3] http://web.archiveorange.com/archive/v/yKUXMLzGBexXA3PccJa6
> 
> But is this a clean way to call 'ngx_http_output_filter' more than once?
> (please see below to know why I have to call it multiple times)

The ngx_http_output_filter() function can be called more than 
once, but usually it doesn't make sense - instead, one should 
install r->write_event_handler and do subsequent calls once it's 
possible to write additional data to socket buffer.  Working with 
event's isn't something trivial though.

> FYI: Previous try did not fail in SECOND attempt but it failed when client
> successfully download 936 bytes of repeated "Hello, World!"s (72 attempts).
> 
> >Trivial aproach is to prepare full output chain, and then send it
> >using single ngx_http_output_filter() call.
> 
> The full output chain will be usually a long video which is not a file but
> will be generated in memory on the fly. I have to send each chunk as soon as
> it's ready because the stream generation is time consuming and client COULD
> NOT wait for all to be done. Suppose it's a 1 hour video which dynamically
> has been generated and I would like to send each minute as soon as it's
> ready without waiting for all 1 hour transcoding. I'm aware about nginx's
> mp4 module but it does not support time consuming dynamically generated
> video on memory.
> 
> WHAT WILL BE THE CORRECT WAY TO DO THIS IN NGINX?

Quoting Winnie-the-Pooh, "You needn't shout so loud".

Doing time-consuming transcoding in nginx worker isn't correct in 
any case, as it will block all connections in this worker process.  
So you have to do transcoding in some external process, and talk 
to this process to get transcoded data.  This is basically what 
upstream module do (as used by proxy, fastcgi, etc.), and it 
can be used as an example of "how to do this in nginx".

-- 
Maxim Dounin
http://nginx.org/