How do proxy_module response buffering options work?

[Maxim Dounin]

Hello!

On Mon, Apr 25, 2011 at 10:27:05AM +0200, Hongli Lai wrote:

> Maxim Dounin wrote in post #994792:
> > No, nginx passes downstream chain of buffers, not a single buffer.
> 
> I understand this, but I don't understand how there can be more than 1 
> buffer at a time that's only partially passed downstream. I had in mind 
> that it works like this:
> 
> Suppose that Nginx is configured with 4 buffers, each 100 bytes in size. 
> The upstream response is 400 bytes.
> Bytes 0-99 from the response are put in buffer 1, bytes 100-199 are put 
> in buffer 2, bytes 200-299 are put in buffer 3, bytes 300-399 are put in 
> buffer 4.
> Nginx flushes buffer 1 downstream, thus buffer 1 is now a "busy buffer" 
> while all the other ones are not. Only when buffer 1 is entirely flushed 
> will Nginx continue to flush buffer number 2 (buffer 1 is now marked 
> "ready" of some sort so that it may be reused.)
> 
> How can there be multiple busy buffers? Is my description correct?

No.  nginx passes downstream (or, more precisely, passes to output 
filter chain) multiple buffers, and these buffers can't be reused 
unless completely send (see below).

E.g. with 4 buffers as in your example something like this 
happens:

1. nginx gets 0-99 from upstream to buf 1, and calls output filter 
on it.

>From this point buf 1 is busy - it can't be touched unless 
completely send, as some filter may have been already modified it 
(e.g. converted charset in charset filter) or done some other 
related work on it (e.g. added chunk-size as per chunked transfer 
encoding).  We can't spool it to disk and so on.

2. nginx gets 100-199 from upstream to buf 2, and calls output 
filter on it.  The same as the above now applies to buf 2 as well.

Buffers 1 and 2 are now linked somewhere along output filter chain 
(most likely in writer filter, waiting for client to allow sending 
of additional data).

(Both (1) and (2) may also happen at the same time with single 
output filter call if data to both buffers happened to be read at 
once.)

Somewhere near here proxy_busy_buffers_size starts to play it's 
role: once data is read to buffers 3 and 4 - they are not passed 
to output filter (assuming proxy_busy_buffers_size is 200) and 
nginx may spool data to disk and reuse these buffers to read more 
data from upstream.

(Obviously the case with 400-byte response isn't intresting: if we 
have enough buffers to read the whole response all buffers will be 
just passed to output filters.  All of the above is actually 
needed when the response is bigger than the available buffers.)

Maxim Dounin