question about proxy_buffers

[Jérôme Loyet]

2009/6/11 Maxim Dounin <mdounin at mdounin.ru>:
> Hello!
>
> On Thu, Jun 11, 2009 at 08:00:04PM +0200, Jérôme Loyet wrote:
>
>> 2009/6/10 merlin corey <merlincorey at dc949.org>:
>> > I'm not Igor so I cannot be totally sure, but my understanding of it
>> > has always been 1- requests to the backend can use only one buffer and
>> > the response gets buffered to disk.  As such, I generally start out
>> > sites with log level info (might be too low) and try to tune down
>> > requests getting buffered if at all possible.  I also try to stick
>> > with magic numbers good for the architecture (so 8 and 16 based,
>> > generally).
>> >
>> Thx for the answer merlin,
>>
>> I did some test. The request I make to nginx is proxied to the
>> backend. There is only one request made at the same time.
>> I request a 5Mo file.
>>
>> First: with minimum buffers:
>> proxy_buffers 2 4k;
>>
>> I have this in the log: 2009/06/11 19:40:05 [warn] 24607#0: *104 an
>> upstream response is buffered to a temporary file
>> /LIBRE/nginx/proxy_temp/0000000000 while reading upstream, ....
>>
>> Then I set 10 buffers of 1Mo:
>> proxy_buffers 10 1m;
>> and nothing appears in logs.
>>
>> This tells me that a request can use more than one buffer.
>
> Yes.
>
>> I'll set up small buffers but a large number:
>> proxy_buffers 2048 8k;
>>
>> Is it too much ?
>
> Each connection that uses proxy can use up to 16M of memory with
> this setting (even after actual connection to backend has been
> already closed).  With 1024 client connections this means up to
> 16G of memory, probably a bit too many.

Do you mean that proxy_buffers are dedicated to each connection ?
I thought it was shared by all connections.

>
> And keep in mind that:
>
> - Doing reconfiguration will spawn new worker processes and memory
>  usage by nginx nearly doubles for a while.
>
> - You also need some memory for OS's disk cache.
>
> Some hints about seveal big vs. many small buffers:
>
> - Using bigger buffers may help a bit with CPU usage, but usually
>  it's not a big deal.
>
> - Using bigger buffers means less disk seeks if reply won't fit
>  into memory and will be buffered to disk.
>
> - On the other hand, small buffers allow better granularity (and
>  less memory wasted for last partially filled buffer).
>
> Since there are different load patterns (and different hardware)
> it's hard to recommend some particular setting, but the about
> hints should allow you to tune it for your system appropriately.
>
> Maxim Dounin
>
>
>>
>> >
>> > 2009/6/9 Jérôme Loyet <jerome at loyet.net>:
>> >> Hi all,
>> >>
>> >> I'm not sure how proxy_buffers are used.
>> >>
>> >> 1- requests to the backend can use only one buffer. If the reponse is
>> >> bigger than one buffer, some part of the response is buffered to disk.
>> >> 2- requests to the backen can use multiple buffers. If the response is
>> >> bigger than on buffer, nginx will use more buffers (if available of
>> >> course).
>> >>
>> >> So depending on the answer, I would use :
>> >> 1- big buffers so that 80% of the requests can feet in on buffer. And
>> >> enough buffers so that 80% of the requests can be buffered
>> >> 2- small buffers and a lots of buffers
>> >>
>> >> don't know what to use :) any clue ?
>> >>
>> >> thx
>> >> ++ jerome
>> >>
>> >>
>> >
>> >
>>
>
>