Tuning client request buffering in ngx_http_proxy_module

[Maxim Dounin]

Hello!

On Tue, Feb 01, 2022 at 03:19:55PM -0500, bengalih wrote:

[...]

> My direct https speeds to the backend are about 100mbps. (sans NGINX)
> As soon as I go through NGINX those speeds are almost cut in half (~55mbps)
> *if* "proxy_request_buffering off".
> My NGINX (v1.18) is running on embedded hardware and only has about 100MB
> RAM available (2 GB swap).
> I do see very high CPU utilization when uploading and blame the decrease in
> bandwidth to this - something I need to live with for now.
> However memory utilization is almost nothing with this configuration.

Note that SSL is likely the most important contributor to CPU 
utilization in this setup.  It might be a good idea to carefully 
tune ciphers used.

> If I keep proxy_request_buffering the default of "on" then my speeds are
> further reduced to about 30mbps.
> In addition to high CPU usage I also see memory usage of about 75% of my
> available memory (50-75MB).

Memory usage in this setup is likely related to OS disk cache.  
Not really something to care about: OS simply uses available 
free memory for caching.

[...]

> So based on the above I have two questions:
> 1) While I understand that the entire file must be written to disk because
> the upload size is greater than the buffers, where is this file being
> written?  It is clearly being written to the disk but LSOF shows it as
> deleted even though it continues to grow (as reflected by LSOF and df) as
> the upload continues.

To re-iterate what was already written in the previous message: 
nginx opens a temporary file, immediately deletes it, and then 
uses this file for disk buffering.  The file is expected to be 
deleted from the very start, and it is expected to grow over time 
as it is used for disk buffering.  

> 2) It would seem, that with such large uploads it makes the most sense to
> keep "proxy_request_buffering off" but assuming you needed the advantages of
> this (like you specify in your first reply), is there anything that can be
> done to tune this so that the speeds are faster and especially there isn't
> such a long delay at the 99% upload?  I played around with some buffer
> settings, but none of them seem to really make any noticeable effect.

As far as I understand from your description, the "long delay at 
the 99% upload" you see with request buffering switched on isn't 
really at 99%, but instead at 100%, when the request is fully sent 
from the client to nginx, and the delay corresponds to sending the 
request body from nginx to the backend server.  To speed up this 
process, you have to speed up your backend server and connection 
between nginx and the backend, as well as the disk on nginx 
server.  You may also try to tune how nginx sends the request body 
to the backend, such as sendfile (http://nginx.org/r/sendfile), 
though unlikely it will have noticeable effect with a single 
client and on embedded hardware.

Given limited resources on nginx box, as well as small number of 
clients and only one backend server, "proxy_request_buffering 
off;" might be actually a better choice in your setup.

-- 
Maxim Dounin
http://mdounin.ru/