Tuning client request buffering in ngx_http_proxy_module

[Maxim Dounin]

Hello!

On Mon, Jan 31, 2022 at 07:53:24PM -0500, bengalih wrote:

> I had/have an issue where I am proxying from NGINX to a backend WebDAV
> server.
> My upload speeds were slow and had long pauses in them including very long
> pauses at the end (5 minutes or more on uploads around 1GB).
> 
> Via packet captures I found that the NGINX server was not transmitting data
> synchronously to the backend WebDAV server, but was clearly doing buffering
> despite the fact I had set "proxy_buffering off".
> Looking at the documentation led me to "proxy_request_buffering off" which
> seems to have solved my problem and seems to immediately send the entire
> WebDAV client request directly to the backend server in a synchronous
> manner.

Just in case it's no clear, here are some background.

By default, nginx tries to minimize time needed for request 
processing on the backend.  It is a common situation when backend 
servers use process-per-connection model, and thus very 
ineffective in handling slow clients.  In contrast, nginx is an 
event-based server, and can handle slow clients and lots of 
connections efficiently.

To do so, nginx:

- buffers responses, see http://nginx.org/r/proxy_buffering, and
- buffers requests, see http://nginx.org/r/proxy_request_buffering

Additionally, buffering requests makes it possible to retry 
requests to a different backend server if the is a failure (see 
http://nginx.org/r/proxy_next_upstream).  Note that using 
"proxy_request_buffering off;" means retries won't work in most 
cases for requests with bodies.

> I did however want to experiment with proxy_request_buffering not disabled
> and see if setting smaller buffers to the backend would perhaps not result
> in such log asynchronous delays.
> 
> With "proxy_request_buffering on" (default) I have also set the
> "client_body_temp_path /tmp/cache" and set "client_body_in_file_only on".
> In this configuration I can see the client request get placed into the
> /tmp/cache directory, so I know my directive and paths are working.
> 
> However, if I set "client_body_in_file_only off" (default) no file gets
> created at all despite the fact that my client PUT request over WebDAV is
> undoubtedly larger than any buffer settings.
> 
> By doing a "df" on my NGINX box I can see that the space on my drive it
> being eaten up by the same amount equivalent to my upload (i.e. if I upload
> a 500 MB file I can see my free space decrease by 500 MB).  I cannot however
> see anything in /tmp/cache and have no idea where these files are being
> placed.

When nginx creates temporary files which are expected to be 
removed, it opens a file and immediately deletes it.  This way 
temporary files are automatically cleaned up by the system as soon 
as nginx closes them.  Further, if nginx process dies or killed, 
temporary files are also correctly cleaned up.

You can use "lsof" to see such files while they are still open by 
nginx, they are shown by lsof as "deleted".

> I also don't know how/why NGINX is caching/buffering the entire file and
> what is controlling how it is sending this to the backend server.  I have
> tried playing with the "client_body_buffer_size" but it does not appear to
> have any effect in how the data gets buffered or the size of the file.

The "client_body_buffer_size" controls the size of a memory buffer 
used for reading the request body.  If request buffering is used, 
anything larger than that will be written to disk, and using 
larger client_body_buffer_size might help to reduce disk load.  If 
request buffering is switched off, the buffer is used to read the 
request body and send it to the backend.

> Please help me understand how NGINX is working here in the background and
> how I can tune these settings.

Hope the above explanation helps.

-- 
Maxim Dounin
http://mdounin.ru/