Tuning client request buffering in ngx_http_proxy_module

[bengalih]

I had/have an issue where I am proxying from NGINX to a backend WebDAV
server.
My upload speeds were slow and had long pauses in them including very long
pauses at the end (5 minutes or more on uploads around 1GB).

Via packet captures I found that the NGINX server was not transmitting data
synchronously to the backend WebDAV server, but was clearly doing buffering
despite the fact I had set "proxy_buffering off".
Looking at the documentation led me to "proxy_request_buffering off" which
seems to have solved my problem and seems to immediately send the entire
WebDAV client request directly to the backend server in a synchronous
manner.

I did however want to experiment with proxy_request_buffering not disabled
and see if setting smaller buffers to the backend would perhaps not result
in such log asynchronous delays.

With "proxy_request_buffering on" (default) I have also set the
"client_body_temp_path /tmp/cache" and set "client_body_in_file_only on".
In this configuration I can see the client request get placed into the
/tmp/cache directory, so I know my directive and paths are working.

However, if I set "client_body_in_file_only off" (default) no file gets
created at all despite the fact that my client PUT request over WebDAV is
undoubtedly larger than any buffer settings.

By doing a "df" on my NGINX box I can see that the space on my drive it
being eaten up by the same amount equivalent to my upload (i.e. if I upload
a 500 MB file I can see my free space decrease by 500 MB).  I cannot however
see anything in /tmp/cache and have no idea where these files are being
placed.

I also don't know how/why NGINX is caching/buffering the entire file and
what is controlling how it is sending this to the backend server.  I have
tried playing with the "client_body_buffer_size" but it does not appear to
have any effect in how the data gets buffered or the size of the file.

Please help me understand how NGINX is working here in the background and
how I can tune these settings.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,293558,293558#msg-293558