disable request body buffering for file upload

[Maxim Dounin]

Hello!

On Wed, Jul 19, 2017 at 10:33:01AM -0400, garyc wrote:

> Hello, hopefully someone can tell me if i am attempting the impossible. 
> 
> We use nginx and php5-fpm to handle the upload of test files used by our web
> app that may be several gigabytes in size. 
> 
> To achieve this we use a location block for the upload url and define the
> fastcgi_pass directive to provide the location of the fastcgi socket i.e.
> 
> location /api/filter/analysis/upload {
>    fastcgi_pass unix:/tmp/php5-fpm.sock;
>    include fastcgi_params;
>    fastcgi_params SCRIPTFILENAME $document_root/PHP/uploadFile.php;
> }
> 
> The above configuration functions however we have observed that nginx will
> cache the entire request body before calling the php script which
> effectively copies the uploaded temporary file to a different location. 
> 
> Under certain circumstances this can be a major issue for us as our host
> environment is within an instrument that is designed to capture data
> continuously and will fill the hard drive close to 100% capacity before
> stopping. If a user instigates the upload of a 1gb file that is cached first
> by nginx and then copied elsewhere by the php script we can find ourselves
> out of disk space at which point nginx (and other processes) understandably
> stops functioning leaving the instrument in a state that is difficult to
> recover from.
> 
> If we can prevent nginx from caching the entire request body and have it
> pass it straight to our php script we should be able to take preventative
> steps in the script and reject the upload attempt if disk space is too low.
> 
> I have looked at the directives 'proxy_request_buffering off;' and
> 'fastcgi_request_buffering off;' however i have not been successful in
> stopping the initial nginx upload before our php script is called.

With "fastcgi_request_buffering off;" nginx will send the request 
body to the FastCGI application immediately, without trying to 
buffer it anywhere.

An example configuration:

    location / {
        fastcgi_pass 127.0.0.1:10002;
        fastcgi_request_buffering off;
    }

It is up to your FastCGI application to handle this though, 
and PHP as well as PHP-FPM may impose additional limitations 
and/or require additional configuration for this to work.

> If this is the wrong approach can anyone suggest a different one? basically
> we need to intercept the file upload request and reject it if we are too low
> on disk space the current setup effectively uploads the file before we can
> perform such a check.

Doing such checks in nginx (for example, you can use embedded perl 
to do such checks) and/or keeping client_max_body_size low enough might 
be a better solution.  Nevertheless, fastcgi_request_buffering is 
expected to work as well.

-- 
Maxim Dounin
http://nginx.org/