No request body when using fastcgi and chunked transfer-encoding

[Sergey Kandaurov]

> On 13 Sep 2021, at 12:32, Tim Siebels <tim.siebels at iserv.eu> wrote:
> 
> Hi everyone,
> 
> We are experiencing an issue when using `fastcgi_request_buffering off;` in combination with chunked transfer-encoding. The application will not receive any body. Depending on the system, we *sometimes* receive a body. Enabling `fastcgi_request_buffering` fixes the issue. However, we would like not to enable this for every request.
> 
> We are using Debian Bullseye [0]. Additionally, I was able to reproduce this issue using a self compiled 1.21.3 [1].
> 
> We are using a very simple configuration [2], only disabling `fastcgi_request_buffering`. The application is using PHP over fastcgi [3].
> 
> Furthermore, we log `$request_body` in a custom log file.
> Whenever this logfile contains the expected request body, the application receives the expected body. As far as I understand, this cannot work without buffering. I assume this to be an in-memory buffer. This goes in line with the body not being passed, when we increase the size. The implementation of `ngx_http_read_client_request_body` does have an optimization, if the entire body fits into header_in.
> `sleep`ing between chunks also removes the possibility that the body is passed to the application.
> We could not reproduce that the body is *sometimes* passed in a self-contained docker container or using a self-compiled version of nginx.
> These never work with fastcgi_request_buffering off. At least not, if the body is large enough.


> 
> Apache had a bug that had a similar effect [4,5,6].
> Our understanding from these bugreports is, that the fastcgi protocol expects a defined content-length to be able to read the request body.

FastCGI is essentially a protocol with builtin chunked encoding,
see http://www.mit.edu/~yandros/doc/specs/fcgi-spec.html#S3.3.
Hence, the Content-Length header is rather complementary (like in HTTP/2),
e.g. it allows to communicate the message length knowledge in advance.

So, when nginx doesn't buffer request body, and it's not wholly present
in the request body memory buffers at the time of forming request headers,
in this case the length cannot be calculated for chunked body requests.
See also the description of $content_length: nginx.org/r/$content_length

> Our conclusion is, that nginx is supposed to buffer all request with a chunked transfer-encoding, when using fastcgi. Regardless of the fastcgi_request_buffering option. A comment on a older, unrelated, bugreport for nginx confirms this [7].
> Therefore, we expect this to be a supported use case.

When a client uses chunked transfer encoding, and nginx is configured
to pass requests to fastcgi with disabled fastcgi_request_buffering,
the intent is to pass request body chunks to backend (with conversion
to fastcgi records) as soon as they are received from a client.

> 
> There are some bug reports out there experiencing similar issues [8,9].
> 

Looks like they are suffering from the same sort of a problem.

> We tried to set `client_body_in_file_only` to `on` to be able to see the buffer files. However, these are never created with buffering off.
> 

It is disabled in the unbuffered mode.

> error.log is empty.
> 
> Can anyone help us how to debug this further?

For a start, you may want to examine what is actually passed to backend.

> 
> Thanks,
> Tim
> 
> [..]
> [2]
> user nginx;
> worker_processes auto;
> pid /run/nginx.pid;
> 
> events {
>    worker_connections 768;
> }
> 
> http {
>    access_log /var/log/nginx/access.log;
>    error_log /var/log/nginx/error.log;
> 
>    fastcgi_request_buffering off;
> 
>    include /etc/nginx/sites-enabled/*;
> }
> 
> [3]
> log_format postdata '"$request" $status $request_body ($request_length)';
> 
> server {
>    listen *:982;
> 
>    location /iserv/helloworld {
>        root /usr/share/iserv/helloworld/public;
> 
>        access_log /var/log/nginx/postdata.log postdata;
> 
>        fastcgi_param  QUERY_STRING       $query_string;
>        fastcgi_param  REQUEST_METHOD     $request_method;
>        fastcgi_param  CONTENT_TYPE       $content_type;
>        fastcgi_param  CONTENT_LENGTH     $content_length;

Looks like the backend is expecting to receive something
in the CONTENT_LENGTH header to read the request body.

The best solution is to teach your FastCGI backend how to receive
the request body without CONTENT_LENGTH.

> 
>        fastcgi_param  SCRIPT_NAME        $fastcgi_script_name;
>        fastcgi_param  REQUEST_URI        $request_uri;
>        fastcgi_param  DOCUMENT_URI       $document_uri;
>        fastcgi_param  DOCUMENT_ROOT      $document_root;
> 
>        fastcgi_param  SCRIPT_FILENAME $document_root/index.php$fastcgi_script_name;
> 
>        fastcgi_pass unix:/run/php/php-fpm.iserv-helloworld.sock;
>    }
> 
> }
> 
> [4] https://bz.apache.org/bugzilla/show_bug.cgi?id=53332
> [5] https://bz.apache.org/bugzilla/show_bug.cgi?id=57087
> [6] https://bugs.php.net/bug.php?id=60826
> [7] https://trac.nginx.org/nginx/ticket/1344
> [8] https://trac.cyberduck.io/wiki/help/en/howto/mount/issues/fastcgi
> [9] https://github.com/nextcloud/server/issues/7995
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx

-- 
Sergey Kandaurov