No request body when using fastcgi and chunked transfer-encoding
Tim Siebels
tim.siebels at iserv.eu
Mon Sep 13 09:32:19 UTC 2021
Hi everyone,
We are experiencing an issue when using `fastcgi_request_buffering off;`
in combination with chunked transfer-encoding. The application will not
receive any body. Depending on the system, we *sometimes* receive a
body. Enabling `fastcgi_request_buffering` fixes the issue. However, we
would like not to enable this for every request.
We are using Debian Bullseye [0]. Additionally, I was able to reproduce
this issue using a self compiled 1.21.3 [1].
We are using a very simple configuration [2], only disabling
`fastcgi_request_buffering`. The application is using PHP over fastcgi [3].
Furthermore, we log `$request_body` in a custom log file.
Whenever this logfile contains the expected request body, the
application receives the expected body. As far as I understand, this
cannot work without buffering. I assume this to be an in-memory buffer.
This goes in line with the body not being passed, when we increase the
size. The implementation of `ngx_http_read_client_request_body` does
have an optimization, if the entire body fits into header_in.
`sleep`ing between chunks also removes the possibility that the body is
passed to the application.
We could not reproduce that the body is *sometimes* passed in a
self-contained docker container or using a self-compiled version of nginx.
These never work with fastcgi_request_buffering off. At least not, if
the body is large enough.
Apache had a bug that had a similar effect [4,5,6].
Our understanding from these bugreports is, that the fastcgi protocol
expects a defined content-length to be able to read the request body.
Our conclusion is, that nginx is supposed to buffer all request with a
chunked transfer-encoding, when using fastcgi. Regardless of the
fastcgi_request_buffering option. A comment on a older, unrelated,
bugreport for nginx confirms this [7].
Therefore, we expect this to be a supported use case.
There are some bug reports out there experiencing similar issues [8,9].
We tried to set `client_body_in_file_only` to `on` to be able to see the
buffer files. However, these are never created with buffering off.
error.log is empty.
Can anyone help us how to debug this further?
Thanks,
Tim
[0] nginx -V for the debian bullseye version
nginx version: nginx/1.18.0
built with OpenSSL 1.1.1k 25 Mar 2021
TLS SNI support enabled
configure arguments: --with-cc-opt='-g -O2
-ffile-prefix-map=/build/nginx-q9LD4J/nginx-1.18.0=.
-fstack-protector-strong -Wformat -Werror=format-security -fPIC
-Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-z,relro -Wl,-z,now
-fPIC' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf
--http-log-path=/var/log/nginx/access.log
--error-log-path=/var/log/nginx/error.log
--lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid
--modules-path=/usr/lib/nginx/modules
--http-client-body-temp-path=/var/lib/nginx/body
--http-fastcgi-temp-path=/var/lib/nginx/fastcgi
--http-proxy-temp-path=/var/lib/nginx/proxy
--http-scgi-temp-path=/var/lib/nginx/scgi
--http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-compat --with-debug
--with-pcre-jit --with-http_ssl_module --with-http_stub_status_module
--with-http_realip_module --with-http_auth_request_module
--with-http_v2_module --with-http_dav_module --with-http_slice_module
--with-threads --with-http_addition_module --with-http_flv_module
--with-http_geoip_module=dynamic --with-http_gunzip_module
--with-http_gzip_static_module --with-http_image_filter_module=dynamic
--with-http_mp4_module --with-http_perl_module=dynamic
--with-http_random_index_module --with-http_secure_link_module
--with-http_sub_module --with-http_xslt_module=dynamic
--with-mail=dynamic --with-mail_ssl_module --with-stream=dynamic
--with-stream_geoip_module=dynamic --with-stream_ssl_module
--with-stream_ssl_preread_module
--add-dynamic-module=/build/nginx-q9LD4J/nginx-1.18.0/debian/modules/http-headers-more-filter
--add-dynamic-module=/build/nginx-q9LD4J/nginx-1.18.0/debian/modules/http-auth-pam
--add-dynamic-module=/build/nginx-q9LD4J/nginx-1.18.0/debian/modules/http-cache-purge
--add-dynamic-module=/build/nginx-q9LD4J/nginx-1.18.0/debian/modules/http-dav-ext
--add-dynamic-module=/build/nginx-q9LD4J/nginx-1.18.0/debian/modules/http-ndk
--add-dynamic-module=/build/nginx-q9LD4J/nginx-1.18.0/debian/modules/http-echo
--add-dynamic-module=/build/nginx-q9LD4J/nginx-1.18.0/debian/modules/http-fancyindex
--add-dynamic-module=/build/nginx-q9LD4J/nginx-1.18.0/debian/modules/http-geoip2
--add-dynamic-module=/build/nginx-q9LD4J/nginx-1.18.0/debian/modules/nchan
--add-dynamic-module=/build/nginx-q9LD4J/nginx-1.18.0/debian/modules/http-lua
--add-dynamic-module=/build/nginx-q9LD4J/nginx-1.18.0/debian/modules/rtmp
--add-dynamic-module=/build/nginx-q9LD4J/nginx-1.18.0/debian/modules/http-uploadprogress
--add-dynamic-module=/build/nginx-q9LD4J/nginx-1.18.0/debian/modules/http-upstream-fair
--add-dynamic-module=/build/nginx-q9LD4J/nginx-1.18.0/debian/modules/http-subs-filter
[1] nginx -V for the self compiled version
ginx version: nginx/1.21.3
built by gcc 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)
built with OpenSSL 1.1.1f 31 Mar 2020
TLS SNI support enabled
configure arguments: --with-cc-opt='-g -O2
-ffile-prefix-map=/build/nginx-q9LD4J/nginx-1.18.0=.
-fstack-protector-strong -Wformat -Werror=format-security -fPIC
-Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-z,relro -Wl,-z,now
-fPIC' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf
--http-log-path=/var/log/nginx/access.log
--error-log-path=/var/log/nginx/error.log
--lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid
--modules-path=/usr/lib/nginx/modules
--http-client-body-temp-path=/var/lib/nginx/body
--http-fastcgi-temp-path=/var/lib/nginx/fastcgi
--http-proxy-temp-path=/var/lib/nginx/proxy
--http-scgi-temp-path=/var/lib/nginx/scgi
--http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-compat --with-debug
--with-pcre-jit --with-http_ssl_module --with-http_stub_status_module
--with-http_realip_module --with-http_auth_request_module
--with-http_v2_module --with-http_dav_module --with-http_slice_module
--with-threads --with-http_addition_module --with-http_flv_module
--with-http_geoip_module=dynamic --with-http_gunzip_module
--with-http_gzip_static_module --with-http_image_filter_module=dynamic
--with-http_mp4_module --with-http_perl_module=dynamic
--with-http_random_index_module --with-http_secure_link_module
--with-http_sub_module --with-http_xslt_module=dynamic
--with-mail=dynamic --with-mail_ssl_module --with-stream=dynamic
--with-stream_geoip_module=dynamic --with-stream_ssl_module
--with-stream_ssl_preread_module --with-debug
[2]
user nginx;
worker_processes auto;
pid /run/nginx.pid;
events {
worker_connections 768;
}
http {
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
fastcgi_request_buffering off;
include /etc/nginx/sites-enabled/*;
}
[3]
log_format postdata '"$request" $status $request_body ($request_length)';
server {
listen *:982;
location /iserv/helloworld {
root /usr/share/iserv/helloworld/public;
access_log /var/log/nginx/postdata.log postdata;
fastcgi_param QUERY_STRING $query_string;
fastcgi_param REQUEST_METHOD $request_method;
fastcgi_param CONTENT_TYPE $content_type;
fastcgi_param CONTENT_LENGTH $content_length;
fastcgi_param SCRIPT_NAME $fastcgi_script_name;
fastcgi_param REQUEST_URI $request_uri;
fastcgi_param DOCUMENT_URI $document_uri;
fastcgi_param DOCUMENT_ROOT $document_root;
fastcgi_param SCRIPT_FILENAME
$document_root/index.php$fastcgi_script_name;
fastcgi_pass unix:/run/php/php-fpm.iserv-helloworld.sock;
}
}
[4] https://bz.apache.org/bugzilla/show_bug.cgi?id=53332
[5] https://bz.apache.org/bugzilla/show_bug.cgi?id=57087
[6] https://bugs.php.net/bug.php?id=60826
[7] https://trac.nginx.org/nginx/ticket/1344
[8] https://trac.cyberduck.io/wiki/help/en/howto/mount/issues/fastcgi
[9] https://github.com/nextcloud/server/issues/7995
More information about the nginx
mailing list