<div dir="auto">Bit off-topic, but, if you really want to improve the performance and ditch an upstream service which just takes in file uploads, you can do it directly in nginx with some Lua. For example <a href="https://www.yanxurui.cc/posts/server/2017-03-21-NGINX-as-a-file-server/">https://www.yanxurui.cc/posts/server/2017-03-21-NGINX-as-a-file-server/</a>. I used this method for large sites (20k online) and it worked way better than having to pass the file to a backend for saving.<div dir="auto"><br></div><div dir="auto">There are many other articles on "nginx file server" or "nginx image server" with details on how to also process images. Hope this helps.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Nov 10, 2019, 20:03 frank.muller <<a href="mailto:nginx-forum@forum.nginx.org">nginx-forum@forum.nginx.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi everyone<br>
<br>
I'm new here, and i've searched if the problem appeared before but couldn't<br>
find anything useful.<br>
<br>
[DESCRIPTION] I've an upstream backend service behind Niginx (1.16.1,<br>
openssl-1.1.1) which allow people upload files from their browser. The files<br>
are simply stored on disk. Nothing else is performed on them.<br>
<br>
[SYSTEM CONFIG]<br>
. Linux 4.15.0 Ubuntu 18.04 LTS SMP x86_64<br>
. RAM: 32GB<br>
. CPU: 8-Cores Intel(R) Xeon(R) CPU E3-1270 v6 @ 3.80GHz<br>
. DISK: 1TB SSD<br>
. NETWORK CARD: 10Gbps<br>
. System is never under load. We usually upload 10 files per hour at max.<br>
<br>
[DATA CONFIG]<br>
. File size is between 5MB to 20MB<br>
<br>
[NGINX CONFIG]<br>
<br>
We are running Nginx 1.16.1 with TLSv1.3 support (built on openssl 1.1.1).<br>
<br>
-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8<br>
<br>
$ cat /etc/nginx/nginx.conf<br>
worker_processes auto;<br>
worker_rlimit_nofile 100000;<br>
pid /run/nginx.pid;<br>
<br>
error_log off; #/var/log/nginx/error.log info;<br>
<br>
events {<br>
worker_connections 655350;<br>
multi_accept on;<br>
use epoll;<br>
}<br>
<br>
http {<br>
include mime.types;<br>
default_type application/octet-stream;<br>
<br>
server_tokens off;<br>
<br>
keepalive_timeout 3600;<br>
<br>
access_log off; #/var/log/nginx/access.log;<br>
sendfile on;<br>
tcp_nopush on;<br>
tcp_nodelay on;<br>
<br>
types_hash_max_size 2048;<br>
<br>
open_file_cache max=10000 inactive=10m;<br>
open_file_cache_valid 1h;<br>
open_file_cache_min_uses 1;<br>
open_file_cache_errors on;<br>
include /etc/nginx/conf.d/*.conf;<br>
<br>
}<br>
<br>
-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8<br>
<br>
$ cat /etc/nginx/conf.d/uploader.conf<br>
server {<br>
listen 443 ssl;<br>
<br>
server_name BACKEND_HOST_NAME:<br>
<br>
ssl_certificate /etc/nginx/certs/bundle.pem;<br>
ssl_certificate_key /etc/nginx/certs/key.pem;<br>
ssl_dhparam /etc/nginx/certs/dh.pem; ## 2048-bit<br>
<br>
ssl_protocols TLSv1.2 TLSv1.3;<br>
<br>
ssl_prefer_server_ciphers on;<br>
ssl_ciphers<br>
ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384;<br>
<br>
client_max_body_size 30m;<br>
<br>
location / {<br>
proxy_pass <a href="http://127.0.0.1:7777" rel="noreferrer noreferrer" target="_blank">http://127.0.0.1:7777</a>;<br>
}<br>
}<br>
<br>
-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8<br>
<br>
[PROBLEM] A 5MB file takes almost 30 seconds to upload via Nginx.<br>
When uploading it directly to the upstream backend, it takes ~400 millisec<br>
at max.<br>
<br>
Running strace, we've got this:<br>
% time seconds usecs/call calls errors syscall<br>
------ ----------- ----------- --------- --------- ----------------<br>
47.96 0.041738 11 3887 1489 read<br>
21.73 0.018909 13 1509 epoll_wait<br>
17.95 0.015622 22 708 writev<br>
10.62 0.009241 13 712 write<br>
0.47 0.000407 19 21 21 connect<br>
0.39 0.000338 17 20 close<br>
0.21 0.000180 8 22 epoll_ctl<br>
0.20 0.000173 8 21 socket<br>
0.13 0.000110 110 1 accept4<br>
0.11 0.000095 5 21 getsockopt<br>
0.10 0.000091 4 21 recvfrom<br>
0.07 0.000060 3 21 ioctl<br>
0.04 0.000037 12 3 brk<br>
0.03 0.000023 8 3 setsockopt<br>
------ ----------- ----------- --------- --------- ----------------<br>
100.00 0.087024 6970 1510 total<br>
<br>
A lot of errors in "read" calls: 1489 errors. They all correspond to (thanks<br>
again to strace):<br>
<br>
22807 read(3,<br>
"\26\3\1\2\0\1\0\1\374\3\3\304\353\3\333\314\0\36\223\244z\246\322n\375\205\360\322\35\237_\240"...,<br>
16709) = 517<br>
22807 read(3, 0x559de2a23f03, 16709) = -1 EAGAIN (Resource temporarily<br>
unavailable)<br>
22807 read(3,<br>
"\24\3\3\0\1\1\26\3\3\2\0\1\0\1\374\3\3\304\353\3\333\314\0\36\223\244z\246\322n\375\205"...,<br>
16709) = 523<br>
22807 read(3, 0x559de2a23f03, 16709) = -1 EAGAIN (Resource temporarily<br>
unavailable)<br>
22807 read(3, "\27\3\3\0E\271m'\306\262\26X\36J\25lC/\202_7\241\32\342XN<br>
\357\303%\264\0"..., 16709) = 74<br>
22807 read(3,<br>
"\27\3\3\0\245\240\204\304KJ\260\207\301\232\3147\217\357I$\243\266p+*\343L\335\6v\276\323"...,<br>
16709) = 478<br>
22807 read(3, 0x559de2a1e9a3, 16709) = -1 EAGAIN (Resource temporarily<br>
unavailable)<br>
22807 read(3,<br>
"\27\3\3\0\32`\324\324\237\v\266n\300x\24\277\357z\374)\365\260F\235\24\346#A%\300\376",<br>
16709) = 31<br>
22807 read(3, 0x559de2a1e9a3, 16709) = -1 EAGAIN (Resource temporarily<br>
unavailable)<br>
22807 read(3,<br>
"\27\3\3\0\177\310*W\352\265\230\357\325\177\302\275\357=\246`\246^\372\214T\206\264b\352;\273z"...,<br>
16709) = 814<br>
22807 read(3, 0x559de2a1e9a3, 16709) = -1 EAGAIN (Resource temporarily<br>
unavailable)<br>
22807 read(3,<br>
"\27\3\3\0Y\330\276PNY\220\245\254E\0066\2016\355\334\237Yo\2510\253\320+\26z\342\275"...,<br>
16709) = 644<br>
22807 read(3, 0x559de2a229e3, 16709) = -1 EAGAIN (Resource temporarily<br>
unavailable)<br>
22807 read(3, "\27\3\3\0Z<br>
\237j\230\f\331\222\246\325\1\272Y]\252\255%\31\257L\25\10\226\267<br>
\253\353\367"..., 16709) = 285<br>
22807 read(3, 0x559de2a1e9a3, 16709) = -1 EAGAIN (Resource temporarily<br>
unavailable)<br>
22807 read(3,<br>
"\27\3\3\0\212\216j6\256\370\367\310\366Hjs\275r\276>\217\216\374\377a\375\363\4\2yr\23"...,<br>
16709) = 176<br>
22807 read(3, 0x559de2a1e9a3, 16709) = -1 EAGAIN (Resource temporarily<br>
unavailable)<br>
22807 read(3,<br>
"\27\3\3\0\227K2\345P\200Ls\234\10\230f\362\221\273\270V\2371X\261|\245\315\240B\177\224"...,<br>
16709) = 1717<br>
22807 read(3, 0x559de2a1e9a3, 16709) = -1 EAGAIN (Resource temporarily<br>
unavailable)<br>
22807 read(3,<br>
"\27\3\3>\232\344\316\245i\375hM\362\376\frr\340\21umx&\3311\373}\35\4\3069`"...,<br>
16709) = 4380<br>
22807 read(3, 0x559de2a1fabf, 11651) = -1 EAGAIN (Resource temporarily<br>
unavailable)<br>
<br>
<br>
We tried to tune our Nginx's config, but the result is always the same:<br>
22807 read(3, 0x559de2a1fabf, 11651) = -1 EAGAIN (Resource temporarily<br>
unavailable)<br>
<br>
<br>
Help appreciated<br>
<br>
/F.<br>
<br>
Posted at Nginx Forum: <a href="https://forum.nginx.org/read.php?2,286161,286161#msg-286161" rel="noreferrer noreferrer" target="_blank">https://forum.nginx.org/read.php?2,286161,286161#msg-286161</a><br>
<br>
_______________________________________________<br>
nginx mailing list<br>
<a href="mailto:nginx@nginx.org" target="_blank" rel="noreferrer">nginx@nginx.org</a><br>
<a href="http://mailman.nginx.org/mailman/listinfo/nginx" rel="noreferrer noreferrer" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a><br>
</blockquote></div>