Extremely slow file (~5MB) upload via POST
Andrei
lagged at gmail.com
Sun Nov 10 22:58:36 UTC 2019
Bit off-topic, but, if you really want to improve the performance and ditch
an upstream service which just takes in file uploads, you can do it
directly in nginx with some Lua. For example
https://www.yanxurui.cc/posts/server/2017-03-21-NGINX-as-a-file-server/. I
used this method for large sites (20k online) and it worked way better than
having to pass the file to a backend for saving.
There are many other articles on "nginx file server" or "nginx image
server" with details on how to also process images. Hope this helps.
On Sun, Nov 10, 2019, 20:03 frank.muller <nginx-forum at forum.nginx.org>
wrote:
> Hi everyone
>
> I'm new here, and i've searched if the problem appeared before but couldn't
> find anything useful.
>
> [DESCRIPTION] I've an upstream backend service behind Niginx (1.16.1,
> openssl-1.1.1) which allow people upload files from their browser. The
> files
> are simply stored on disk. Nothing else is performed on them.
>
> [SYSTEM CONFIG]
> . Linux 4.15.0 Ubuntu 18.04 LTS SMP x86_64
> . RAM: 32GB
> . CPU: 8-Cores Intel(R) Xeon(R) CPU E3-1270 v6 @ 3.80GHz
> . DISK: 1TB SSD
> . NETWORK CARD: 10Gbps
> . System is never under load. We usually upload 10 files per hour at max.
>
> [DATA CONFIG]
> . File size is between 5MB to 20MB
>
> [NGINX CONFIG]
>
> We are running Nginx 1.16.1 with TLSv1.3 support (built on openssl 1.1.1).
>
> -8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8
>
> $ cat /etc/nginx/nginx.conf
> worker_processes auto;
> worker_rlimit_nofile 100000;
> pid /run/nginx.pid;
>
> error_log off; #/var/log/nginx/error.log info;
>
> events {
> worker_connections 655350;
> multi_accept on;
> use epoll;
> }
>
> http {
> include mime.types;
> default_type application/octet-stream;
>
> server_tokens off;
>
> keepalive_timeout 3600;
>
> access_log off; #/var/log/nginx/access.log;
> sendfile on;
> tcp_nopush on;
> tcp_nodelay on;
>
> types_hash_max_size 2048;
>
> open_file_cache max=10000 inactive=10m;
> open_file_cache_valid 1h;
> open_file_cache_min_uses 1;
> open_file_cache_errors on;
> include /etc/nginx/conf.d/*.conf;
>
> }
>
> -8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8
>
> $ cat /etc/nginx/conf.d/uploader.conf
> server {
> listen 443 ssl;
>
> server_name BACKEND_HOST_NAME:
>
> ssl_certificate /etc/nginx/certs/bundle.pem;
> ssl_certificate_key /etc/nginx/certs/key.pem;
> ssl_dhparam /etc/nginx/certs/dh.pem; ## 2048-bit
>
> ssl_protocols TLSv1.2 TLSv1.3;
>
> ssl_prefer_server_ciphers on;
> ssl_ciphers
>
> ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384;
>
> client_max_body_size 30m;
>
> location / {
> proxy_pass http://127.0.0.1:7777;
> }
> }
>
> -8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8
>
> [PROBLEM] A 5MB file takes almost 30 seconds to upload via Nginx.
> When uploading it directly to the upstream backend, it takes ~400 millisec
> at max.
>
> Running strace, we've got this:
> % time seconds usecs/call calls errors syscall
> ------ ----------- ----------- --------- --------- ----------------
> 47.96 0.041738 11 3887 1489 read
> 21.73 0.018909 13 1509 epoll_wait
> 17.95 0.015622 22 708 writev
> 10.62 0.009241 13 712 write
> 0.47 0.000407 19 21 21 connect
> 0.39 0.000338 17 20 close
> 0.21 0.000180 8 22 epoll_ctl
> 0.20 0.000173 8 21 socket
> 0.13 0.000110 110 1 accept4
> 0.11 0.000095 5 21 getsockopt
> 0.10 0.000091 4 21 recvfrom
> 0.07 0.000060 3 21 ioctl
> 0.04 0.000037 12 3 brk
> 0.03 0.000023 8 3 setsockopt
> ------ ----------- ----------- --------- --------- ----------------
> 100.00 0.087024 6970 1510 total
>
> A lot of errors in "read" calls: 1489 errors. They all correspond to
> (thanks
> again to strace):
>
> 22807 read(3,
>
> "\26\3\1\2\0\1\0\1\374\3\3\304\353\3\333\314\0\36\223\244z\246\322n\375\205\360\322\35\237_\240"...,
> 16709) = 517
> 22807 read(3, 0x559de2a23f03, 16709) = -1 EAGAIN (Resource temporarily
> unavailable)
> 22807 read(3,
>
> "\24\3\3\0\1\1\26\3\3\2\0\1\0\1\374\3\3\304\353\3\333\314\0\36\223\244z\246\322n\375\205"...,
> 16709) = 523
> 22807 read(3, 0x559de2a23f03, 16709) = -1 EAGAIN (Resource temporarily
> unavailable)
> 22807 read(3, "\27\3\3\0E\271m'\306\262\26X\36J\25lC/\202_7\241\32\342XN
> \357\303%\264\0"..., 16709) = 74
> 22807 read(3,
>
> "\27\3\3\0\245\240\204\304KJ\260\207\301\232\3147\217\357I$\243\266p+*\343L\335\6v\276\323"...,
> 16709) = 478
> 22807 read(3, 0x559de2a1e9a3, 16709) = -1 EAGAIN (Resource temporarily
> unavailable)
> 22807 read(3,
>
> "\27\3\3\0\32`\324\324\237\v\266n\300x\24\277\357z\374)\365\260F\235\24\346#A%\300\376",
> 16709) = 31
> 22807 read(3, 0x559de2a1e9a3, 16709) = -1 EAGAIN (Resource temporarily
> unavailable)
> 22807 read(3,
>
> "\27\3\3\0\177\310*W\352\265\230\357\325\177\302\275\357=\246`\246^\372\214T\206\264b\352;\273z"...,
> 16709) = 814
> 22807 read(3, 0x559de2a1e9a3, 16709) = -1 EAGAIN (Resource temporarily
> unavailable)
> 22807 read(3,
>
> "\27\3\3\0Y\330\276PNY\220\245\254E\0066\2016\355\334\237Yo\2510\253\320+\26z\342\275"...,
> 16709) = 644
> 22807 read(3, 0x559de2a229e3, 16709) = -1 EAGAIN (Resource temporarily
> unavailable)
> 22807 read(3, "\27\3\3\0Z
> \237j\230\f\331\222\246\325\1\272Y]\252\255%\31\257L\25\10\226\267
> \253\353\367"..., 16709) = 285
> 22807 read(3, 0x559de2a1e9a3, 16709) = -1 EAGAIN (Resource temporarily
> unavailable)
> 22807 read(3,
>
> "\27\3\3\0\212\216j6\256\370\367\310\366Hjs\275r\276>\217\216\374\377a\375\363\4\2yr\23"...,
> 16709) = 176
> 22807 read(3, 0x559de2a1e9a3, 16709) = -1 EAGAIN (Resource temporarily
> unavailable)
> 22807 read(3,
>
> "\27\3\3\0\227K2\345P\200Ls\234\10\230f\362\221\273\270V\2371X\261|\245\315\240B\177\224"...,
> 16709) = 1717
> 22807 read(3, 0x559de2a1e9a3, 16709) = -1 EAGAIN (Resource temporarily
> unavailable)
> 22807 read(3,
>
> "\27\3\3>\232\344\316\245i\375hM\362\376\frr\340\21umx&\3311\373}\35\4\3069`"...,
> 16709) = 4380
> 22807 read(3, 0x559de2a1fabf, 11651) = -1 EAGAIN (Resource temporarily
> unavailable)
>
>
> We tried to tune our Nginx's config, but the result is always the same:
> 22807 read(3, 0x559de2a1fabf, 11651) = -1 EAGAIN (Resource temporarily
> unavailable)
>
>
> Help appreciated
>
> /F.
>
> Posted at Nginx Forum:
> https://forum.nginx.org/read.php?2,286161,286161#msg-286161
>
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20191111/4f378d61/attachment.htm>
More information about the nginx
mailing list