Extremely slow file (~5MB) upload via POST

Andrei lagged at gmail.com
Sun Nov 10 22:58:36 UTC 2019


Bit off-topic, but, if you really want to improve the performance and ditch
an upstream service which just takes in file uploads, you can do it
directly in nginx with some Lua. For example
https://www.yanxurui.cc/posts/server/2017-03-21-NGINX-as-a-file-server/. I
used this method for large sites (20k online) and it worked way better than
having to pass the file to a backend for saving.

There are many other articles on "nginx file server" or "nginx image
server" with details on how to also process images. Hope this helps.

On Sun, Nov 10, 2019, 20:03 frank.muller <nginx-forum at forum.nginx.org>
wrote:

> Hi everyone
>
> I'm new here, and i've searched if the problem appeared before but couldn't
> find anything useful.
>
> [DESCRIPTION] I've an upstream backend service behind Niginx (1.16.1,
> openssl-1.1.1) which allow people upload files from their browser. The
> files
> are simply stored on disk. Nothing else is performed on them.
>
> [SYSTEM CONFIG]
> . Linux 4.15.0 Ubuntu 18.04 LTS SMP x86_64
> . RAM: 32GB
> . CPU: 8-Cores Intel(R) Xeon(R) CPU E3-1270 v6 @ 3.80GHz
> . DISK: 1TB SSD
> . NETWORK CARD: 10Gbps
> . System is never under load. We usually upload 10 files per hour at max.
>
> [DATA CONFIG]
> . File size is between 5MB to 20MB
>
> [NGINX CONFIG]
>
> We are running Nginx 1.16.1 with TLSv1.3 support (built on openssl 1.1.1).
>
> -8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8
>
> $ cat /etc/nginx/nginx.conf
> worker_processes auto;
> worker_rlimit_nofile 100000;
> pid /run/nginx.pid;
>
> error_log  off; #/var/log/nginx/error.log  info;
>
> events {
>         worker_connections 655350;
>         multi_accept           on;
>         use                 epoll;
> }
>
> http {
>       include       mime.types;
>       default_type  application/octet-stream;
>
>       server_tokens off;
>
>       keepalive_timeout 3600;
>
>       access_log    off; #/var/log/nginx/access.log;
>       sendfile      on;
>       tcp_nopush    on;
>       tcp_nodelay   on;
>
>       types_hash_max_size 2048;
>
>       open_file_cache          max=10000 inactive=10m;
>       open_file_cache_valid    1h;
>       open_file_cache_min_uses 1;
>       open_file_cache_errors   on;
>       include /etc/nginx/conf.d/*.conf;
>
> }
>
> -8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8
>
> $ cat /etc/nginx/conf.d/uploader.conf
> server {
>         listen  443 ssl;
>
>         server_name  BACKEND_HOST_NAME:
>
>         ssl_certificate        /etc/nginx/certs/bundle.pem;
>         ssl_certificate_key /etc/nginx/certs/key.pem;
>         ssl_dhparam         /etc/nginx/certs/dh.pem; ## 2048-bit
>
>         ssl_protocols       TLSv1.2 TLSv1.3;
>
>         ssl_prefer_server_ciphers on;
>         ssl_ciphers
>
> ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384;
>
>         client_max_body_size 30m;
>
>         location / {
>            proxy_pass http://127.0.0.1:7777;
>         }
> }
>
> -8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8
>
> [PROBLEM] A 5MB file takes almost 30 seconds to upload via Nginx.
> When uploading it directly to the upstream backend, it takes ~400 millisec
> at max.
>
> Running strace, we've got this:
> % time     seconds  usecs/call     calls    errors syscall
> ------ ----------- ----------- --------- --------- ----------------
>  47.96    0.041738          11      3887      1489 read
>  21.73    0.018909          13      1509           epoll_wait
>  17.95    0.015622          22       708           writev
>  10.62    0.009241          13       712           write
>   0.47    0.000407          19        21        21 connect
>   0.39    0.000338          17        20           close
>   0.21    0.000180           8        22           epoll_ctl
>   0.20    0.000173           8        21           socket
>   0.13    0.000110         110         1           accept4
>   0.11    0.000095           5        21           getsockopt
>   0.10    0.000091           4        21           recvfrom
>   0.07    0.000060           3        21           ioctl
>   0.04    0.000037          12         3           brk
>   0.03    0.000023           8         3           setsockopt
> ------ ----------- ----------- --------- --------- ----------------
> 100.00    0.087024                  6970      1510 total
>
> A lot of errors in "read" calls: 1489 errors. They all correspond to
> (thanks
> again to strace):
>
> 22807 read(3,
>
> "\26\3\1\2\0\1\0\1\374\3\3\304\353\3\333\314\0\36\223\244z\246\322n\375\205\360\322\35\237_\240"...,
> 16709) = 517
> 22807 read(3, 0x559de2a23f03, 16709)    = -1 EAGAIN (Resource temporarily
> unavailable)
> 22807 read(3,
>
> "\24\3\3\0\1\1\26\3\3\2\0\1\0\1\374\3\3\304\353\3\333\314\0\36\223\244z\246\322n\375\205"...,
> 16709) = 523
> 22807 read(3, 0x559de2a23f03, 16709)    = -1 EAGAIN (Resource temporarily
> unavailable)
> 22807 read(3, "\27\3\3\0E\271m'\306\262\26X\36J\25lC/\202_7\241\32\342XN
> \357\303%\264\0"..., 16709) = 74
> 22807 read(3,
>
> "\27\3\3\0\245\240\204\304KJ\260\207\301\232\3147\217\357I$\243\266p+*\343L\335\6v\276\323"...,
> 16709) = 478
> 22807 read(3, 0x559de2a1e9a3, 16709)    = -1 EAGAIN (Resource temporarily
> unavailable)
> 22807 read(3,
>
> "\27\3\3\0\32`\324\324\237\v\266n\300x\24\277\357z\374)\365\260F\235\24\346#A%\300\376",
> 16709) = 31
> 22807 read(3, 0x559de2a1e9a3, 16709)    = -1 EAGAIN (Resource temporarily
> unavailable)
> 22807 read(3,
>
> "\27\3\3\0\177\310*W\352\265\230\357\325\177\302\275\357=\246`\246^\372\214T\206\264b\352;\273z"...,
> 16709) = 814
> 22807 read(3, 0x559de2a1e9a3, 16709)    = -1 EAGAIN (Resource temporarily
> unavailable)
> 22807 read(3,
>
> "\27\3\3\0Y\330\276PNY\220\245\254E\0066\2016\355\334\237Yo\2510\253\320+\26z\342\275"...,
> 16709) = 644
> 22807 read(3, 0x559de2a229e3, 16709)    = -1 EAGAIN (Resource temporarily
> unavailable)
> 22807 read(3, "\27\3\3\0Z
> \237j\230\f\331\222\246\325\1\272Y]\252\255%\31\257L\25\10\226\267
> \253\353\367"..., 16709) = 285
> 22807 read(3, 0x559de2a1e9a3, 16709)    = -1 EAGAIN (Resource temporarily
> unavailable)
> 22807 read(3,
>
> "\27\3\3\0\212\216j6\256\370\367\310\366Hjs\275r\276>\217\216\374\377a\375\363\4\2yr\23"...,
> 16709) = 176
> 22807 read(3, 0x559de2a1e9a3, 16709)    = -1 EAGAIN (Resource temporarily
> unavailable)
> 22807 read(3,
>
> "\27\3\3\0\227K2\345P\200Ls\234\10\230f\362\221\273\270V\2371X\261|\245\315\240B\177\224"...,
> 16709) = 1717
> 22807 read(3, 0x559de2a1e9a3, 16709)    = -1 EAGAIN (Resource temporarily
> unavailable)
> 22807 read(3,
>
> "\27\3\3>\232\344\316\245i\375hM\362\376\frr\340\21umx&\3311\373}\35\4\3069`"...,
> 16709) = 4380
> 22807 read(3, 0x559de2a1fabf, 11651)    = -1 EAGAIN (Resource temporarily
> unavailable)
>
>
> We tried to tune our Nginx's config, but the result is always the same:
> 22807 read(3, 0x559de2a1fabf, 11651)    = -1 EAGAIN (Resource temporarily
> unavailable)
>
>
> Help appreciated
>
> /F.
>
> Posted at Nginx Forum:
> https://forum.nginx.org/read.php?2,286161,286161#msg-286161
>
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20191111/4f378d61/attachment.htm>


More information about the nginx mailing list