Extremely slow file (~5MB) upload via POST

[Francis Daly]

On Sun, Nov 10, 2019 at 01:03:20PM -0500, frank.muller wrote:

Hi there,

I don't have an answer for you, but there are some things you could
perhaps try, if you are happy to keep investigating.

> [DESCRIPTION] I've an upstream backend service behind Niginx (1.16.1,
> openssl-1.1.1) which allow people upload files from their browser. The files
> are simply stored on disk. Nothing else is performed on them.

The sequence is: client writes to nginx; nginx writes to upstream.

Can you see: is the extra slowness in the first part, or the second?

The usual first place to look is in the log files.

> error_log  off; #/var/log/nginx/error.log  info;

You can probably look in the file /usr/local/nginx/off to see what nginx
says is happening; but you might want to increase the log level to see
more details.

>       access_log    off; #/var/log/nginx/access.log;

You don't have an access log to look in. That probably does not matter
much here.

>       sendfile      on;
>       tcp_nopush    on;
>       tcp_nodelay   on;
> 
>       types_hash_max_size 2048;
> 
>       open_file_cache          max=10000 inactive=10m;
>       open_file_cache_valid    1h;
>       open_file_cache_min_uses 1;
>       open_file_cache_errors   on;

I think that those directives should not affect this test, either way.

>         listen  443 ssl;
>         location / {
>            proxy_pass http://127.0.0.1:7777;
>         }

> [PROBLEM] A 5MB file takes almost 30 seconds to upload via Nginx.
> When uploading it directly to the upstream backend, it takes ~400 millisec
> at max.

That does sound unnecessarily slow. The (presumed) ssl/no-ssl difference
should not account for that much overhead.

> Running strace, we've got this:
> % time     seconds  usecs/call     calls    errors syscall
> ------ ----------- ----------- --------- --------- ----------------
>  47.96    0.041738          11      3887      1489 read
>  21.73    0.018909          13      1509           epoll_wait
>  17.95    0.015622          22       708           writev
>  10.62    0.009241          13       712           write
>   0.47    0.000407          19        21        21 connect

I don't know the details, but what is nginx trying to connect() to that
is erroring every time? Is that likely relevant to the problem?

> A lot of errors in "read" calls: 1489 errors. They all correspond to (thanks
> again to strace):

It is possible that the nginx debug log might have more nginx-related
details than the bare strace.

Good luck with it,

	f
-- 
Francis Daly        francis at daoine.org