Socket read errors/dropped connections during reload

[dmitry.murashenkov]

I've been writing a custom load test that performs nginx reload (with same
actual config) and noticed that sometimes a single connection get's dropped
during reload. The client was in Java, nginx on localhost in Docker under
RHEL 7.5 and about 6k req/sec.

Can somebody comment - if this is expected behavior or possibly
bug/configuration error?

I've managed to dump traffic and find that single request that failed (GET
in packet 14968 fails):
https://drive.google.com/file/d/1I-orMdoZ-zCTiCBFJszWyba2qsO-c1oz/view?usp=sharing

The connection has served several requests at this point already and it goes
like this:
>GET
<200
>GET
<200
>GET
<FIN

Next I've installed nginx on host (Ubuntu 17.10) directly and run wrk
against it, performing single manual reload in the process - got 100 socket
read errors:

$ wrk -t1 -c200 -d10s http://localhost:8080/news/index.html
Running 10s test @ http://localhost:8080/news/index.html
  1 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    33.05ms   56.88ms 230.82ms   83.12%
    Req/Sec    48.53k     4.88k   94.30k    96.00%
  482892 requests in 10.00s, 770.43MB read
  Socket errors: connect 0, read 100, write 0, timeout 0
Requests/sec:  48273.36
Transfer/sec:     77.02MB

I've also tried manually reproduce the case with no luck:

1. Launch nc -l as upstream
2. Send a single curl request to upstream via nginx and let it hang
3. Perform reload
4. Request is still hanging, old workers remain alive, new workers launched
as expected.

During all cases - no errors were logged in error.log, only notice messages
about reload.

----------------- 

Nginx version in "in docker" case:

[root at 89afa6c35cb4 /]# nginx -V
nginx version: nginx/1.14.1
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC) 
built with OpenSSL 1.0.2k-fips  26 Jan 2017
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx
--modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf
--error-log-path=/var/log/nginx/error.log
--http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid
--lock-path=/var/run/nginx.lock
--http-client-body-temp-path=/var/cache/nginx/client_temp
--http-proxy-temp-path=/var/cache/nginx/proxy_temp
--http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp
--http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp
--http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx
--with-compat --with-file-aio --with-threads --with-http_addition_module
--with-http_auth_request_module --with-http_dav_module
--with-http_flv_module --with-http_gunzip_module
--with-http_gzip_static_module --with-http_mp4_module
--with-http_random_index_module --with-http_realip_module
--with-http_secure_link_module --with-http_slice_module
--with-http_ssl_module --with-http_stub_status_module --with-http_sub_module
--with-http_v2_module --with-mail --with-mail_ssl_module --with-stream
--with-stream_realip_module --with-stream_ssl_module
--with-stream_ssl_preread_module --with-cc-opt='-O2 -g -pipe -Wall
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong
--param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC'
--with-ld-opt='-Wl,-z,relro -Wl,-z,now -pie'

[root at 89afa6c35cb4 /]# uname -a
Linux 89afa6c35cb4 4.13.0-46-generic #51-Ubuntu SMP Tue Jun 12 12:36:29 UTC
2018 x86_64 x86_64 x86_64 GNU/Linux

---------------------------

nginx version in "on host" case:

$ nginx -V
nginx version: nginx/1.12.1 (Ubuntu)
built with OpenSSL 1.0.2g  1 Mar 2016
TLS SNI support enabled
configure arguments: --with-cc-opt='-g -O2
-fdebug-prefix-map=/build/nginx-K5HqCe/nginx-1.12.1=.
-fstack-protector-strong -Wformat -Werror=format-security -fPIC -Wdate-time
-D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro
-Wl,-z,now -fPIC' --prefix=/usr/share/nginx
--conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log
--error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock
--pid-path=/run/nginx.pid --modules-path=/usr/lib/nginx/modules
--http-client-body-temp-path=/var/lib/nginx/body
--http-fastcgi-temp-path=/var/lib/nginx/fastcgi
--http-proxy-temp-path=/var/lib/nginx/proxy
--http-scgi-temp-path=/var/lib/nginx/scgi
--http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit
--with-http_ssl_module --with-http_stub_status_module
--with-http_realip_module --with-http_auth_request_module
--with-http_v2_module --with-http_dav_module --with-http_slice_module
--with-threads --with-http_addition_module --with-http_geoip_module=dynamic
--with-http_gunzip_module --with-http_gzip_static_module
--with-http_image_filter_module=dynamic --with-http_sub_module
--with-http_xslt_module=dynamic --with-stream=dynamic
--with-stream_ssl_module --with-mail=dynamic --with-mail_ssl_module

$ uname -a
Linux user-MS-7850 4.13.0-46-generic #51-Ubuntu SMP Tue Jun 12 12:36:29 UTC
2018 x86_64 x86_64 x86_64 GNU/Linux

-------------------------------------------

nginx config in all cases:

worker_processes  1;

error_log  /var/log/nginx/error.log info;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    client_max_body_size 2048m;

    access_log  /dev/null;

    sendfile        on;

    keepalive_timeout  65;

    gzip  on;
    gzip_types text/plain text/css application/json application/x-javascript
text/xml application/xml application/xml+rss text/javascript
application/javascript;

    include /etc/nginx/upstream.conf.d/*.conf;

    server {
        listen 8080 default;
        charset utf-8;

        include /etc/nginx/location.conf.d/*.conf;
    }
}

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,282126,282126#msg-282126