SSL_shutdown() failed (SSL: error:14094123:SSL routines:ssl3_read_bytes:application data after close notify) while proxying connection

[ckchauhan]

Hello All,

We have our applications running on NGINX server with CentOS configured to
use upstream servers. We have all applications working fine. Below are the
details:

-	NGINX Version 1.20.1
-	OpenSSL version 1.1.11
-	NGINX is not configured to use SSL but upstreams are, below are the
snapshot of the configuration.

cisco.upstream
upstream ciscoapi {
    server 127.0.0.1:6302; ## ${ADMIN_STREAM_PORT}
    keepalive 32;
#    server OTHERSERVER:6302 backup; ## ${ADMIN_STREAM_PORT} ${OTHER_SERVER}
${PRIVATE_ELB}
}

cisco.stream
server {
    listen 6302 ssl; ## ${ADMIN_STREAM_PORT}
    ssl_certificate /opt/lynx/cert/public.pem; ## ${INSTALL_BASE_PATH}
    ssl_certificate_key /opt/lynx/cert/private.key; ## ${INSTALL_BASE_PATH}
    proxy_pass localhost:6301; ## ${ADMIN_SVC_PORT}
}

ciscomiddleware.stream
server {
    listen 6307 ssl; ## ${MW_STREAM_PORT}
    ssl_certificate /opt/lynx/cert/public.pem; ## ${INSTALL_BASE_PATH}
    ssl_certificate_key /opt/lynx/cert/private.key; ## ${INSTALL_BASE_PATH}
    proxy_pass localhost:6306; ## ${MW_SVC_PORT}
    proxy_ssl_server_name on;
}

ciscomiddleware.upstream
upstream ciscomiddlewareapi {
    server 127.0.0.1:6307; ## ${MW_STREAM_PORT}
    keepalive 32;
#    server OTHERSERVER:6307 backup; ## ${MW_STREAM_PORT} ${OTHER_SERVER}
${PRIVATE_ELB}
}

Nginx.conf
# For more information on configuration, see:
#   * Official English Documentation: http://nginx.org/en/docs/
#   * Official Russian Documentation: http://nginx.org/ru/docs/

user nginx;
worker_processes auto;
worker_rlimit_nofile 16384;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 16384;
#    multi_accept   off;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request"
'
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main buffer=16k;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
	
    keepalive_requests 100000;
    keepalive_timeout   300;
	
#    client_body_timeout 600;
#    client_header_timeout 600;
#    server_tokens       off;

    types_hash_max_size 4096;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    proxy_buffering off;
    proxy_buffer_size 8k;
    proxy_read_timeout 300s;
    proxy_connect_timeout 75s;
    proxy_send_timeout 600s;
    send_timeout 600s;
    large_client_header_buffers 4 64k;

    client_max_body_size 128m;
    client_body_buffer_size 128m;
    client_header_buffer_size 128m;
    

    # Load modular configuration files from the /etc/nginx/conf.d
directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;

    server {
        listen       127.0.0.1:80;
#        listen       [::]:80 default_server;
        server_name  _;
        root         /usr/share/nginx/html;

        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        location / {
#	proxy_read_timeout 300;
#   	proxy_connect_timeout 75;
#   	proxy_send_timeout 600;
		proxy_http_version 1.1;
        proxy_set_header Connection "";
        }

        error_page 404 /404.html;
        location = /404.html {
        }

        error_page 500 502 503 504 /50x.html;
        location = /50x.html {
        }
    }

# Settings for a TLS enabled server.
#
#    server {
#        listen       443 ssl http2 default_server;
#        listen       [::]:443 ssl http2 default_server;
#        server_name  _;
#        root         /usr/share/nginx/html;
#
#        ssl_certificate "/etc/pki/nginx/server.crt";
#        ssl_certificate_key "/etc/pki/nginx/private/server.key";
#        ssl_session_cache shared:SSL:1m;
#        ssl_session_timeout  10m;
#        ssl_ciphers HIGH:!aNULL:!MD5;
#        ssl_prefer_server_ciphers on;
#
#        # Load configuration files for the default server block.
#        include /etc/nginx/default.d/*.conf;
#
#        location / {
#        }
#
#        error_page 404 /404.html;
#        location = /404.html {
#        }
#
#        error_page 500 502 503 504 /50x.html;
#        location = /50x.html {
#        }
#    }

}

Recently, we have been performing Load Test on this using JMETER as load
generation tool. Mostly it runs as expected however we get the below error
on random basis.

In logs we have observed that,

-	It is showing 502 Bad Gateway error [SSL Shutdown].
-	“SSL_shutdown() failed (SSL: error:14094123:SSL
routines:ssl3_read_bytes:application data after close notify) while proxying
connection, client: 127.0.0.1, server: 0.0.0.0:6307, upstream:
"127.0.0.1:6306", bytes from/to client:0/0, bytes from/to upstream:0/0”.
-	The error occurs when max response time breaches 120 sec.

We have tried to identify the cause by, 
-	Following the nginx, github and stackoveflow.
-	Made changes multiple times to proxy read timeout, upgrading OPENSSL
version and other tweaks.

But still we are not able to get to the root cause of the issue or fix. We
have been struggling since more than four weeks now.

Can you help us please?

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,293640,293640#msg-293640