<div dir="ltr">Please refer to the attachments for reference.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Feb 19, 2024 at 4:24 PM Kin Seng <<a href="mailto:ckinseng@gmail.com">ckinseng@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">My current nginx setup always kill the TCP connection after 5 minutes of inactivity, i.e no transaction.</span><div>[From wireshark, nginx send RST to upstream server and then send FIN,ACK to downstream client]<br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">I have this setup which requires TLS1.2 connection connecting from my internal network [client application] to public network [server]. It only use TCP ports (not http/https) and establish with a server located at public network. The client application does not support TLS1.2 connection hence the introduction of nginx proxy/reverse proxy for TLS wrapping purpose. You may refer below :</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"> <br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">                       Internal Network                                      | INTERNET/Public</span><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium"><br></span><div><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">[Client Application] <-----> [NGINX Reverse Proxy] <--- | ---> [Public Server]</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">                  <Non TLS TCP Traffic>                        <TLS 1.2></span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">- using stream module</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">- no error shown in nginx error log</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">- access log showing TCP 200 Status but the session only last 300s everytime. [Recorded in the access_log]</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">Below is my nginx configuration</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium"># more nginx.conf</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">user nginx;</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">worker_processes auto;</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">error_log /var/log/nginx/error.log;</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">pid /run/nginx.pid;</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium"># Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">include /usr/share/nginx/modules/*.conf;</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">events {</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">worker_connections 2048;</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">}</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">stream {</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">resolver 127.0.0.1;</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">include /etc/nginx/conf.d/*.conf;</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">log_format basic '$remote_addr [$time_local] '</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">'$protocol $status $bytes_sent $bytes_received '</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">'$session_time $upstream_addr'</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">'"$upstream_bytes_sent" "$upstream_bytes_received" "$upstream_connect_time"';</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">access_log /var/log/nginx/stream.access.log basic;</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">error_log log_file;</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">error_log /var/log/nginx/error_log;</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">server {</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">listen 35012;</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">proxy_pass X.X.X.X:35012;</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">proxy_timeout 86400s;</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">proxy_connect_timeout 1200s;</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">proxy_socket_keepalive on;</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">ssl_session_cache shared:SSL:5m;</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">ssl_session_timeout 30m;</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium"># For securing TCP Traffic with upstream servers.</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">proxy_ssl on;</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">proxy_ssl_certificate /etc/ssl/certs/backend.crt;</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">proxy_ssl_certificate_key /etc/ssl/certs/backend.key;</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">proxy_ssl_protocols TLSv1.2;</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">proxy_ssl_ciphers HIGH:!aNULL:!MD5;</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium"># proxy_ssl_trusted_certificate /etc/ssl/certs/trusted_ca_cert.crt;</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium"># proxy_ssl_verify on;</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">proxy_ssl_verify_depth 2;</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">#To have NGINX proxy previously negotiated connection parameters and use a so-called abbreviated handshake - Fast</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">proxy_ssl_session_reuse on;</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">}</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">}</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">After capturing the tcp packet and check via wireshark, I found out that the nginx is sending out the RST to the public server and then send FIN/ACK (refer attached pcap picture) to client application.</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">I have tried to enable keepalive related parameters as per the nginx config above and also check on the OS's TCP tunable and i could not find any related settings which make NGINX to kill the TCP connection.</span><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><br style="clear:both;color:rgb(0,0,0);font-family:Arial;font-size:medium"><span style="color:rgb(0,0,0);font-family:Arial;font-size:medium">Anyone encountering the same issues?</span></div></div></div>
</blockquote></div>