<div dir="ltr"><div dir="ltr">Francis,<div><br></div><div>WIth your help and suggestions of starting with the simplest and slowly adding steps, it is working.</div><div>Here is the working config below. I started by getting the web server part only working well. You suggested not to use scheme=http</div><div>for redirect, so I'm listening on 80 and redirecting. I put each function into a distinct server which seems to have helped.</div><div>The HTTPS server is serving the requests, and serving our app as the index page.</div><div><br></div><div><br></div><div><br></div><div>Once that was working, I tried using port 25565 within NGINX and also the app inside docker. </div><div>NGINX refused saying there was already a listener on that port. So I had to go back to putting the client on a different port than the server.</div><div>I took out the map block and the upstream block, as it seems fine to put the commands inline in the server.<br></div><div>The two things that seemed to clinch it were adding ssl to the listener line in the 25566 server, and adding the timeout.</div><div>I turns out, our app server negotiates everything on 25565, and does not use 80 or 443 to start the WSS negotiation.</div><div>Once the timeout was added, it worked. There seemed to be a lot of discussion around WSS timeouts on the web, so I tried it.</div><div>Frankly I'm not sure why it is required. I don't know what I would have done if the app server needed to start the WS negotiation on 443 and then switch to </div><div>25565.</div><div><br></div><div>I suspect there may be some additional ssl settings I will need.</div><div><br></div><div>Many thanks again for your time and wisdom and sharing it.</div><div><br></div><div><br></div><div>server_tokens off;<br>ssl_certificate /etc/letsencrypt/live/esports1.totalvu.live/fullchain.pem; # managed by Certbot<br>ssl_certificate_key /etc/letsencrypt/live/esports1.totalvu.live/privkey.pem; # managed by Certbot<br>include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot<br>ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot<br>ssl_protocols TLSv1.1 TLSv1.2;<br><br>log_format withport '$remote_addr - $remote_user [$time_local] '<br> '"$request" $status $body_bytes_sent '<br> '"$http_referer" "$http_user_agent" ' "srv p " $server_port " rem p " $remote_port;<br>access_log /var/log/nginx/access.log withport;<br><br># Port 80 web server<br>server {<br># first redirect to https<br> listen 80;<br> return 301 https://$host$request_uri;<br>}<br><br># Port 443 web and wss server<br>server {<br> root /var/www/html;<br> index cc.html;<br><br># Web server for https<br> listen [::]:443 ssl ipv6only=on; # managed by Certbot<br> listen 443 ssl; # managed by Certbot<br>}<br><br>server {<br> listen 25566 ssl;<br><br> location / {<br> proxy_http_version 1.1;<br> proxy_set_header Upgrade $http_upgrade;<br> proxy_set_header Connection "upgrade";<br> proxy_pass <a href="http://172.31.24.191:25565">http://172.31.24.191:25565</a>;<br> proxy_read_timeout 86400;<br> }<br>}<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Sep 15, 2022 at 7:30 AM Francis Daly <<a href="mailto:francis@daoine.org">francis@daoine.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Mon, Sep 12, 2022 at 05:46:21PM -0700, Michael Williams wrote:<br>
<br>
Hi there,<br>
<br>
> Wow thank you. This really helps all the guidance and instruction. I really<br>
> appreciate your time.<br>
<br>
No worries.<br>
<br>
> One thing to clarify, is that if I turn off NGINX, the client page works<br>
> fine and connects to the app server inside the docker OK.<br>
<br>
I confess that I am confused as to what your current architecture is.<br>
<br>
Can you describe it? Along the lines of:<br>
<br>
Without nginx involved, we have (http service) running on (docker<br>
ip:port) and when we tell the client to access (http:// docker ip:port)<br>
everything works, including the websocket thing.<br>
<br>
With nginx involved to test reverse-proxying http, the docker side is<br>
identical, but we tell the client to access (http:// nginx ip:port)<br>
and everything works? Or not everything works?<br>
<br>
With nginx involved to test reverse-proxying https, the docker side is<br>
identical, and we tell the client to access (https:// nginx ip:port),<br>
and some things work?<br>
<br>
With that information, it might be clear to someone where the first<br>
problem appears.<br>
<br>
In this configuration:<br>
<br>
> server {<br>
> index index.html index.htm;<br>
> listen [::]:443 ssl ipv6only=on; # managed by Certbot<br>
> listen 443 ssl; # managed by Certbot<br>
> listen 25566 ssl;<br>
<br>
nginx is listening for https on two ports.<br>
<br>
What test are you running? Which port are you using?<br>
<br>
> location @wss {<br>
<br>
> proxy_http_version 1.1;<br>
> proxy_set_header Upgrade $http_upgrade;<br>
> proxy_set_header Connection $connection_upgrade;<br>
<br>
> proxy_pass <a href="http://172.31.24.191:25565" rel="noreferrer" target="_blank">http://172.31.24.191:25565</a>;<br>
<br>
nginx is talking to this port without https.<br>
<br>
What works here / what fails here? What do the logs say?<br>
<br>
> My idea was to try changing our client webpage to access a different port #<br>
> than the one our app server in the docker is listening to.<br>
<br>
I'm afraid I am not sure what that means. I thought the client webpage<br>
was accessing nginx on port 443 and the backend / upstream http server<br>
was listening on the high port?<br>
<br>
Maybe I am getting confused among multiple tests that you are running.<br>
<br>
> With that change<br>
> I see from WIreshark on my local that the WSS connection seems to go<br>
> through OK with NGINX:<br>
> <br>
> [image: Screen Shot 2022-09-12 at 5.29.50 PM.png]<br>
<br>
I'm seeing a picture; but I'm not seeing anything that obviously says<br>
that a WSS connection is working anywhere.<br>
<br>
I'm seeing a TLS connection between the client and nginx that is cleanly<br>
closed after a fraction of a second. I see nothing that suggests that<br>
nginx is doing a proxy_pass to the upstream server. (But maybe that was<br>
excluded from the tcpdump?)<br>
<br>
> Our app server shows that the connection to the server also starts but then<br>
> disconnect it:<br>
> (22:36:59) Disconnected <client IP address> (unknown opcode 22)<br>
<br>
With nginx involved, the app server should never see the client IP<br>
address directly; it should only see connections from nginx. (It might<br>
see the client IP listed in the http headers.)<br>
<br>
> My question here, does NGINX negotiate the entire handshake for HTTPS to<br>
> WSS upgrade itself, without forwarding the same pages to our app server ?<br>
> Is there a way to forward those pages to the app server also ? I think our<br>
> app server may insist on negotiating a ws:// connection itself, but not a<br>
> wss:// connection.<br>
<br>
As I understand it: the client makes a TLS connection to nginx,<br>
and sends a http request inside that TLS connection (== a https<br>
request). Separately, nginx makes a http connection to the upstream<br>
server, and (through config) passes along the Upgrade-and-friends headers<br>
that the client sent to nginx, requesting that this connection switch to<br>
a websocket connection. And after that works, nginx effectively becomes a<br>
"blind tunnel" for the connection contents, passing unencrypted things<br>
on the nginx-to-upstream side and encrypted things on the nginx-to-client<br>
side, and generally not caring about what is inside.<br>
<br>
<br>
If things are still not working as wanted, I suggest simplifying things<br>
as much as possible.<br>
<br>
Make the nginx config be not much more than what step 6 on<br>
<a href="https://www.nginx.com/blog/websocket-nginx/" rel="noreferrer" target="_blank">https://www.nginx.com/blog/websocket-nginx/</a> shows, and include enough<br>
information in any report of a test, so that someone else will be able<br>
to repeat the test on their system to see what happens there.<br>
<br>
Good luck with it,<br>
<br>
f<br>
-- <br>
Francis Daly <a href="mailto:francis@daoine.org" target="_blank">francis@daoine.org</a><br>
_______________________________________________<br>
nginx mailing list -- <a href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a><br>
To unsubscribe send an email to <a href="mailto:nginx-leave@nginx.org" target="_blank">nginx-leave@nginx.org</a><br>
</blockquote></div><img src="https://t.sidekickopen04.com/s3t/o/5/f18dQhb0S7n28bNTLVW7zKHFs1jkhdLW1_k-L-1qZM43N2TrPlL5Z0xmW4cH2872z6fYSf7_cSGl02?si=8000000020094730&pi=a3943f82-b43d-4a9f-937a-1479c088f7be" alt="" style="display:none!important" height="1" width="1"><div></div></div>