help with https to http and WSS to WS reverse proxy conf
Michael Williams
michael.glenn.williams at totalvu.tv
Thu Sep 15 17:00:31 UTC 2022
Francis,
WIth your help and suggestions of starting with the simplest and slowly
adding steps, it is working.
Here is the working config below. I started by getting the web server part
only working well. You suggested not to use scheme=http
for redirect, so I'm listening on 80 and redirecting. I put each function
into a distinct server which seems to have helped.
The HTTPS server is serving the requests, and serving our app as the index
page.
Once that was working, I tried using port 25565 within NGINX and also the
app inside docker.
NGINX refused saying there was already a listener on that port. So I had
to go back to putting the client on a different port than the server.
I took out the map block and the upstream block, as it seems fine to put
the commands inline in the server.
The two things that seemed to clinch it were adding ssl to the listener
line in the 25566 server, and adding the timeout.
I turns out, our app server negotiates everything on 25565, and does not
use 80 or 443 to start the WSS negotiation.
Once the timeout was added, it worked. There seemed to be a lot of
discussion around WSS timeouts on the web, so I tried it.
Frankly I'm not sure why it is required. I don't know what I would have
done if the app server needed to start the WS negotiation on 443 and then
switch to
25565.
I suspect there may be some additional ssl settings I will need.
Many thanks again for your time and wisdom and sharing it.
server_tokens off;
ssl_certificate /etc/letsencrypt/live/esports1.totalvu.live/fullchain.pem;
# managed by Certbot
ssl_certificate_key
/etc/letsencrypt/live/esports1.totalvu.live/privkey.pem; # managed by
Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
ssl_protocols TLSv1.1 TLSv1.2;
log_format withport '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent" ' "srv p "
$server_port " rem p " $remote_port;
access_log /var/log/nginx/access.log withport;
# Port 80 web server
server {
# first redirect to https
listen 80;
return 301 https://$host$request_uri;
}
# Port 443 web and wss server
server {
root /var/www/html;
index cc.html;
# Web server for https
listen [::]:443 ssl ipv6only=on; # managed by Certbot
listen 443 ssl; # managed by Certbot
}
server {
listen 25566 ssl;
location / {
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_pass http://172.31.24.191:25565;
proxy_read_timeout 86400;
}
}
On Thu, Sep 15, 2022 at 7:30 AM Francis Daly <francis at daoine.org> wrote:
> On Mon, Sep 12, 2022 at 05:46:21PM -0700, Michael Williams wrote:
>
> Hi there,
>
> > Wow thank you. This really helps all the guidance and instruction. I
> really
> > appreciate your time.
>
> No worries.
>
> > One thing to clarify, is that if I turn off NGINX, the client page works
> > fine and connects to the app server inside the docker OK.
>
> I confess that I am confused as to what your current architecture is.
>
> Can you describe it? Along the lines of:
>
> Without nginx involved, we have (http service) running on (docker
> ip:port) and when we tell the client to access (http:// docker ip:port)
> everything works, including the websocket thing.
>
> With nginx involved to test reverse-proxying http, the docker side is
> identical, but we tell the client to access (http:// nginx ip:port)
> and everything works? Or not everything works?
>
> With nginx involved to test reverse-proxying https, the docker side is
> identical, and we tell the client to access (https:// nginx ip:port),
> and some things work?
>
> With that information, it might be clear to someone where the first
> problem appears.
>
> In this configuration:
>
> > server {
> > index index.html index.htm;
> > listen [::]:443 ssl ipv6only=on; # managed by Certbot
> > listen 443 ssl; # managed by Certbot
> > listen 25566 ssl;
>
> nginx is listening for https on two ports.
>
> What test are you running? Which port are you using?
>
> > location @wss {
>
> > proxy_http_version 1.1;
> > proxy_set_header Upgrade $http_upgrade;
> > proxy_set_header Connection $connection_upgrade;
>
> > proxy_pass http://172.31.24.191:25565;
>
> nginx is talking to this port without https.
>
> What works here / what fails here? What do the logs say?
>
> > My idea was to try changing our client webpage to access a different
> port #
> > than the one our app server in the docker is listening to.
>
> I'm afraid I am not sure what that means. I thought the client webpage
> was accessing nginx on port 443 and the backend / upstream http server
> was listening on the high port?
>
> Maybe I am getting confused among multiple tests that you are running.
>
> > With that change
> > I see from WIreshark on my local that the WSS connection seems to go
> > through OK with NGINX:
> >
> > [image: Screen Shot 2022-09-12 at 5.29.50 PM.png]
>
> I'm seeing a picture; but I'm not seeing anything that obviously says
> that a WSS connection is working anywhere.
>
> I'm seeing a TLS connection between the client and nginx that is cleanly
> closed after a fraction of a second. I see nothing that suggests that
> nginx is doing a proxy_pass to the upstream server. (But maybe that was
> excluded from the tcpdump?)
>
> > Our app server shows that the connection to the server also starts but
> then
> > disconnect it:
> > (22:36:59) Disconnected <client IP address> (unknown opcode 22)
>
> With nginx involved, the app server should never see the client IP
> address directly; it should only see connections from nginx. (It might
> see the client IP listed in the http headers.)
>
> > My question here, does NGINX negotiate the entire handshake for HTTPS to
> > WSS upgrade itself, without forwarding the same pages to our app server ?
> > Is there a way to forward those pages to the app server also ? I think
> our
> > app server may insist on negotiating a ws:// connection itself, but not a
> > wss:// connection.
>
> As I understand it: the client makes a TLS connection to nginx,
> and sends a http request inside that TLS connection (== a https
> request). Separately, nginx makes a http connection to the upstream
> server, and (through config) passes along the Upgrade-and-friends headers
> that the client sent to nginx, requesting that this connection switch to
> a websocket connection. And after that works, nginx effectively becomes a
> "blind tunnel" for the connection contents, passing unencrypted things
> on the nginx-to-upstream side and encrypted things on the nginx-to-client
> side, and generally not caring about what is inside.
>
>
> If things are still not working as wanted, I suggest simplifying things
> as much as possible.
>
> Make the nginx config be not much more than what step 6 on
> https://www.nginx.com/blog/websocket-nginx/ shows, and include enough
> information in any report of a test, so that someone else will be able
> to repeat the test on their system to see what happens there.
>
> Good luck with it,
>
> f
> --
> Francis Daly francis at daoine.org
> _______________________________________________
> nginx mailing list -- nginx at nginx.org
> To unsubscribe send an email to nginx-leave at nginx.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20220915/7765d846/attachment.htm>
More information about the nginx
mailing list