help with https to http and WSS to WS reverse proxy conf
Michael Williams
michael.glenn.williams at totalvu.tv
Tue Sep 13 00:46:21 UTC 2022
Francis!
Wow thank you. This really helps all the guidance and instruction. I really
appreciate your time.
One thing to clarify, is that if I turn off NGINX, the client page works
fine and connects to the app server inside the docker OK.
I've changed the conf.d to the following, but still fail to get my app's
server to work.
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
upstream to-websocket {
server 172.31.24.191:25565;
}
server_tokens off;
server {
# first redirect to https
if ($scheme = "http") {
return 301 https://$host$request_uri;
}
}
server {
server_name esports1.totalvu.live;
root /var/www/html;
index index.html index.htm;
# Proxy our outside https to local http
listen [::]:443 ssl ipv6only=on; # managed by Certbot
listen 443 ssl; # managed by Certbot
listen 25566 ssl;
ssl_certificate
/etc/letsencrypt/live/esports1.totalvu.live/fullchain.pem; # managed by
Certbot
ssl_certificate_key
/etc/letsencrypt/live/esports1.totalvu.live/privkey.pem; # managed by
Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
location / {
try_files $uri /static/ @wss;
}
location @wss {
error_log /var/log/nginx/wsserror.log;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header X-Real-IP $remote_addr;
# proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host esports1.totalvu.live;
proxy_set_header Referer https://esports1.totalvu.live;
proxy_set_header Referrer https://esports1.totalvu.live;
proxy_pass http://172.31.24.191:25565;
# proxy_pass http://to-websocket;
}
location /static/ {
try_files $uri =404;
}
}
My idea was to try changing our client webpage to access a different port #
than the one our app server in the docker is listening to. With that change
I see from WIreshark on my local that the WSS connection seems to go
through OK with NGINX:
[image: Screen Shot 2022-09-12 at 5.29.50 PM.png]
Our app server shows that the connection to the server also starts but then
disconnect it:
(22:36:59) Disconnected <client IP address> (unknown opcode 22)
I confirmed that using local host or 127.0.0.1 was not where our app was
listening as you said, so I changed to the local IP.
My question here, does NGINX negotiate the entire handshake for HTTPS to
WSS upgrade itself, without forwarding the same pages to our app server ?
Is there a way to forward those pages to the app server also ? I think our
app server may insist on negotiating a ws:// connection itself, but not a
wss:// connection.
Again Francis Many thanks!
On Mon, Sep 12, 2022 at 1:37 PM Francis Daly <francis at daoine.org> wrote:
> On Sun, Sep 11, 2022 at 11:53:43AM -0700, Michael Williams wrote:
>
> Hi there,
>
> > Francis thanks very much for taking the time to look at this.
> > Based on your suggestion, I commented out these 3 lines and it got rid
> of
> > the looping. I thought the same process that wants the WS feed also
> looked
> > for inbound on port 80, but that is not the case after all.
> >
> > location @ {
> > proxy_set_header X-Real-IP $remote_addr;
> > proxy_set_header Host esports1.totalvu.live;
> > proxy_set_header Referer https://esports1.totalvu.live;
> > proxy_set_header Referrer https://esports1.totalvu.live;
> > # proxy_set_header X-Forwarded-Proto $scheme;
> > # proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
> > # proxy_pass http://localhost:80;
> > }
>
> That location{} now has no "*_pass" directive, so if it is used, it will
> end up trying to serve a file from the filesystem.
>
> If that is what you want it to do, it's fine. If not, you will probably
> want to decide what you want your nginx to do with a request, and then
> configure nginx to match.
>
> > I thought that localhost was a different route to the Debian kernel, than
> > the network interface... so listening to localhost:80 wouldn't hear
> traffic
> > on the network interface port 80 and vice versa. Is that wrong?
>
> "It depends".
>
> In this context, where you have nginx listening on port 80 on the
> "everything" address, localhost counts as part of everything.
>
> > Unfortunately, WSS inbound proxied to WS on localhost isn't working. The
> > process that is listening is running inside a docker.
>
> Once you introduce docker, you are introducing the docker networking
> system.
>
> In docker networking, it is simplest to imagine that there is no
> localhost. (More strictly: there is not exactly one localhost; so you are
> better off keeping a very clear idea of what IP address is being used,
> from the perspective of what system.)
>
> > When the webpage tries to connect to NGINX to start a WSS from a testing
> > site like https://websocketking.com/ going to the host without the port,
> > just to test conf.d :
> >
> > wss://myFQDN
> >
> > the access log shows:
> >
> > myIPAddr - - [11/Sep/2022:18:42:41 +0000] "GET / HTTP/1.1" 502 552 "-"
> > "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36
> (KHTML,
> > like Gecko) Chrome/105.0.0.0 Safari/537.36" "-"
> >
> >
> > Should it say HTTPS here ? When I try with the port:
> >
> > wss://myFQDN:25565
> >
> > *the request hangs in Pending state forever.*
>
> I am confused as to what exactly you are doing.
>
> The overview is: somewhere, you are running the eventual "upstream"
> websocket service. That is listening on one specific IP:port. You will
> want to configure your nginx to proxy_pass to that IP:port when nginx
> receives the websocket connection-upgrade thing.
>
> If that upstream service is running inside docker, then the IP:port that
> you will connect to from outside that docker container, is whatever port
> is exposed by docker -- by the EXPOSE Dockerfile line, or by the -p or
> -P arguments to "docker run".
>
> > map $http_upgrade $connection_upgrade {
> > default upgrade;
> > '' close;
> > }
>
> So "$connection_upgrade" is either the word "upgrade" or the word "close".
>
> But you don't use "$connection_upgrade" anywhere in the config that you
> show.
>
> > upstream to-websocket {
> > server localhost:25565;
> > }
>
> That is referring to nginx's idea of localhost, which may or may not
> correspond to your in-docker service.
>
> Can you access that IP:port from the machine that nginx is running on? If
> not, change it to be whatever IP:port you can use the talk to your
> upstream websocket service.
>
> > server {
> >
> > # first redirect to https
> > if ($scheme = "http") {
> > return 301 https://$host$request_uri;
> > }
>
> This entire server{} block is equivalent to
>
> server { return 301 https://$host$request_uri; }
>
> because of the directive default values. If you don't want to listen
> for http, just don't have a server with (effectively) "listen 80;",
> (which is what this one has).
>
> > server {
> >
> > root /var/www/html;
> > index index.html index.htm;
> > server_name myFQDN;
> >
> > # Proxy our outside https to local http
> > listen [::]:443 ssl ipv6only=on; # managed by Certbot
> > listen 443 ssl; # managed by Certbot
>
> <snip>
>
> > location / {
> > try_files /nonexistent @$http_upgrade;
> > }
>
> That will do an internal redirect to a location that can be chosen by
> the client. You hope the http "Upgrade" header will either be empty,
> or have the value "websocket". If it is, then one of the following
> location{}s will be used; otherwise there will probably be an error
> returned to the client.
>
> > location @websocket {
> > proxy_http_version 1.1;
> > proxy_set_header Upgrade $http_upgrade;
> > proxy_set_header Connection $connection_upgrade;
> > proxy_set_header X-Real-IP $remote_addr;
> > proxy_set_header X-Forwarded-Proto $scheme;
> > proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
> > proxy_set_header Host myFQDN;
> > proxy_set_header Referer https://myFQDN;
> > proxy_set_header Referrer https://myFQDN;
> > # proxy_pass http://localhost:25565;
> > proxy_pass http://to-websocket;
>
> From below, your websocket service appears to be listening on
> ip-172-31-24-191.:25565. You'll want to invite nginx to talk to that
> IP:port, not localhost.
>
> > location @ {
>
> And this is what should be used if the incoming request has no "Upgrade"
> header. This entire block is equivalent to "location @ { }"
>
> > Here is the listener process on netstat:
> >
> > netstat -a -o | grep 255
> >
> > tcp 0 0 ip-172-31-24-191.:25565 0.0.0.0:*
> LISTEN
> > off (0.00/0/0)
>
> If you can access that IP:port from the nginx server to talk to the
> websocket service, that's what you should configure nginx to try to
> talk to.
>
> > Here is the interface being used:
>
> In this case: nginx is talking to an IP. It does not care what the
> physical interface is. (iptables and the like do care; but that part
> all looks good from here.)
>
> > Here are the iptables stats:
>
> If these rules block nginx from talking to the IP:port and getting the
> response, that will want fixing. Otherwise, it's good.
>
> > iptables -L -n -v
>
> These appear to say "accept almost everything; nothing has been dropped",
> so these rules are presumably not blocking nginx.
>
> Good luck with it,
>
> f
> --
> Francis Daly francis at daoine.org
> _______________________________________________
> nginx mailing list -- nginx at nginx.org
> To unsubscribe send an email to nginx-leave at nginx.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20220912/b5579abf/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2022-09-12 at 5.29.50 PM.png
Type: image/png
Size: 199223 bytes
Desc: not available
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20220912/b5579abf/attachment.png>
More information about the nginx
mailing list