help with https to http and WSS to WS reverse proxy conf

[Francis Daly]

On Sun, Sep 11, 2022 at 11:53:43AM -0700, Michael Williams wrote:

Hi there,

> Francis thanks very much for taking the time to look at this.
> Based on your suggestion, I commented out these 3 lines  and it got rid of
> the looping. I thought the same process that wants the WS feed also looked
> for inbound on port 80, but that is not the case after all.
> 
>     location @ {
>         proxy_set_header X-Real-IP $remote_addr;
>         proxy_set_header Host esports1.totalvu.live;
>         proxy_set_header Referer https://esports1.totalvu.live;
>         proxy_set_header Referrer https://esports1.totalvu.live;
> #       proxy_set_header X-Forwarded-Proto $scheme;
> #       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
> #       proxy_pass http://localhost:80;
>     }

That location{} now has no "*_pass" directive, so if it is used, it will
end up trying to serve a file from the filesystem.

If that is what you want it to do, it's fine. If not, you will probably
want to decide what you want your nginx to do with a request, and then
configure nginx to match.

> I thought that localhost was a different route to the Debian kernel, than
> the network interface... so listening to localhost:80 wouldn't hear traffic
> on the network interface port 80 and vice versa. Is that wrong?

"It depends".

In this context, where you have nginx listening on port 80 on the
"everything" address, localhost counts as part of everything.

> Unfortunately, WSS inbound proxied to WS  on localhost isn't working. The
> process that is listening is running inside a docker.

Once you introduce docker, you are introducing the docker networking system.

In docker networking, it is simplest to imagine that there is no
localhost. (More strictly: there is not exactly one localhost; so you are
better off keeping a very clear idea of what IP address is being used,
from the perspective of what system.)

> When the webpage tries to connect to NGINX to start a WSS from a testing
> site like https://websocketking.com/ going to the host without the port,
> just to test conf.d :
> 
> wss://myFQDN
> 
> the access log shows:
> 
> myIPAddr - - [11/Sep/2022:18:42:41 +0000] "GET / HTTP/1.1" 502 552 "-"
> "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML,
> like Gecko) Chrome/105.0.0.0 Safari/537.36" "-"
> 
> 
> Should it say HTTPS here ? When I try with the port:
> 
> wss://myFQDN:25565
> 
> *the request hangs in Pending state forever.*

I am confused as to what exactly you are doing.

The overview is: somewhere, you are running the eventual "upstream"
websocket service. That is listening on one specific IP:port. You will
want to configure your nginx to proxy_pass to that IP:port when nginx
receives the websocket connection-upgrade thing.

If that upstream service is running inside docker, then the IP:port that
you will connect to from outside that docker container, is whatever port
is exposed by docker -- by the EXPOSE Dockerfile line, or by the -p or
-P arguments to "docker run".

> map $http_upgrade $connection_upgrade {
>     default upgrade;
>     '' close;
> }

So "$connection_upgrade" is either the word "upgrade" or the word "close".

But you don't use "$connection_upgrade" anywhere in the config that you show.

> upstream to-websocket {
>     server localhost:25565;
> }

That is referring to nginx's idea of localhost, which may or may not
correspond to your in-docker service.

Can you access that IP:port from the machine that nginx is running on? If
not, change it to be whatever IP:port you can use the talk to your
upstream websocket service.

> server {
> 
> #   first redirect to https
>     if ($scheme = "http") {
>         return 301 https://$host$request_uri;
>     }

This entire server{} block is equivalent to

    server { return 301 https://$host$request_uri; }

because of the directive default values. If you don't want to listen
for http, just don't have a server with (effectively) "listen 80;",
(which is what this one has).

> server {
> 
>     root /var/www/html;
>     index  index.html index.htm;
>     server_name myFQDN;
> 
> #   Proxy our outside https to local http
>     listen [::]:443 ssl ipv6only=on; # managed by Certbot
>     listen 443 ssl; # managed by Certbot

<snip>

>     location / {
>         try_files /nonexistent @$http_upgrade;
>     }

That will do an internal redirect to a location that can be chosen by
the client. You hope the http "Upgrade" header will either be empty,
or have the value "websocket". If it is, then one of the following
location{}s will be used; otherwise there will probably be an error
returned to the client.

>     location @websocket {
>         proxy_http_version 1.1;
>         proxy_set_header Upgrade $http_upgrade;
>         proxy_set_header Connection $connection_upgrade;
>         proxy_set_header X-Real-IP $remote_addr;
>         proxy_set_header X-Forwarded-Proto $scheme;
>         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
>         proxy_set_header Host myFQDN;
>         proxy_set_header Referer https://myFQDN;
>         proxy_set_header Referrer https://myFQDN;
> #       proxy_pass http://localhost:25565;
>         proxy_pass http://to-websocket;

>From below, your websocket service appears to be listening on
ip-172-31-24-191.:25565. You'll want to invite nginx to talk to that
IP:port, not localhost.

>     location @ {

And this is what should be used if the incoming request has no "Upgrade"
header. This entire block is equivalent to "location @ { }"

> Here is the listener  process on netstat:
> 
> netstat -a -o | grep 255
> 
> tcp        0      0 ip-172-31-24-191.:25565 0.0.0.0:*               LISTEN
>     off (0.00/0/0)

If you can access that IP:port from the nginx server to talk to the
websocket service, that's what you should configure nginx to try to
talk to.

> Here is the interface being used:

In this case: nginx is talking to an IP. It does not care what the
physical interface is. (iptables and the like do care; but that part
all looks good from here.)

> Here are the iptables stats:

If these rules block nginx from talking to the IP:port and getting the
response, that will want fixing. Otherwise, it's good.

> iptables -L -n -v

These appear to say "accept almost everything; nothing has been dropped",
so these rules are presumably not blocking nginx.

Good luck with it,

	f
-- 
Francis Daly        francis at daoine.org