help with https to http and WSS to WS reverse proxy conf

[Francis Daly]

On Mon, Sep 12, 2022 at 05:46:21PM -0700, Michael Williams wrote:

Hi there,

> Wow thank you. This really helps all the guidance and instruction. I really
> appreciate your time.

No worries.

> One thing to clarify, is that if I turn off NGINX, the client page works
> fine and connects to the app server inside the docker OK.

I confess that I am confused as to what your current architecture is.

Can you describe it? Along the lines of:

Without nginx involved, we have (http service) running on (docker
ip:port) and when we tell the client to access (http:// docker ip:port)
everything works, including the websocket thing.

With nginx involved to test reverse-proxying http, the docker side is
identical, but we tell the client to access (http:// nginx ip:port)
and everything works? Or not everything works?

With nginx involved to test reverse-proxying https, the docker side is
identical, and we tell the client to access (https:// nginx ip:port),
and some things work?

With that information, it might be clear to someone where the first
problem appears.

In this configuration:

> server {
>     index  index.html index.htm;
>     listen [::]:443 ssl ipv6only=on; # managed by Certbot
>     listen 443 ssl; # managed by Certbot
>     listen 25566 ssl;

nginx is listening for https on two ports.

What test are you running? Which port are you using?

>     location @wss {

>         proxy_http_version 1.1;
>         proxy_set_header Upgrade $http_upgrade;
>         proxy_set_header Connection $connection_upgrade;

>         proxy_pass http://172.31.24.191:25565;

nginx is talking to this port without https.

What works here / what fails here? What do the logs say?

> My idea was to try changing our client webpage to access a different port #
> than the one our app server in the docker is listening to.

I'm afraid I am not sure what that means. I thought the client webpage
was accessing nginx on port 443 and the backend / upstream http server
was listening on the high port?

Maybe I am getting confused among multiple tests that you are running.

> With that change
> I see from WIreshark on my local that the WSS connection seems to go
> through OK with NGINX:
> 
> [image: Screen Shot 2022-09-12 at 5.29.50 PM.png]

I'm seeing a picture; but I'm not seeing anything that obviously says
that a WSS connection is working anywhere.

I'm seeing a TLS connection between the client and nginx that is cleanly
closed after a fraction of a second. I see nothing that suggests that
nginx is doing a proxy_pass to the upstream server. (But maybe that was
excluded from the tcpdump?)

> Our app server shows that the connection to the server also starts but then
> disconnect it:
> (22:36:59) Disconnected <client IP address>  (unknown opcode 22)

With nginx involved, the app server should never see the client IP
address directly; it should only see connections from nginx. (It might
see the client IP listed in the http headers.)

> My question here, does NGINX negotiate the entire handshake for HTTPS to
> WSS upgrade itself, without forwarding the same pages to our app server ?
> Is there a way to forward those pages to the app server also ? I think our
> app server may insist on negotiating a ws:// connection itself, but not a
> wss:// connection.

As I understand it: the client makes a TLS connection to nginx,
and sends a http request inside that TLS connection (== a https
request). Separately, nginx makes a http connection to the upstream
server, and (through config) passes along the Upgrade-and-friends headers
that the client sent to nginx, requesting that this connection switch to
a websocket connection. And after that works, nginx effectively becomes a
"blind tunnel" for the connection contents, passing unencrypted things
on the nginx-to-upstream side and encrypted things on the nginx-to-client
side, and generally not caring about what is inside.


If things are still not working as wanted, I suggest simplifying things
as much as possible.

Make the nginx config be not much more than what step 6 on
https://www.nginx.com/blog/websocket-nginx/ shows, and include enough
information in any report of a test, so that someone else will be able
to repeat the test on their system to see what happens there.

Good luck with it,

	f
-- 
Francis Daly        francis at daoine.org