Proxy_Protocol with local Proxy_Pass to local secondary nginx process

[Roman Arutyunyan]

Hello Joseph,

On Wed, Aug 22, 2018 at 10:24:23AM -0400, Joseph Wonesh wrote:
> Hello,
> 
> I am trying to route requests such that those requiring websockets will
> route to a long-lived nginx process, and all others will go to the general
> reverse-proxy which handles all other traffic. These nginx processes exist
> in our AWS cloud behind an ELB that has been configured to use Proxy
> Protocol. Note that all of this works correctly with our current setup
> which uses only one nginx process that is configured to use proxy_protocol.
> 
> The change to this setup is as follows:
> 
> The first nginx server handling all ingress uses proxy_protocol and
> forwards requests to either the websocket or non-websocket nginx servers
> locally:
> 
> server {
>     listen 8080 proxy_protocol;
>     real_ip_header proxy_protocol;
>     charset utf-8;
>     client_max_body_size 20M;
> 
>     #send to websocket process
>     location /client {
>       proxy_pass http://localhost:8084;
>       proxy_set_header Host $host;
>       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
>       proxy_set_header X-Real-IP $proxy_protocol_addr;
>       proxy_set_header X-NginX-Proxy true;
>       proxy_set_header X-Proxy-Scheme $scheme;
>       proxy_set_header X-Proxy-Port $proxy_port;
>       proxy_set_header X-ELB-Proxy-Scheme "https";
>       proxy_set_header X-ELB-Proxy-Port "443";
> 
>       # Always support web socket connection upgrades
>       proxy_http_version 1.1;
>       proxy_set_header Upgrade $http_upgrade;
>       proxy_set_header Connection "upgrade";
>     }
> 
>   #send to non-websocket process
>   location / {
>           proxy_pass http://localhost:8082;
>           proxy_set_header Host $host;
>           proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
>           proxy_set_header X-Real-IP $proxy_protocol_addr;
>           proxy_set_header X-NginX-Proxy true;
>           proxy_set_header X-Proxy-Scheme $scheme;
>           proxy_set_header X-Proxy-Port $proxy_port;
>           proxy_set_header X-ELB-Proxy-Scheme "https";
>           proxy_set_header X-ELB-Proxy-Port "443";
> 
>           # Always support web socket connection upgrades
>           proxy_http_version 1.1;
>           proxy_set_header Upgrade $http_upgrade;
>           proxy_set_header Connection "upgrade";
>   }
> 
> 
> When any non-websocket request is sent to localhost:8082, I get an empty
> reply. If I remove proxy_protocol from the first server, I get a response
> as expected.

If removing the proxy_protocol parameter from a server's listen directive fixes
the issue with a specific connection, this means that this connection does not
support the PROXY protocol, whether it is curl or a proxied connection from AWS.
So please make sure the PROXY protocol header is actually sent over the
connection.  It's easy to check by tcpdumping at most 107 first bytes,
especially if PROXY protocol v1 is used.

> Obviously, I need proxy_protocl to support the ingress from
> our ELB, so removing it is not an option. However, I would like to know
> what pieces I am missing to route traffic correctly -- and I would also
> like to know why proxying a request locally from a proxy_protocol enabled
> server to another nginx process (regardless of this second process using
> proxy_protocol or not) fails.

If you configure nginx to expect PROXY protocol (listen ... proxy_protocol),
then the client should actually send it.  AWS balancers can send it, as well
as nginx stream module can (proxy_protocol on).  The nginx http module
cannot send PROXY protocol header (use X-Forwarded-For instead), so you
cannot proxy an HTTP connection to a PROXY-protocol-enabled server.

> For reference, the basic configuration of this secondary nginx process is
> below:
> 
> upstream console {
>    server localhost:3000 max_fails=3 fail_timeout=60 weight=1;
> }
> 
> server {
>     listen 8082;
>    client_max_body_size 20M;
> 
>    location /console {
>      proxy_pass http://console
>    }
> 
>    .
>    .
>    .
> 
> 
> }
> 
> 
> 
> Thank you all for your time,
> Joseph Wonesh
> 
> -- 
> This message is private and confidential. If you have received message in 
> error, please notify us and remove from your system. 

> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx


-- 
Roman Arutyunyan