Proxy_Protocol with local Proxy_Pass to local secondary nginx process

[Joseph Wonesh]

Thank you for your reply, Roman.

It turns out the issue was adding the various proxy and forwarded headers
in that entry. Changing it to the below block fixed my issue:

#send to non-websocket process
  location / {
          proxy_pass http://localhost:8082;
          proxy_set_header Host $host;
  }


Now everything is working as expected.

Cheers,
Joseph Wonesh

On Thu, Aug 23, 2018 at 6:46 AM, Roman Arutyunyan <arut at nginx.com> wrote:

> Hello Joseph,
>
> On Wed, Aug 22, 2018 at 10:24:23AM -0400, Joseph Wonesh wrote:
> > Hello,
> >
> > I am trying to route requests such that those requiring websockets will
> > route to a long-lived nginx process, and all others will go to the
> general
> > reverse-proxy which handles all other traffic. These nginx processes
> exist
> > in our AWS cloud behind an ELB that has been configured to use Proxy
> > Protocol. Note that all of this works correctly with our current setup
> > which uses only one nginx process that is configured to use
> proxy_protocol.
> >
> > The change to this setup is as follows:
> >
> > The first nginx server handling all ingress uses proxy_protocol and
> > forwards requests to either the websocket or non-websocket nginx servers
> > locally:
> >
> > server {
> >     listen 8080 proxy_protocol;
> >     real_ip_header proxy_protocol;
> >     charset utf-8;
> >     client_max_body_size 20M;
> >
> >     #send to websocket process
> >     location /client {
> >       proxy_pass http://localhost:8084;
> >       proxy_set_header Host $host;
> >       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
> >       proxy_set_header X-Real-IP $proxy_protocol_addr;
> >       proxy_set_header X-NginX-Proxy true;
> >       proxy_set_header X-Proxy-Scheme $scheme;
> >       proxy_set_header X-Proxy-Port $proxy_port;
> >       proxy_set_header X-ELB-Proxy-Scheme "https";
> >       proxy_set_header X-ELB-Proxy-Port "443";
> >
> >       # Always support web socket connection upgrades
> >       proxy_http_version 1.1;
> >       proxy_set_header Upgrade $http_upgrade;
> >       proxy_set_header Connection "upgrade";
> >     }
> >
> >   #send to non-websocket process
> >   location / {
> >           proxy_pass http://localhost:8082;
> >           proxy_set_header Host $host;
> >           proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
> >           proxy_set_header X-Real-IP $proxy_protocol_addr;
> >           proxy_set_header X-NginX-Proxy true;
> >           proxy_set_header X-Proxy-Scheme $scheme;
> >           proxy_set_header X-Proxy-Port $proxy_port;
> >           proxy_set_header X-ELB-Proxy-Scheme "https";
> >           proxy_set_header X-ELB-Proxy-Port "443";
> >
> >           # Always support web socket connection upgrades
> >           proxy_http_version 1.1;
> >           proxy_set_header Upgrade $http_upgrade;
> >           proxy_set_header Connection "upgrade";
> >   }
> >
> >
> > When any non-websocket request is sent to localhost:8082, I get an empty
> > reply. If I remove proxy_protocol from the first server, I get a response
> > as expected.
>
> If removing the proxy_protocol parameter from a server's listen directive
> fixes
> the issue with a specific connection, this means that this connection does
> not
> support the PROXY protocol, whether it is curl or a proxied connection
> from AWS.
> So please make sure the PROXY protocol header is actually sent over the
> connection.  It's easy to check by tcpdumping at most 107 first bytes,
> especially if PROXY protocol v1 is used.
>
> > Obviously, I need proxy_protocl to support the ingress from
> > our ELB, so removing it is not an option. However, I would like to know
> > what pieces I am missing to route traffic correctly -- and I would also
> > like to know why proxying a request locally from a proxy_protocol enabled
> > server to another nginx process (regardless of this second process using
> > proxy_protocol or not) fails.
>
> If you configure nginx to expect PROXY protocol (listen ...
> proxy_protocol),
> then the client should actually send it.  AWS balancers can send it, as
> well
> as nginx stream module can (proxy_protocol on).  The nginx http module
> cannot send PROXY protocol header (use X-Forwarded-For instead), so you
> cannot proxy an HTTP connection to a PROXY-protocol-enabled server.
>
> > For reference, the basic configuration of this secondary nginx process is
> > below:
> >
> > upstream console {
> >    server localhost:3000 max_fails=3 fail_timeout=60 weight=1;
> > }
> >
> > server {
> >     listen 8082;
> >    client_max_body_size 20M;
> >
> >    location /console {
> >      proxy_pass http://console
> >    }
> >
> >    .
> >    .
> >    .
> >
> >
> > }
> >
> >
> >
> > Thank you all for your time,
> > Joseph Wonesh
> >
> > --
> > This message is private and confidential. If you have received message
> in
> > error, please notify us and remove from your system.
>
> > _______________________________________________
> > nginx mailing list
> > nginx at nginx.org
> > http://mailman.nginx.org/mailman/listinfo/nginx
>
>
> --
> Roman Arutyunyan
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>

-- 
This message is private and confidential. If you have received message in 
error, please notify us and remove from your system. 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20180823/08e21e93/attachment.html>