upstream (tcp stream mode) doesn't detect connecton failure

[Maxim Dounin]

Hello!

On Tue, Jan 09, 2018 at 11:48:54PM +0100, Adam Cecile wrote:

> On 01/09/2018 02:46 PM, Maxim Dounin wrote:
> > Hello!
> >
> > On Mon, Jan 08, 2018 at 12:37:41PM +0000, Cecile, Adam wrote:
> >
> >> Hello,
> >>
> >>
> >> I'm using this quite complicated setup involving SNI routing and proxy_protocol but I'm stuck on something.
> >>
> >>
> >> Here is the configuration file:
> >>
> >> http://paste.debian.net/hidden/62e13f9c/
> >>
> >>
> >> Routing, proxy_protocol, logging stuff is working just fine, the only (quite critical issue) is that the "mag" upstream doesn't see connection failures and does not switch to the second server.
> >>
> >>
> >> In the mag.log file I just see:
> >>
> >> 98.98.98.98 [08/Jan/2018:10:56:10 +0100] proxying to "mag":10.0.0.1:443 TCP 500 0 239 1.01
> >>
> >>
> >> But instead of blacklisting this server and moving to 10.0.0.2 I receive a connection closed error on the client.
> > As far as I understand your configuration, you have two stream
> > proxy layers:
> >
> > 1. The first one uses ssl_preread to obtain SNI name and tries to
> >     do some routing based on it.  This layer also adds to the PROXY
> >     protocol to backend connections.
> >
> > 2. The second one strips PROXY protocol header.
> >
> > The problem with "upstream doesn't see connection failures" is
> > because connection failures are only seen at the second layer (the
> > log line above belongs to the second layer).  The first layer will
> > only see a connection close, and it won't know if there was an
> > error or not.
> >
> > Also note:
> >
> > - You use $proxy_protocol_addr in the "upstream mag {...}" block,
> >    but the upstream block is used only in the first layer, where
> >    $proxy_protocol_addr won't be available according to your
> >    configuration.
> >
> > - You use $name in the logs of the second layer.  It will always
> >    point to "map", as there is no ssl_preread in the second layer,
> >    hence $ssl_preread_server_name will be not available.
> >
> > Depending on what you actually want to achieve, the most
> > straightforward solution might be to actually remove the second
> > proxy layer.
> Hello,
> 
> The proxy protocol was used for the "non-stream" routing on SNI when 
> forwarding to nginx itself as "local_https". At this point it's using 
> regular https vhost, that's why I added proxy_protocol to easily be able 
> to extract the original client address.
> 
> Aim of the two servers on 8080 and 8181 are only to strip proxy_protocol 
> before going to upstream mag. I'd be happy to remove them but if I do 
> that I need a way to strip out proxy_protocol inside the "upstream mag" 
> block. Is it possible ?

Ok, so you use multiple proxy layers to be able to combine 
backends which support/need PROXY protocol and ones which do not, 
right?  This looks like a valid reason, as "proxy_protocol" is 
either on or off in a particular server.

If you want nginx to switch to a different backend while 
maintaining two proxy layers, consider moving balancing to the 
second layer instead.  This way balancing will happen where 
connection errors can be seen, and so nginx will be able to switch 
to a different server on errors.

-- 
Maxim Dounin
http://mdounin.ru/