upstream (tcp stream mode) doesn't detect connecton failure

[Adam Cecile]

On 01/10/2018 05:54 PM, Maxim Dounin wrote:
> Hello!
>
> On Tue, Jan 09, 2018 at 11:48:54PM +0100, Adam Cecile wrote:
>
>> On 01/09/2018 02:46 PM, Maxim Dounin wrote:
>>> Hello!
>>>
>>> On Mon, Jan 08, 2018 at 12:37:41PM +0000, Cecile, Adam wrote:
>>>
>>>> Hello,
>>>>
>>>>
>>>> I'm using this quite complicated setup involving SNI routing and proxy_protocol but I'm stuck on something.
>>>>
>>>>
>>>> Here is the configuration file:
>>>>
>>>> http://paste.debian.net/hidden/62e13f9c/
>>>>
>>>>
>>>> Routing, proxy_protocol, logging stuff is working just fine, the only (quite critical issue) is that the "mag" upstream doesn't see connection failures and does not switch to the second server.
>>>>
>>>>
>>>> In the mag.log file I just see:
>>>>
>>>> 98.98.98.98 [08/Jan/2018:10:56:10 +0100] proxying to "mag":10.0.0.1:443 TCP 500 0 239 1.01
>>>>
>>>>
>>>> But instead of blacklisting this server and moving to 10.0.0.2 I receive a connection closed error on the client.
>>> As far as I understand your configuration, you have two stream
>>> proxy layers:
>>>
>>> 1. The first one uses ssl_preread to obtain SNI name and tries to
>>>      do some routing based on it.  This layer also adds to the PROXY
>>>      protocol to backend connections.
>>>
>>> 2. The second one strips PROXY protocol header.
>>>
>>> The problem with "upstream doesn't see connection failures" is
>>> because connection failures are only seen at the second layer (the
>>> log line above belongs to the second layer).  The first layer will
>>> only see a connection close, and it won't know if there was an
>>> error or not.
>>>
>>> Also note:
>>>
>>> - You use $proxy_protocol_addr in the "upstream mag {...}" block,
>>>     but the upstream block is used only in the first layer, where
>>>     $proxy_protocol_addr won't be available according to your
>>>     configuration.
>>>
>>> - You use $name in the logs of the second layer.  It will always
>>>     point to "map", as there is no ssl_preread in the second layer,
>>>     hence $ssl_preread_server_name will be not available.
>>>
>>> Depending on what you actually want to achieve, the most
>>> straightforward solution might be to actually remove the second
>>> proxy layer.
>> Hello,
>>
>> The proxy protocol was used for the "non-stream" routing on SNI when
>> forwarding to nginx itself as "local_https". At this point it's using
>> regular https vhost, that's why I added proxy_protocol to easily be able
>> to extract the original client address.
>>
>> Aim of the two servers on 8080 and 8181 are only to strip proxy_protocol
>> before going to upstream mag. I'd be happy to remove them but if I do
>> that I need a way to strip out proxy_protocol inside the "upstream mag"
>> block. Is it possible ?
> Ok, so you use multiple proxy layers to be able to combine
> backends which support/need PROXY protocol and ones which do not,
> right?  This looks like a valid reason, as "proxy_protocol" is
> either on or off in a particular server.
Yes exactly !

Aim of this setup is to do SNI routing to TCP endpoints (with failover) 
or HTTPS virtual hosts.
>
> If you want nginx to switch to a different backend while
> maintaining two proxy layers, consider moving balancing to the
> second layer instead.  This way balancing will happen where
> connection errors can be seen, and so nginx will be able to switch
> to a different server on errors.

Could you be more specific and show me how to do this with my current 
configuration ? I'm a bit lost...

Thanks !