upstream (tcp stream mode) doesn't detect connecton failure

Adam Cecile adam.cecile at
Wed Jan 10 18:18:36 UTC 2018

On 01/10/2018 05:54 PM, Maxim Dounin wrote:
> Hello!
> On Tue, Jan 09, 2018 at 11:48:54PM +0100, Adam Cecile wrote:
>> On 01/09/2018 02:46 PM, Maxim Dounin wrote:
>>> Hello!
>>> On Mon, Jan 08, 2018 at 12:37:41PM +0000, Cecile, Adam wrote:
>>>> Hello,
>>>> I'm using this quite complicated setup involving SNI routing and proxy_protocol but I'm stuck on something.
>>>> Here is the configuration file:
>>>> Routing, proxy_protocol, logging stuff is working just fine, the only (quite critical issue) is that the "mag" upstream doesn't see connection failures and does not switch to the second server.
>>>> In the mag.log file I just see:
>>>> [08/Jan/2018:10:56:10 +0100] proxying to "mag": TCP 500 0 239 1.01
>>>> But instead of blacklisting this server and moving to I receive a connection closed error on the client.
>>> As far as I understand your configuration, you have two stream
>>> proxy layers:
>>> 1. The first one uses ssl_preread to obtain SNI name and tries to
>>>      do some routing based on it.  This layer also adds to the PROXY
>>>      protocol to backend connections.
>>> 2. The second one strips PROXY protocol header.
>>> The problem with "upstream doesn't see connection failures" is
>>> because connection failures are only seen at the second layer (the
>>> log line above belongs to the second layer).  The first layer will
>>> only see a connection close, and it won't know if there was an
>>> error or not.
>>> Also note:
>>> - You use $proxy_protocol_addr in the "upstream mag {...}" block,
>>>     but the upstream block is used only in the first layer, where
>>>     $proxy_protocol_addr won't be available according to your
>>>     configuration.
>>> - You use $name in the logs of the second layer.  It will always
>>>     point to "map", as there is no ssl_preread in the second layer,
>>>     hence $ssl_preread_server_name will be not available.
>>> Depending on what you actually want to achieve, the most
>>> straightforward solution might be to actually remove the second
>>> proxy layer.
>> Hello,
>> The proxy protocol was used for the "non-stream" routing on SNI when
>> forwarding to nginx itself as "local_https". At this point it's using
>> regular https vhost, that's why I added proxy_protocol to easily be able
>> to extract the original client address.
>> Aim of the two servers on 8080 and 8181 are only to strip proxy_protocol
>> before going to upstream mag. I'd be happy to remove them but if I do
>> that I need a way to strip out proxy_protocol inside the "upstream mag"
>> block. Is it possible ?
> Ok, so you use multiple proxy layers to be able to combine
> backends which support/need PROXY protocol and ones which do not,
> right?  This looks like a valid reason, as "proxy_protocol" is
> either on or off in a particular server.
Yes exactly !

Aim of this setup is to do SNI routing to TCP endpoints (with failover) 
or HTTPS virtual hosts.
> If you want nginx to switch to a different backend while
> maintaining two proxy layers, consider moving balancing to the
> second layer instead.  This way balancing will happen where
> connection errors can be seen, and so nginx will be able to switch
> to a different server on errors.

Could you be more specific and show me how to do this with my current 
configuration ? I'm a bit lost...

Thanks !

More information about the nginx mailing list