upstream (tcp stream mode) doesn't detect connecton failure

[Adam Cecile]

On 01/09/2018 02:46 PM, Maxim Dounin wrote:
> Hello!
>
> On Mon, Jan 08, 2018 at 12:37:41PM +0000, Cecile, Adam wrote:
>
>> Hello,
>>
>>
>> I'm using this quite complicated setup involving SNI routing and proxy_protocol but I'm stuck on something.
>>
>>
>> Here is the configuration file:
>>
>> http://paste.debian.net/hidden/62e13f9c/
>>
>>
>> Routing, proxy_protocol, logging stuff is working just fine, the only (quite critical issue) is that the "mag" upstream doesn't see connection failures and does not switch to the second server.
>>
>>
>> In the mag.log file I just see:
>>
>> 98.98.98.98 [08/Jan/2018:10:56:10 +0100] proxying to "mag":10.0.0.1:443 TCP 500 0 239 1.01
>>
>>
>> But instead of blacklisting this server and moving to 10.0.0.2 I receive a connection closed error on the client.
> As far as I understand your configuration, you have two stream
> proxy layers:
>
> 1. The first one uses ssl_preread to obtain SNI name and tries to
>     do some routing based on it.  This layer also adds to the PROXY
>     protocol to backend connections.
>
> 2. The second one strips PROXY protocol header.
>
> The problem with "upstream doesn't see connection failures" is
> because connection failures are only seen at the second layer (the
> log line above belongs to the second layer).  The first layer will
> only see a connection close, and it won't know if there was an
> error or not.
>
> Also note:
>
> - You use $proxy_protocol_addr in the "upstream mag {...}" block,
>    but the upstream block is used only in the first layer, where
>    $proxy_protocol_addr won't be available according to your
>    configuration.
>
> - You use $name in the logs of the second layer.  It will always
>    point to "map", as there is no ssl_preread in the second layer,
>    hence $ssl_preread_server_name will be not available.
>
> Depending on what you actually want to achieve, the most
> straightforward solution might be to actually remove the second
> proxy layer.
Hello,

The proxy protocol was used for the "non-stream" routing on SNI when 
forwarding to nginx itself as "local_https". At this point it's using 
regular https vhost, that's why I added proxy_protocol to easily be able 
to extract the original client address.

Aim of the two servers on 8080 and 8181 are only to strip proxy_protocol 
before going to upstream mag. I'd be happy to remove them but if I do 
that I need a way to strip out proxy_protocol inside the "upstream mag" 
block. Is it possible ?

Thanks a lot,

Adam.