Needing TLS handshake to fail

[Phillip Odam]

Hi Maxim

Thanks for the prompt feedback. My understanding for requiring the TLS 
itself to fail, as opposed to doing exactly what you described which is 
also exactly what we've done for other endponts... I quite like nginx's 
ability here, is that it prevents being able to take advantage of 
exploits further down the chain even if these are perceived abilities to 
exploit. What I'm referring to is if the TLS handshake succeeds and a 
HTTP response is returned was any part of the HTTP request read in to 
buffers by nginx?

I'm no networking / security expert so can't talk in more detail and 
about other possible lines of attack. While I certainly get the default 
being a more user friendly response it'd certainly be convenient if 
nginx were able to be configured to be rigid.

Also, the following reference was provided providing a basis for the TLS 
handshake requirement, sections 7.2.1 and 7.2.2 - 
https://tools.ietf.org/html/rfc5246#section-7.2.1. Admittedly production 
implementations vs RFCs don't always work well together and I'm 
certainly not nginx expert but from this lay persons perspective I 
certainly see value in allowing nginx to be configurable and fail the 
TLS handshake.

My C is something like 20yrs old and rusty and certainly have no desire 
to have to build nginx from source each time we want to release a new 
version. I can certainly understand there's no desire within the 
existing development community to make the change but if the community 
is willing to adopt the change from the perspective of the idea I'm 
certainly interested in giving it a crack and seeing if I can neat and 
tidily work it in to the code base and offer up a patch.

Cheers

Phillip


On 9/4/19 8:48 AM, Maxim Dounin wrote:
> Hello!
>
> On Wed, Sep 04, 2019 at 08:35:05AM -0400, Phillip Odam wrote:
>
>> Hi,
>>
>> I tried asking the following on the general mailing list but I'm
>> guessing this is tending more towards development.
>>
>> I have a project that involves mutual / two way TLS and one of the
>> requirements is that the TLS handshake must fail ie. be terminated
>> before completion if the handshake is in anyway unsuccessful, eg. no
>> client certificate provided or client certificate not trusted.
>>
>> After having no success getting nginx (v1.16.1) & openssl (v1.0.2k-fips)
>> to fail the handshake I ended up looking at the nginx source code, in
>> particular src/event/ngx_event_openssl.c, and from what I read here
>> https://www.openssl.org/docs/man1.0.2/man3/SSL_CTX_set_verify.html I
>> think a small but necessary code change is required.
>>
>> Some possible approaches when choosing to remain using nginx as the
>> server end of the mutual TLS connection
>>
>>    * in *static int ngx_ssl_verify_callback(int ok, X509_STORE_CTX
>>      *x509_store)* make it configurable whether *1* is always returned or
>>      the value of *ok*
>>    * in *ngx_int_t ngx_ssl_client_certificate(ngx_conf_t *cf, ngx_ssl_t
>>      *ssl, ngx_str_t *cert, ngx_int_t depth)* make it configurable
>>      whether *SSL_CTX_set_verify(ssl->ctx, SSL_VERIFY_PEER,
>>      ngx_ssl_verify_callback);* is called or
>>      *SSL_CTX_set_verify(ssl->ctx, SSL_VERIFY_PEER, NULL);*
>>
>> Is a code change required or is there a way for the handshake failure to
>> be 'enabled' as opposed to ending up with a successfully established TLS
>> connection. Admittedly within nginx there's all the detail that the TLS
>> connection doesn't conform to the configured requirements of the TLS
>> connection but this doesn't satisfy the requirements for the project.
>>
>> I won't bother going in to the details of the project but will just say
>> it's a third party certification body that requires the TLS handshake to
>> be terminated before completion if the handshake is in anyway
>> unsuccessful. We're currently looking at alternate software but would
>> really love to be able to pull this back in to nginx when/if the day
>> comes that nginx supports this.
>>
>> Assuming development is required, is this something already on the
>> backlog? My only comment for enabling the TLS handshake failure is it'd
>> be really nice if it were configurable at the level of the virtual host.
> No, there is nothing like this in the backlog.  As you can see
> from the code, nginx approach is to proceed with handshake
> regardless of client certificate verification results, and then
> reject requests on HTTP level (and/or just make verification
> results available for logging).  This approach is believed to be
> more user friendly - and, in particular, it allows one to
> configure an error page explaining the problem.
>
> Note well that patches to change this are likely to be rejected
> unless there is some justification better than "there is a third
> party that requires this".
>