Needing TLS handshake to fail

[Phillip Odam]

Hi,

I tried asking the following on the general mailing list but I'm 
guessing this is tending more towards development.

I have a project that involves mutual / two way TLS and one of the 
requirements is that the TLS handshake must fail ie. be terminated 
before completion if the handshake is in anyway unsuccessful, eg. no 
client certificate provided or client certificate not trusted.

After having no success getting nginx (v1.16.1) & openssl (v1.0.2k-fips) 
to fail the handshake I ended up looking at the nginx source code, in 
particular src/event/ngx_event_openssl.c, and from what I read here 
https://www.openssl.org/docs/man1.0.2/man3/SSL_CTX_set_verify.html I 
think a small but necessary code change is required.

Some possible approaches when choosing to remain using nginx as the 
server end of the mutual TLS connection

  * in *static int ngx_ssl_verify_callback(int ok, X509_STORE_CTX
    *x509_store)* make it configurable whether *1* is always returned or
    the value of *ok*
  * in *ngx_int_t ngx_ssl_client_certificate(ngx_conf_t *cf, ngx_ssl_t
    *ssl, ngx_str_t *cert, ngx_int_t depth)* make it configurable
    whether *SSL_CTX_set_verify(ssl->ctx, SSL_VERIFY_PEER,
    ngx_ssl_verify_callback);* is called or
    *SSL_CTX_set_verify(ssl->ctx, SSL_VERIFY_PEER, NULL);*

Is a code change required or is there a way for the handshake failure to 
be 'enabled' as opposed to ending up with a successfully established TLS 
connection. Admittedly within nginx there's all the detail that the TLS 
connection doesn't conform to the configured requirements of the TLS 
connection but this doesn't satisfy the requirements for the project.

I won't bother going in to the details of the project but will just say 
it's a third party certification body that requires the TLS handshake to 
be terminated before completion if the handshake is in anyway 
unsuccessful. We're currently looking at alternate software but would 
really love to be able to pull this back in to nginx when/if the day 
comes that nginx supports this.

Assuming development is required, is this something already on the 
backlog? My only comment for enabling the TLS handshake failure is it'd 
be really nice if it were configurable at the level of the virtual host.

Regards,

Phillip
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20190904/a4b398be/attachment.htm>