Problem with SSL handshake

[Maxim Dounin]

Hello!

On Wed, Aug 17, 2016 at 12:05:24PM +0000, Mik J wrote:

> nginx version: 1.6.2
> Hello,
> The client and Nginx server seem to have problem to establish a SSL connection. In the logs I have this[crit] 18386#0: *1 SSL_do_handshake() failed (SSL: error:14094456:SSL routines:SSL3_READ_BYTES:tlsv1 unsupported extension:SSL alert number 110) whle SSL handshaking, client: @IP_client, server: 0.0.0.0:443I have searched this message on google but couldn't see anything that would help
> My vhost configurationserver {
>         listen 80;
>         listen 443 ssl;        server_name www.example.org;
> ...       ssl  on;

Note: such a configuration is invalid and will try to negotiate 
SSL on the port 80.  You should remove "ssl on", just "listen ... 
ssl" on appropriate sockets is enough.  See 
http://nginx.org/en/docs/http/configuring_https_servers.html for 
details.

>        ssl_certificate         /etc/ssl/certs/cert.crt;
>        ssl_certificate_key     /etc/ssl/private/key.key;        ssl_session_cache      shared:SSL:10m;}
> Do you know what could be wrong and where should I dig to solve this problem.

The message suggests that the client aborted the connection.  The 
reason claimed is defined as follows, 
https://tools.ietf.org/html/rfc5246#section-7.2.2:

   unsupported_extension
      sent by clients that receive an extended server hello containing
      an extension that they did not put in the corresponding client
      hello.  This message is always fatal.

You may try looking at the handshake using Wireshark to see if 
it's indeed what happens.  You may also try looking for additional 
information on the client side.

Quick search suggests such errors previously appeared due to bugs 
in OpenSSL beta versions, see, e.g., here:

http://openssl.6102.n7.nabble.com/1-0-1beta1-incompatibility-with-gnutls-td8366.html

If you are using some attic version of OpenSSL (much like the 
version of nginx you are using), it may be a good idea to check if 
an upgrade fixes things.

This also can be a bug in the client.  In this case, probably 
disabling TLS via ssl_protocols is the only option if you want to 
support the client, though it's not a solution to be used 
nowadays.

-- 
Maxim Dounin
http://nginx.org/