ssl_reject_handshake breaks other server blocks

[J Carter]

Hello,

On Wed, 28 Feb 2024 21:45:37 -0300
Taco de Wolff <tacodewolff at gmail.com> wrote:

> Hi,
> 
> I've noticed at least in 1.24.0 and 1.25.4 that adding an
> ssl_reject_handshake to the default server breaks SNI for other
> servers. Example:
> 
> ```
> server {
>     server_name _;
>     listen 80 default_server;
>     listen 443 default_server ssl;
>     listen 443 default_server quic reuseport;
>     listen [::]:80 default_server;
>     listen [::]:443 default_server ssl;
>     listen [::]:443 default_server quic reuseport;
> 
>     http2 on;
> 
>     # SSL
>     ssl_certificate         /etc/pki/lego/certificates/server.crt;
>     ssl_certificate_key     /etc/pki/lego/certificates/server.key;
>     ssl_trusted_certificate /etc/pki/lego/certificates/server.crt;
>     ssl_reject_handshake on;
> 
>     return 444;
> }
> 
> server {
>     server_name domain.com;
>     listen 443 ssl;
>     listen 443 quic;
>     listen [::]:443 ssl;
>     listen [::]:443 quic;
> 
>     http2 on;
> 
>     root /srv/www/html;
> 
>     # SSL
>     ssl_certificate         /etc/pki/lego/certificates/server.crt;
>     ssl_certificate_key     /etc/pki/lego/certificates/server.key;
>     ssl_trusted_certificate /etc/pki/lego/certificates/server.crt;
> 
>     location / {
>         try_files /index.html =404;
>     }
> }
> ```
> 
> There are two remarks for this example:
> - While enabling HTTP/3 I had to add the ssl_certificate lines to the
> default server, while using solely HTTP/2 this wasn't necessary. It
> will throw an error on trying to start Nginx, is that a bug?

TLS is mandatory for HTTP/3 (well, more accurately for QUIC).

https://stackoverflow.com/questions/72826710/quic-transfer-protocol-need-not-tls

> - The ssl_reject_handshake in the default server will prevent proper
> SNI matching for domain.com. If I run `curl https://domain.com/` it
> works fine, but `curl -k -H 'Host: domain.com'
> https://ipaddress-of-server/` does not. When I remove
> ssl_reject_handshake it works as expected
> 

If you curl an IP Address rather than an FQDN, curl will not include
SNI extension in client hello at all.

ssl_reject_handshake, as the name suggests, rejects TLS handshakes prior
to completion. Nginx cannot perform secondary search for correct server
block using host/authority header, as that would require first
completing handshake, and then parsing host/authority header.

> My intent is to have a default server that responds to non-existing
> domain names. Preferably it responds with 444, but requests over TLS
> (such as old domains names with HTST) will throw a security warning
> that the server's certificates don't match the request's virtual
> host's domain name (as expected).
> 

return 444; just a special return value that causes nginx to terminate
connection, nothing get's sent back to the client at all. return
directives (rewrite module more accurately) runs post TLS handshake
though. For default server TLS connections with your present
configuration - it will never get to that point.

Generally ssl_reject_hanshake is preferable for terminating connections
anyway, as it saves performing heavy TLS handshake.

The return 444 is still relevant for plain text connections that reach
your default server though, so I'd recommend still keeping it.

> Instead of showing a security warning in the browser I prefer a
> connection error, which is why I want to employ ssl_reject_handshake.

Your present configuration should work fine then.

> Kind regards,
> Taco de Wolff