SSL handshake attack mitigation

[mogwai]

Greetings!

I run a bunch of sites on nginx-plus-r19 (OpenSSL 1.0.2k-fips) and was
recently hit by a nasty DDoS SSL handshake attack.

I noticed nginx worker processes suddenly eating all available CPU and the
"Handshakes failed" counter in the nginx plus dashboard suddenly climbing
out of proportion to the successful handshakes.

If I understand correctly, the limit_req directive would not be effective in
mitigating this type of attack since the SSL handshake occurs earlier in the
request chain.

I ended up setting the error_log level to "info" and feeding the failed
handshake client IPs to fail2ban.

My first question is regarding the particular error log messages produced
during the attack - see example below:

[info] 8050#8050: *146 SSL_do_handshake() failed (SSL: error:14094416:SSL
routines:ssl3_read_bytes:sslv3 alert certificate unknown:SSL alert number
46) while SSL handshaking, client: XXX.XXX.XXX.XXX, server: 0.0.0.0:443

The "certificate unknown" seems to suggest that nginx is trying to verify
the certificate of the client, yet "ssl_verify_client" should be off by
default, so why does nginx care about that certificate?

My second question - is there a better way of mitigating this type of
attack? (Preferably without putting an expensive firewall in front of
nginx)

I would also like to put in a feature request to have a limit_req equivalent
for SSL handshakes.

Thanks!

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,286113,286113#msg-286113