SSLv2 bad detection, patch

[Maxim Dounin]

Hello!

On Thu, Jul 01, 2010 at 05:00:49PM +0200, Mirosław Jaworski wrote:

> Problem:
> old web application supporting SSLv2 only can't connect to the 
> web service migrated behind ssl-offloading nginx.
> 
> Analysis:
> nginx considers connection as plain http. Traffic analysis using
> various client options shows that 0x80 isn't the only possibility
> to show up as first byte in ssl client hello.

A bit more details analisys:

Non-0x80 first byte indicate that message length is more than 255 
bytes.  Session id and challenge data in client hello are 
limited to 16 and 32 bytes, and fixed fields use 9 bytes, this 
gives us something about 198 bytes for cipher specs, i.e. 66 
ciphers listed.  Typical openssl have about 30 ciphers by default, 
so 66 isn't something unrealistic.

In fact, first byte may not even contain high bit set if message 
length is more than 32k.  But it's really unlike to happen in real 
life (and won't work with openssl at least as far as I see).

See here for SSLv2 specs:

http://www.mozilla.org/projects/security/pki/nss/ssl/draft02.html

> Nginx code excerpt ( nginx-0.7.67, src/http/ngx_http_request.c:551 ):
>         if (buf[0] == 0x80 /* SSLv2 */ || buf[0] == 0x16 /* SSLv3/TLSv1
> */) {
>                    ^^
> 
> openssl code excerpt ( openssl-0.9.8k, as such version is installed on
> server side, ssl/s23_srvr.c:268 ):
> 
>                 if ((p[0] & 0x80) && (p[2] == SSL2_MT_CLIENT_HELLO))
>                           ^^
> Difference in SSL logic detection underlined.

Actually, the openssl also checks first byte of message as you can 
see here.  But just checking for & 0x80 should be fine as it 
catches typical http-to-https case.

> Solution:
> Fix SSLv2 detection. Patch attached.

Thanks for the patch.  Looks good for me.

Maxim Dounin