Build error --with-debug; ECDHE key exchange TLS problem. [nginx 0.7.62]

[kyleb]

nginx version: 0.7.62
OpenSSL version: 1.0.0-beta3
Platform: Linux 2.6.18 x64

* Short description of problem: *

(a) nginx seems not to handle ephemeral DH key exchanges with EC.  (kx=ECDHE, auth=ECDSA)  Connection dies on handshake.  (b) A build error in 0.7.62 seems to indicate that the problem is in nginx, and not an openssl misconfiguration on my part. =)

* Long description of problem: *

* Steps to reproduce TLS problem: *

1. Compile nginx-0.7.62 with openssl-1.0.0-beta3 (minimal tested 
   configure line below, without --with-debug).
2. Use relevant section of nginx.conf as quoted below.  See comment 
   below about use of EC, which I think is a red herring.
3. Try to connect with openssl s_client, latest Firefox, IE on Vista, etc.

Expected behavior:  Successful TLS connection.
Actual behavior:  Handshake failure; connection dropped.  E.g.:

openssl s_client -connect 127.0.0.1:443
CONNECTED(00000003)
47491508352976:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:656:
---
no peer certificate available
---
No client certificate CA names sent
--
SSL handshake has read 7 bytes and written 186 bytes
---
New, (NONE), Cipher is (NONE)
Compression: NONE
Expansion: NONE
---

Note:  I have tried with various s_client options, e.g., expressly selecting -tls1 -no_ssl3, selecting the appropriate ciphers, etc.  No substantial difference.

Note:  nginx's error log reports *nothing* on the above s_client connection; so I tried to make a debug build...

* Steps to reproduce build error: *

1. Add --with-debug to the same configure line.  (Tried untgzing into fresh build tree, too.)
2. Run make.

Expected behavior:  Successful debug build.
Actual behavior:

cc1: warnings being treated as errors
src/event/ngx_event_openssl.c: In function `ngx_ssl_handshake':
src/event/ngx_event_openssl.c:505: warning: assignment discards qualifiers from pointer target type
make[1]: ***  Error 1
make[1]: Leaving directory `/my-build-path/nginx-0.7.62'
make: ***  Error 2

* Comments: *

My ECDHE problem is in the SSL handshake.  The build error is in ngx_ssl_handshake.  Coincidence?

I am trying to use EC crypto (ECDHE-ECDSA-AES256-SHA).  But ECDH-ECDSA-AES256-SHA (note lack of "E") works fine, so I suspect it is not an EC problem.  Have not tried non-EC DHE.

* Minimal configure line confirmed to produce problem: *

# Semi-minimal build configuration:
# Besides OpenSSL, only including items used by default modules.
./configure --prefix=/test --with-http_ssl_module \
	--with-openssl=/path/to/openssl-1.0.0-beta3 \
	--with-pcre=/path/to/pcre-7.9 \
	--with-zlib=/path/to/zlib-1.2.3 \
	--with-debug

(Make completes without --with-debug line.)

##################################
# Relevant section of nginx.conf #
##################################
ssl_certificate                 /path/to/the.key;
ssl_certificate_key             /path/to/the.crt;
ssl_ciphers                     ECDHE-ECDSA-AES256-SHA;
# Above does not work.
# ECDH-ECDSA-AES256-SHA works, but is not DHE, and seems not supported 
# by recent MSIE.
ssl_prefer_server_ciphers       on;
ssl_protocols                   TLSv1;
# Have tried with "ssl_protocols TLSv1 SSLv3" also.

* Key generation: *

openssl ecparam -name secp384r1 -genkey -out /path/to/the.key -outform PEM
# Have tried the above also with -noout to see if EC parameters were 
# confusing nginx.  No such luck.  Did I make some stupid openssl error? 
openssl req -key /path/to/the.key -keyform PEM -new -out /path/to/the.crt \
	-pubkey -nodes -x509 -days 365 -verify

* Note of Interest *

lighttpd died with segfault on some similar configurations.  At least nginx stays up; it just doesn't complete the handshake.

kyleb

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,11737,11737#msg-11737