nginx 1.0.6 OpenSSL SIGSEGV with AES-NI hardware

[Maxim Dounin]

Hello!

On Tue, Sep 20, 2011 at 07:12:37PM +0200, Srebrenko Šehić wrote:

> Hi,
> 
> nginx (vanilla 1.0.6) will die with a segmentation fault as soon as an
> SSL client connects. This is on OpenBSD 4.8, amd64 (OpenSSL 0.9.8k as
> distributed by OpenBSD). CPU is Intel(R) Xeon(R) CPU E31240 @ 3.30GHz
> where AES is accelerated in hardware.
> 
> $ openssl engine -t
> (cryptodev) BSD cryptodev engine
>      [ available ]
> (aesni) Intel AES-NI engine
>      [ available ]
> (dynamic) Dynamic engine loading support
>      [ unavailable ]
> 
> Below is the backtrace. If I add "ssl_engine aesni" in the main nginx
> config, the problem goes away.
> 
> Any clues?
> 
> Program received signal SIGSEGV, Segmentation fault.
> 0x000000020f1d8478 in ?? ()
> (gdb) bt
> #0  0x000000020f1d8478 in ?? ()
> #1  0x0000000205e11f93 in ENGINE_get_cipher (e=0x20e543800, nid=419)
> at /usr/src/lib/libssl/crypto/../src/crypto/engine/tb_cipher.c:123
> #2  0x0000000205e11c92 in EVP_CipherInit_ex (ctx=0x7f7ffffea480,
> cipher=0x206066800, impl=0x20e543800, key=0x2034877d0
> "�=\211T\026�~}J1�\1771Q�\027",
>     iv=0x7f7ffffea540 "�)#r��\205\0218Ƣ\035�\005w�", enc=Variable
> "enc" is not available.
> ) at /usr/src/lib/libssl/crypto/../src/crypto/evp/enc_min.c:165
> #3  0x0000000203d2ac70 in ssl3_send_newsession_ticket (s=0x20fd8ce00)
> at /usr/src/lib/libssl/ssl/../src/ssl/s3_srvr.c:2777
> #4  0x0000000203d2e07e in ssl3_accept (s=0x20fd8ce00) at
> /usr/src/lib/libssl/ssl/../src/ssl/s3_srvr.c:532
> #5  0x000000000043f6f1 in ngx_ssl_handshake (c=0x21cf5c6c0) at
> src/event/ngx_event_openssl.c:575

[...]

Backtrace suggests the problem is somewhere in OpenSSL.  Try 
building newer/vanilla one?

Maxim Dounin