[nginx] 1.4.1 + spdy + centos 6 + openssl-1.0.1e (static), firefox 21 ajax requests ssl spdy = segfault

[epsilon2930]

Hello, on one of my servers, nginx suddenly started crashing on some
AJAX-heavy pages when accessed via SSL+SPDY. It seems to happen only when
Firefox is the client (tested with Firefox 21), latest version of chrome
uses SPDY without crashing.

uname -a:
Linux myserver.com 2.6.32-358.6.2.el6.x86_64 #1 SMP Thu May 16 20:59:36 UTC
2013 x86_64 x86_64 x86_64 GNU/Linux

nginx compile flags:
CFLAGS="-g -O0" ./configure --with-pcre=/usr/local/src/nginx-1.4.1/pcre-8.32
 --sbin-path=/usr/local/sbin  --conf-path=/etc/nginx/nginx.conf 
--pid-path=/var/run/nginx.pid  --error-log-path=/var/log/nginx/error.log 
--http-log-path=/var/log/nginx/access.log  --with-http_realip_module 
--with-http_ssl_module 
--with-openssl=/usr/local/src/nginx-1.4.1/openssl-1.0.1e
--with-http_spdy_module --http-client-body-temp-path=/tmp/nginx_client 
--http-proxy-temp-path=/tmp/nginx_proxy 
--http-fastcgi-temp-path=/tmp/nginx_fastcgi  --with-http_stub_status_module
--with-debug

nginx log when crash happens:
2013/05/19 18:05:58 [notice] 26737#0: start worker process 26899
2013/05/19 18:05:58 [notice] 26737#0: signal 29 (SIGIO) received
2013/05/19 18:05:59 [notice] 26737#0: signal 17 (SIGCHLD) received
2013/05/19 18:05:59 [alert] 26737#0: worker process 26897 exited on signal
11 (core dumped)
2013/05/19 18:05:59 [notice] 26737#0: start worker process 26907
2013/05/19 18:05:59 [notice] 26737#0: signal 29 (SIGIO) received
2013/05/19 18:06:00 [notice] 26737#0: signal 17 (SIGCHLD) received
2013/05/19 18:06:00 [alert] 26737#0: worker process 26899 exited on signal
11 (core dumped)
2013/05/19 18:06:00 [notice] 26737#0: start worker process 26909
2013/05/19 18:06:00 [notice] 26737#0: signal 29 (SIGIO) received

nginx.conf
http://pastebin.com/G9wAgyeh

gdb backtrace:
# gdb /usr/local/sbin/nginx core.26899

... snip gpl stuff ...

Reading symbols from /usr/local/sbin/nginx...done.
[New Thread 26899]
Missing separate debuginfo for
Try: yum --disablerepo='*' --enablerepo='*-debug*' install
/usr/lib/debug/.build-id/50/fc20fea18a6f375789f0f86e28f463d50714fd
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols
found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libdl.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libz.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libfreebl3.so...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libfreebl3.so
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Core was generated by `nginx: worker process                         '.
Program terminated with signal 11, Segmentation fault.
#0  0x0000003455283c56 in __memset_sse2 () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install
glibc-2.12-1.107.el6.x86_64 nss-softokn-freebl-3.12.9-11.el6.x86_64
zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0  0x0000003455283c56 in __memset_sse2 () from /lib64/libc.so.6
#1  0x0000000000493a67 in ngx_http_spdy_state_data (sc=0x3035ba0,
pos=0x37c78f8 "", end=0x37c78f8 "")
    at src/http/ngx_http_spdy.c:1193
#2  0x0000000000492673 in ngx_http_spdy_state_head (sc=0x3035ba0,
pos=0x37c78f8 "", end=0x37c78f8 "")
    at src/http/ngx_http_spdy.c:699
#3  0x00000000004919e2 in ngx_http_spdy_read_handler (rev=0x7f0318ffe3b8) at
src/http/ngx_http_spdy.c:364
#4  0x000000000042ac31 in ngx_event_process_posted (cycle=0x2893a30,
posted=0x8d1b68)
    at src/event/ngx_event_posted.c:40
#5  0x000000000042887c in ngx_process_events_and_timers (cycle=0x2893a30) at
src/event/ngx_event.c:276
#6  0x0000000000435ebd in ngx_worker_process_cycle (cycle=0x2893a30,
data=0x1)
    at src/os/unix/ngx_process_cycle.c:807
#7  0x00000000004327ca in ngx_spawn_process (cycle=0x2893a30, proc=0x435cf7
<ngx_worker_process_cycle>,
    data=0x1, name=0x609c9b "worker process", respawn=1) at
src/os/unix/ngx_process.c:198
#8  0x0000000000435906 in ngx_reap_children (cycle=0x2893a30) at
src/os/unix/ngx_process_cycle.c:619
#9  0x00000000004345ed in ngx_master_process_cycle (cycle=0x2893a30) at
src/os/unix/ngx_process_cycle.c:180
#10 0x00000000004041b6 in main (argc=3, argv=0x7fffb6c2dbd8) at
src/core/nginx.c:412

Server has a Core i3 540 with HT, OS is 64-bit CentOS 6 fully patched (as of
date of this message).

- kernel log when error occurred:
May 19 18:06:00 saruman kernel: nginx[26899]: segfault at 0 ip
0000003455283c56 sp 00007fffb6c2d498 error 6 in
libc-2.12.so[3455200000+18a000]

The crash is highly reproducible and when it crashes the ip and sp
parameters and offsets are always the same.

I hope I've posted enough info for devs to fix this, sorry for the long
message.

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,239327,239327#msg-239327