nginx/1.10.2 crashes in ngx_http_v2_write_handler

[Dmitriy M.]

Добрый день,

у нас включен HTTP/2, обслуживаются до 2000 req/s и до 300 conn/s в
пике, после обновления OS и nginx до nginx/1.10.2 за несколько суток
получили около 10 крашей с одинаковыми 2мя бектрейсами:

Первый :

#0  0x00000000004a86c4 in ngx_http_v2_write_handler (wev=<value
optimized out>) at src/http/v2/ngx_http_v2.c:435
435             ngx_queue_remove(q);
[New Thread 801c16000 (LWP 100654/<unknown>)]
(gdb) bt
#0  0x00000000004a86c4 in ngx_http_v2_write_handler (wev=<value
optimized out>) at src/http/v2/ngx_http_v2.c:435
#1  0x000000000046d776 in ngx_kqueue_process_events (cycle=<value
optimized out>, timer=<value optimized out>, flags=0) at
src/event/modules/ngx_kqueue_module.c:669
#2  0x0000000000464ac4 in ngx_process_events_and_timers
(cycle=0x85c1e6050) at src/event/ngx_event.c:242
#3  0x000000000046bf68 in ngx_worker_process_cycle (cycle=<value
optimized out>, data=<value optimized out>) at
src/os/unix/ngx_process_cycle.c:753
#4  0x000000000046a616 in ngx_spawn_process (cycle=<value optimized
out>, proc=<value optimized out>, data=<value optimized out>,
name=<value optimized out>,
    respawn=<value optimized out>) at src/os/unix/ngx_process.c:198
#5  0x000000000046b819 in ngx_start_worker_processes (cycle=<value
optimized out>, n=8, type=-4) at src/os/unix/ngx_process_cycle.c:358
#6  0x000000000046b638 in ngx_master_process_cycle (cycle=0x85c1e6050)
at src/os/unix/ngx_process_cycle.c:243
#7  0x00000000004481f4 in main (argc=<value optimized out>,
argv=<value optimized out>) at src/core/nginx.c:367
Current language:  auto; currently minimal
(gdb) fr 0
#0  0x00000000004a86c4 in ngx_http_v2_write_handler (wev=<value
optimized out>) at src/http/v2/ngx_http_v2.c:435
435             ngx_queue_remove(q);
(gdb) l
430         }
431
432         while (!ngx_queue_empty(&h2c->posted)) {
433             q = ngx_queue_head(&h2c->posted);
434
435             ngx_queue_remove(q);
436
437             stream = ngx_queue_data(q, ngx_http_v2_stream_t, queue);
438
439             stream->handled = 0;
(gdb)


Второй :

#0  ngx_http_v2_write_handler (wev=<value optimized out>) at
src/http/v2/ngx_http_v2.c:444
444             wev = stream->request->connection->write;
[New Thread 801c16000 (LWP 100643/<unknown>)]
(gdb) bt
#0  ngx_http_v2_write_handler (wev=<value optimized out>) at
src/http/v2/ngx_http_v2.c:444
#1  0x000000000046d776 in ngx_kqueue_process_events (cycle=<value
optimized out>, timer=<value optimized out>, flags=0) at
src/event/modules/ngx_kqueue_module.c:669
#2  0x0000000000464ac4 in ngx_process_events_and_timers
(cycle=0x801c8b050) at src/event/ngx_event.c:242
#3  0x000000000046bf68 in ngx_worker_process_cycle (cycle=<value
optimized out>, data=<value optimized out>) at
src/os/unix/ngx_process_cycle.c:753
#4  0x000000000046a616 in ngx_spawn_process (cycle=<value optimized
out>, proc=<value optimized out>, data=<value optimized out>,
name=<value optimized out>,
    respawn=<value optimized out>) at src/os/unix/ngx_process.c:198
#5  0x000000000046b819 in ngx_start_worker_processes (cycle=<value
optimized out>, n=8, type=-3) at src/os/unix/ngx_process_cycle.c:358
#6  0x000000000046afe5 in ngx_master_process_cycle (cycle=<value
optimized out>) at src/os/unix/ngx_process_cycle.c:130
#7  0x00000000004481f4 in main (argc=<value optimized out>,
argv=<value optimized out>) at src/core/nginx.c:367
Current language:  auto; currently minimal
(gdb) fr 0
#0  ngx_http_v2_write_handler (wev=<value optimized out>) at
src/http/v2/ngx_http_v2.c:444
444             wev = stream->request->connection->write;
(gdb) l
439             stream->handled = 0;
440
441             ngx_log_debug1(NGX_LOG_DEBUG_HTTP, c->log, 0,
442                            "run http2 stream %ui", stream->node->id);
443
444             wev = stream->request->connection->write;
445
446             wev->active = 0;
447             wev->ready = 1;
448

В логах ничего кроме:
2016/11/18 01:39:36 [alert] 628#100623: worker process 19425 exited on
signal 11 (core dumped)


Опции сборки (собиралось из исходников):

nginx version: nginx/1.10.2
built by clang 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 3.8.0)
built with LibreSSL 2.4.4
TLS SNI support enabled
configure arguments: --prefix=/usr/local/etc/nginx --with-cc-opt='-I
/usr/local/include' --with-ld-opt='-L /usr/local/lib -lrt'
--conf-path=/usr/local/etc/nginx/nginx.conf
--sbin-path=/usr/local/sbin/nginx --pid-path=/var/run/nginx.pid
--error-log-path=/var/log/nginx/error.log --user=www --group=www
--modules-path=/usr/local/libexec/nginx
--http-client-body-temp-path=/var/tmp/nginx/client_body_temp
--http-fastcgi-temp-path=/var/tmp/nginx/fastcgi_temp
--http-proxy-temp-path=/var/tmp/nginx/proxy_temp
--http-scgi-temp-path=/var/tmp/nginx/scgi_temp
--http-uwsgi-temp-path=/var/tmp/nginx/uwsgi_temp
--http-log-path=/var/log/nginx/access.log --with-http_realip_module
--with-http_stub_status_module --with-pcre --with-threads
--add-dynamic-module=./headers-more-nginx-module-84241e4
--with-http_ssl_module --with-http_v2_module
--add-module=./nginx-upload-progress-module-0.9.0u
--with-openssl=./libressl-2.4.4

OS:  FreeBSD 11.0-STABLE #1 r308689


Буду рад помочь в поиске и устранении проблемы (если она еще не
устранена в 1.11.*), если нужны дополнительные данные - попробую
предоставить.

Спасибо!