spdy+eval+memcached=signal 11
Yuri Bayteryakov
yuriymagnum at gmail.com
Fri Dec 21 10:08:33 UTC 2012
Hello.
Testing spdy + eval + memcached module for our deployment. Without spdy
this works fine. Worker processes die with following configuration:
# cat /etc/nginx/nginx.conf
user nginx;
worker_processes 1;
worker_rlimit_nofile 2048;
error_log /var/log/nginx/error.log debug;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
worker_rlimit_core 500M;
working_directory /tmp/dumps/;
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
access_log /var/log/nginx/access.log;
sendfile on;
keepalive_timeout 70;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
server {
listen 443 ssl spdy;
ssl_certificate ssl/ca.crt;
ssl_certificate_key ssl/ca.key;
location / {
eval_escalate on;
eval $auth_cookie_value {
set $memcached_key $host-$cookie_abc;
memcached_next_upstream error timeout invalid_response
not_found;
memcached_pass 192.168.0.55:11211;
}
proxy_pass http://192.168.0.158:8081;
}
}
}
Linux hostname 2.6.32-279.el6.x86_64 #1 SMP Thu Jun 21 15:00:18 EDT 2012
x86_64 x86_64 x86_64 GNU/Linux
error.log:
2012/12/21 15:53:12 [notice] 16252#0: start worker process 16273
2012/12/21 15:53:22 [info] 16273#0: *20 SSL NPN ADVERT while SSL
handshaking, client: 192.168.3.70, server:
2012/12/21 15:53:22 [notice] 16252#0: signal 17 (SIGCHLD) received
2012/12/21 15:53:22 [alert] 16252#0: worker process 16273 exited on signal
11 (core dumped)
2012/12/21 15:53:22 [notice] 16252#0: start worker process 16275
# /usr/sbin/nginx -V
nginx version: nginx/1.3.9
built by gcc 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC)
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx/ --sbin-path=/usr/sbin/nginx
--conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log
--http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid
--lock-path=/var/run/nginx.lock
--http-client-body-temp-path=/var/cache/nginx/client_temp
--http-proxy-temp-path=/var/cache/nginx/proxy_temp
--http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp
--http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp
--http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx
--with-http_ssl_module --with-http_realip_module
--with-http_addition_module --with-http_sub_module
--with-http_gzip_static_module --with-http_random_index_module
--with-http_stub_status_module --with-openssl=/tmp/openssl101-1.0.1c
--with-cc-opt='-O0 -g -I/opt/openssl101/include'
--with-ld-opt=-L/usr/openssl101/lib
--add-module=/home/bld/rpmbuild/BUILD/nginx-1.3.9/nginx_eval_module-1.0.1
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20121221/e0ab08d8/attachment-0001.html>
-------------- next part --------------
# gdb /usr/sbin/nginx core.16273
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-56.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/nginx...Reading symbols from /usr/lib/debug/usr/sbin/nginx.debug...done.
done.
[New Thread 16273]
Missing separate debuginfo for
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/80/1b9608daa2cd5f7035ad415e9c7dd06ebdb0a2
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libpcre.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libpcre.so.0
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libfreebl3.so...(no debugging symbols found)...done.
Loaded symbols for /lib64/libfreebl3.so
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Core was generated by `nginx: worker process '.
Program terminated with signal 11, Segmentation fault.
#0 0x000000000043352c in ngx_epoll_add_event (ev=0xfdd1e0, event=4, flags=18446744071562067968) at src/event/modules/ngx_epoll_module.c:404
404 e = c->read;
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.80.el6.x86_64 nss-softokn-freebl-3.12.9-11.el6.x86_64 pcre-7.8-4.el6.x86_64 zlib-1.2.3-27.el6.x86_64
(gdb) bt
#0 0x000000000043352c in ngx_epoll_add_event (ev=0xfdd1e0, event=4, flags=18446744071562067968) at src/event/modules/ngx_epoll_module.c:404
#1 0x000000000046271d in ngx_http_upstream_init (r=0xfde6c8) at src/http/ngx_http_upstream.c:444
#2 0x00000000004b0a9b in ngx_http_memcached_handler (r=0xfde6c8) at src/http/modules/ngx_http_memcached_module.c:211
#3 0x0000000000441724 in ngx_http_core_content_phase (r=0xfde6c8, ph=0xf8e790) at src/http/ngx_http_core_module.c:1404
#4 0x0000000000440665 in ngx_http_core_run_phases (r=0xfde6c8) at src/http/ngx_http_core_module.c:885
#5 0x00000000004405dc in ngx_http_handler (r=0xfde6c8) at src/http/ngx_http_core_module.c:868
#6 0x000000000044e548 in ngx_http_run_posted_requests (c=0xf77220) at src/http/ngx_http_request.c:1954
#7 0x000000000048815e in ngx_http_spdy_run_request (r=0xfdd278) at src/http/ngx_http_spdy.c:2026
#8 0x0000000000487d22 in ngx_http_spdy_process_headers (sc=0xfdbef0, pos=0x7fff27f2f298, size=0) at src/http/ngx_http_spdy.c:1887
#9 0x0000000000485e66 in ngx_http_spdy_read_handler (rev=0xf98820) at src/http/ngx_http_spdy.c:734
#10 0x000000000048a2ec in ngx_http_spdy_keepalive_handler (rev=0xf98820) at src/http/ngx_http_spdy.c:3254
#11 0x0000000000433c1e in ngx_epoll_process_events (cycle=0xf6ef50, timer=179999, flags=1) at src/event/modules/ngx_epoll_module.c:683
#12 0x0000000000426d16 in ngx_process_events_and_timers (cycle=0xf6ef50) at src/event/ngx_event.c:247
#13 0x0000000000432032 in ngx_worker_process_cycle (cycle=0xf6ef50, data=0x0) at src/os/unix/ngx_process_cycle.c:807
#14 0x000000000042efa2 in ngx_spawn_process (cycle=0xf6ef50, proc=0x431ea5 <ngx_worker_process_cycle>, data=0x0, name=0x5e4f98 "worker process",
respawn=0) at src/os/unix/ngx_process.c:198
#15 0x0000000000431ab8 in ngx_reap_children (cycle=0xf6ef50) at src/os/unix/ngx_process_cycle.c:619
#16 0x0000000000430b65 in ngx_master_process_cycle (cycle=0xf6ef50) at src/os/unix/ngx_process_cycle.c:180
#17 0x0000000000404336 in main (argc=3, argv=0x7fff27f2f948) at src/core/nginx.c:412
(gdb) backtrace full
#0 0x000000000043352c in ngx_epoll_add_event (ev=0xfdd1e0, event=4, flags=18446744071562067968) at src/event/modules/ngx_epoll_module.c:404
op = 32767
events = 4
prev = 0
e = 0x358
c = 0x0
ee = {events = 16675544, data = {ptr = 0xfe71c000000000, fd = 0, u32 = 0, u64 = 71619713532166144}}
#1 0x000000000046271d in ngx_http_upstream_init (r=0xfde6c8) at src/http/ngx_http_upstream.c:444
c = 0xf77220
#2 0x00000000004b0a9b in ngx_http_memcached_handler (r=0xfde6c8) at src/http/modules/ngx_http_memcached_module.c:211
rc = 0
u = 0xfe72d8
ctx = 0xfe7630
mlcf = 0xf8a250
#3 0x0000000000441724 in ngx_http_core_content_phase (r=0xfde6c8, ph=0xf8e790) at src/http/ngx_http_core_module.c:1404
root = 16312136
rc = 16639688
path = {len = 140733863620832, data = 0x4406c1 "H\211E\370H\203}", <incomplete sequence \370>}
#4 0x0000000000440665 in ngx_http_core_run_phases (r=0xfde6c8) at src/http/ngx_http_core_module.c:885
rc = -2
ph = 0xf8e670
cmcf = 0xf707a8
#5 0x00000000004405dc in ngx_http_handler (r=0xfde6c8) at src/http/ngx_http_core_module.c:868
cmcf = 0xf707a8
#6 0x000000000044e548 in ngx_http_run_posted_requests (c=0xf77220) at src/http/ngx_http_request.c:1954
r = 0xfde6c8
ctx = 0xfdd780
pr = 0xfdf128
#7 0x000000000048815e in ngx_http_spdy_run_request (r=0xfdd278) at src/http/ngx_http_spdy.c:2026
i = 13
part = 0xfdd2e8
h = 0xfde280
fc = 0xf77220
hh = 0x0
cmcf = 0xf707a8
#8 0x0000000000487d22 in ngx_http_spdy_process_headers (sc=0xfdbef0, pos=0x7fff27f2f298, size=0) at src/http/ngx_http_spdy.c:1887
z = 0
---Type <return> to continue, or q <return> to quit---
buf = 0xfdd798
rc = -4
last = 1
h = 0xfde4c0
c = 0x7ff2164b9190
r = 0xfdd278
#9 0x0000000000485e66 in ngx_http_spdy_read_handler (rev=0xf98820) at src/http/ngx_http_spdy.c:734
p = 0x7ff2163b825e ""
end = 0x7ff2163b82d8 ""
available = 1048568
n = 694
rc = 0
rest = 0
c = 0x7ff2164b9190
smcf = 0xf710c0
sc = 0xfdbef0
#10 0x000000000048a2ec in ngx_http_spdy_keepalive_handler (rev=0xf98820) at src/http/ngx_http_spdy.c:3254
c = 0x7ff2164b9190
#11 0x0000000000433c1e in ngx_epoll_process_events (cycle=0xf6ef50, timer=179999, flags=1) at src/event/modules/ngx_epoll_module.c:683
events = 1
revents = 1
instance = 0
i = 0
level = 9043008
err = 0
rev = 0xf98820
wev = 0x428b7a
queue = 0x2bf1f
c = 0x7ff2164b9190
#12 0x0000000000426d16 in ngx_process_events_and_timers (cycle=0xf6ef50) at src/event/ngx_event.c:247
flags = 1
timer = 179999
delta = 1356083602067
#13 0x0000000000432032 in ngx_worker_process_cycle (cycle=0xf6ef50, data=0x0) at src/os/unix/ngx_process_cycle.c:807
worker = 0
i = 140733863622976
c = 0x0
---Type <return> to continue, or q <return> to quit---
#14 0x000000000042efa2 in ngx_spawn_process (cycle=0xf6ef50, proc=0x431ea5 <ngx_worker_process_cycle>, data=0x0, name=0x5e4f98 "worker process",
respawn=0) at src/os/unix/ngx_process.c:198
on = 1
pid = 0
s = 0
#15 0x0000000000431ab8 in ngx_reap_children (cycle=0xf6ef50) at src/os/unix/ngx_process_cycle.c:619
i = 0
n = 1
live = 0
ch = {command = 2, pid = 16271, slot = 0, fd = -1}
ccf = 0x50d43188
#16 0x0000000000430b65 in ngx_master_process_cycle (cycle=0xf6ef50) at src/os/unix/ngx_process_cycle.c:180
title = 0xf8e895 "master process /usr/sbin/nginx -c /etc/nginx/nginx.conf"
p = 0xf8e8cc ""
size = 56
i = 3
n = 0
sigio = 0
set = {__val = {0 <repeats 16 times>}}
itv = {it_interval = {tv_sec = 0, tv_usec = 1}, it_value = {tv_sec = 7, tv_usec = 18}}
live = 1
delay = 0
ls = 0x6
ccf = 0xf6ff50
#17 0x0000000000404336 in main (argc=3, argv=0x7fff27f2f948) at src/core/nginx.c:412
i = 57
log = 0x89da40
cycle = 0xf6ef50
init_cycle = {conf_ctx = 0x0, pool = 0xf6ea20, log = 0x89da40, new_log = {log_level = 0, file = 0x0, connection = 0, handler = 0,
data = 0x0, action = 0x0}, files = 0x0, free_connections = 0x0, free_connection_n = 0, reusable_connections_queue = {prev = 0x0,
next = 0x0}, listening = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, paths = {elts = 0x0, nelts = 0, size = 0,
nalloc = 0, pool = 0x0}, open_files = {last = 0x0, part = {elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc = 0, pool = 0x0},
shared_memory = {last = 0x0, part = {elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc = 0, pool = 0x0}, connection_n = 0, files_n = 0,
connections = 0x0, read_events = 0x0, write_events = 0x0, old_cycle = 0x0, conf_file = {len = 21, data = 0x7fff27f2ff73 "ss"},
conf_param = {len = 0, data = 0x0}, conf_prefix = {len = 11, data = 0x7fff27f2ff73 "ss"}, prefix = {len = 12,
data = 0x5e1f0c "/etc/nginx//"}, lock_file = {len = 0, data = 0x0}, hostname = {len = 0, data = 0x0}}
ccf = 0xf6ff50
(gdb)
Linux hostname 2.6.32-279.el6.x86_64 #1 SMP Thu Jun 21 15:00:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/nginx/nginx.conf
user nginx;
worker_processes 1;
worker_rlimit_nofile 2048;
error_log /var/log/nginx/error.log debug;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
worker_rlimit_core 500M;
working_directory /tmp/dumps/;
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
access_log /var/log/nginx/access.log;
sendfile on;
keepalive_timeout 70;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
server {
listen 443 ssl spdy;
ssl_certificate ssl/ca.crt;
ssl_certificate_key ssl/ca.key;
location / {
#error_page 404 = /nginx/getRedirect;
eval_escalate on;
eval $auth_cookie_value {
set $memcached_key $host-$cookie_abc;
memcached_next_upstream error timeout invalid_response not_found;
memcached_pass 192.168.0.55:11211;
}
proxy_pass http://192.168.0.158:8081;
}
}
}
error.log:
2012/12/21 15:53:12 [notice] 16252#0: start worker process 16273
2012/12/21 15:53:22 [info] 16273#0: *20 SSL NPN ADVERT while SSL handshaking, client: 192.168.3.70, server:
2012/12/21 15:53:22 [notice] 16252#0: signal 17 (SIGCHLD) received
2012/12/21 15:53:22 [alert] 16252#0: worker process 16273 exited on signal 11 (core dumped)
2012/12/21 15:53:22 [notice] 16252#0: start worker process 16275
# /usr/sbin/nginx -V
nginx version: nginx/1.3.9
built by gcc 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC)
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx/ --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_gzip_static_module --with-http_random_index_module --with-http_stub_status_module --with-openssl=/tmp/openssl101-1.0.1c --with-cc-opt='-O0 -g -I/opt/openssl101/include' --with-ld-opt=-L/usr/openssl101/lib --add-module=/home/bld/rpmbuild/BUILD/nginx-1.3.9/nginx_eval_module-1.0.1
More information about the nginx-devel
mailing list