Maybe a bug in Nginx 6.0.30 x86_64

François Battail fb at francois.battail.name
Sun May 11 13:04:02 MSD 2008


Hi All,

I've an issue with Nginx 0.6.30. So far it seems that the following
things are needed to reproduce this bug:

- keepalives enabled,
- Nginx running on Linux x86_64,
- many concurrent connections (> 300).

Nginx crashes on a SIGSEGV, but it looks like a stack overflow caused by
recursive calls according to gdb.

Here is the setup:

Ubuntu 8.04 64 bits, kernel 2.6.24
Amd 64 3000+ (single core), 512 MB

Nginx 0.6.30 configuration:

1 worker, no master process, no daemon, epoll, sendfile, worker
connections: 8000, 1 server serving a 10 kB index.html static file.

Compilation:

./configure --prefix=.

Then using Apache Bench like this:

ab -k -c 500 -n 1000000 http://localhost:8000/index.html

will cause a stack overflow (after ~24000 requests).

Using ab without the keepalive option (-k) works on the same computer
without problem so far even with a large number of simultaneous
connections ie:

ab -c 4000 -n 1000000 http://localhost:8000/index.html

Never could reproduce it on my main computer (Ubuntu 8.04 i386 32 bits).
I will try to do another tests on a q6600 (quad core) running Ubuntu
8.04 x86_64 and keep you informed.

A gdb session showing the first 500 stack backtraces is attached, hope
it will help.

Best regards

-------------- next part --------------
Starting program: /home/fb/bench/nginx 

Program received signal SIGSEGV, Segmentation fault.
ngx_http_auth_basic_handler (r=0xd2b600)
    at src/http/modules/ngx_http_auth_basic_module.c:96
96	{
#0  ngx_http_auth_basic_handler (r=0xd2b600)
    at src/http/modules/ngx_http_auth_basic_module.c:96
#1  0x0000000000422549 in ngx_http_core_access_phase (r=<value optimized out>, 
    ph=0x682498) at src/http/ngx_http_core_module.c:930
#2  0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#3  0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#4  0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#5  0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#6  0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#7  0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#8  0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#9  0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#10 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, ph=0x6824f8)
    at src/http/ngx_http_core_module.c:1024
#11 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#12 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#13 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#14 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#15 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#16 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#17 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#18 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#19 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, ph=0x6824f8)
    at src/http/ngx_http_core_module.c:1024
#20 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#21 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#22 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#23 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#24 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#25 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#26 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#27 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#28 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, ph=0x6824f8)
    at src/http/ngx_http_core_module.c:1024
#29 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#30 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#31 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#32 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#33 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#34 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#35 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#36 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#37 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, ph=0x6824f8)
    at src/http/ngx_http_core_module.c:1024
#38 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#39 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#40 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#41 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#42 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#43 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#44 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#45 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#46 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, ph=0x6824f8)
    at src/http/ngx_http_core_module.c:1024
#47 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#48 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#49 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#50 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#51 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#52 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#53 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#54 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#55 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, ph=0x6824f8)
    at src/http/ngx_http_core_module.c:1024
#56 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#57 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#58 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#59 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#60 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#61 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#62 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#63 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#64 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, ph=0x6824f8)
    at src/http/ngx_http_core_module.c:1024
#65 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#66 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#67 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#68 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#69 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#70 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#71 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#72 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#73 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, ph=0x6824f8)
    at src/http/ngx_http_core_module.c:1024
#74 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#75 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#76 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#77 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#78 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#79 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#80 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#81 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#82 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, ph=0x6824f8)
    at src/http/ngx_http_core_module.c:1024
#83 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#84 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#85 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#86 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#87 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#88 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#89 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#90 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#91 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, ph=0x6824f8)
    at src/http/ngx_http_core_module.c:1024
#92 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#93 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#94 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#95 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#96 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#97 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#98 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#99 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#100 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#101 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#102 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#103 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#104 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#105 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#106 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#107 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#108 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#109 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#110 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#111 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#112 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#113 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#114 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#115 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#116 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#117 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#118 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#119 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#120 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#121 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#122 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#123 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#124 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#125 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#126 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#127 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#128 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#129 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#130 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#131 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#132 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#133 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#134 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#135 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#136 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#137 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#138 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#139 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#140 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#141 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#142 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#143 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#144 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#145 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#146 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#147 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#148 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#149 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#150 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#151 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#152 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#153 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#154 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#155 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#156 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#157 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#158 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#159 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#160 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#161 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#162 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#163 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#164 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#165 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#166 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#167 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#168 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#169 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#170 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#171 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#172 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#173 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#174 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#175 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#176 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#177 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#178 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#179 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#180 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#181 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#182 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#183 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#184 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#185 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#186 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#187 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#188 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#189 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#190 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#191 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#192 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#193 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#194 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#195 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#196 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#197 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#198 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#199 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#200 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#201 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#202 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#203 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#204 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#205 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#206 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#207 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#208 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#209 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#210 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#211 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#212 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#213 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#214 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#215 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#216 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#217 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#218 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#219 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#220 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#221 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#222 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#223 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#224 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#225 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#226 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#227 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#228 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#229 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#230 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#231 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#232 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#233 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#234 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#235 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#236 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#237 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#238 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#239 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#240 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#241 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#242 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#243 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#244 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#245 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#246 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#247 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#248 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#249 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#250 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#251 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#252 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#253 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#254 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#255 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#256 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#257 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#258 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#259 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#260 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#261 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#262 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#263 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#264 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#265 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#266 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#267 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#268 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#269 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#270 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#271 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#272 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#273 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#274 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#275 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#276 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#277 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#278 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#279 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#280 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#281 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#282 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#283 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#284 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#285 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#286 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#287 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#288 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#289 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#290 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#291 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#292 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#293 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#294 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#295 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#296 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#297 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#298 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#299 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#300 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#301 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#302 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#303 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#304 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#305 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#306 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#307 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#308 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#309 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#310 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#311 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#312 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#313 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#314 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#315 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#316 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#317 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#318 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#319 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#320 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#321 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#322 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#323 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#324 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#325 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#326 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#327 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#328 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#329 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#330 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#331 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#332 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#333 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#334 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#335 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#336 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#337 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#338 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#339 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#340 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#341 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#342 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#343 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#344 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#345 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#346 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#347 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#348 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#349 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#350 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#351 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#352 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#353 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#354 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#355 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#356 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#357 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#358 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#359 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#360 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#361 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#362 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#363 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#364 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#365 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#366 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#367 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#368 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#369 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#370 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#371 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#372 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#373 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#374 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#375 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#376 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#377 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#378 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#379 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#380 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#381 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#382 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#383 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#384 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#385 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#386 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#387 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#388 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#389 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#390 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#391 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#392 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#393 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#394 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#395 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#396 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#397 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#398 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#399 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#400 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#401 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#402 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#403 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#404 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#405 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#406 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#407 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#408 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#409 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#410 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#411 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#412 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#413 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#414 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#415 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#416 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#417 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#418 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#419 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#420 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#421 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#422 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#423 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#424 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#425 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#426 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#427 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#428 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#429 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#430 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#431 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#432 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#433 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#434 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#435 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#436 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#437 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#438 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#439 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#440 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#441 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#442 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#443 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#444 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#445 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#446 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#447 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#448 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#449 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#450 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#451 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#452 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#453 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#454 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#455 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#456 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#457 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#458 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#459 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#460 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#461 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#462 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#463 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#464 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#465 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#466 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#467 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#468 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#469 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#470 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#471 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#472 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#473 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#474 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#475 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#476 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#477 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#478 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#479 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#480 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#481 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#482 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
#483 0x0000000000426ebc in ngx_http_process_request_line (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:755
#484 0x000000000042463f in ngx_http_init_request (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:451
#485 0x00000000004247d2 in ngx_http_keepalive_handler (rev=0x7fd655f7e090)
    at src/http/ngx_http_request.c:2291
#486 0x0000000000425a91 in ngx_http_finalize_request (r=<value optimized out>, 
    rc=<value optimized out>) at src/http/ngx_http_request.c:2183
#487 0x0000000000422c17 in ngx_http_core_content_phase (r=0xd2b600, 
    ph=0x6824f8) at src/http/ngx_http_core_module.c:1024
#488 0x000000000041e9bf in ngx_http_core_run_phases (r=0xd2b600)
    at src/http/ngx_http_core_module.c:736
#489 0x000000000041eaf5 in ngx_http_handler (r=0xd2b600)
    at src/http/ngx_http_core_module.c:719
#490 0x000000000042601c in ngx_http_process_request (r=0xd2b600)
    at src/http/ngx_http_request.c:1476
#491 0x0000000000426a88 in ngx_http_process_request_headers (
    rev=<value optimized out>) at src/http/ngx_http_request.c:947
Quit
The program is running.  Exit anyway? (y or n) 


More information about the nginx mailing list