nginx 1.0.4 crashes

[Igor Sysoev]

On Fri, Jul 15, 2011 at 10:28:14AM +0200, Q & A wrote:
> Hi,
> I'm experiencing some random nginx crashes. My server is Ubuntu 10.04,
> with glibc 2.11.1-0ubuntu7.8. Here is an excerpt from the log file:
> 
> 2011/07/15 05:15:14 [error] 3299#0: *2788234 open() "[...]" failed (2:
> No such file or directory), client: [...],
> server: [...], request: "GET /robots.txt HTTP/1.0", host: "[...]"
> *** glibc detected *** nginx: worker process: malloc(): memory
> corruption: 0x0000000000c15dd0 ***
> ======= Backtrace: =========
> /lib/libc.so.6(+0x775b6)[0x7fb125cf35b6]
> /lib/libc.so.6(+0x7b6d8)[0x7fb125cf76d8]
> /lib/libc.so.6(__libc_malloc+0x6e)[0x7fb125cf858e]
> nginx: worker process(ngx_alloc+0x1e)[0x430bfe]
> nginx: worker process[0x418405]
> nginx: worker process(ngx_pcalloc+0x16)[0x4185f6]
> nginx: worker process[0x44538e]
> nginx: worker process[0x436424]
> nginx: worker process(ngx_process_events_and_timers+0x5e)[0x42d54e]
> nginx: worker process[0x434ad5]
> nginx: worker process(ngx_spawn_process+0x194)[0x432de4]
> nginx: worker process[0x433f12]
> nginx: worker process(ngx_master_process_cycle+0x1b4)[0x435034]
> nginx: worker process(main+0x947)[0x417517]
> /lib/libc.so.6(__libc_start_main+0xfd)[0x7fb125c9ac4d]
> nginx: worker process[0x415d09]
> ======= Memory map: ========
> 00400000-004d1000 r-xp 00000000 09:01 16612
>   /usr/sbin/nginx
> 006d0000-006d1000 r--p 000d0000 09:01 16612
>   /usr/sbin/nginx
> 006d1000-006e6000 rw-p 000d1000 09:01 16612
>   /usr/sbin/nginx
> 006e6000-00e90000 rw-p 00000000 00:00 0                                  [heap]
> [...]
> 2011/07/15 07:29:59 [alert] 3294#0: worker process 3295 exited on
> signal 6 (core dumped)
> 2011/07/15 08:04:28 [alert] 3294#0: worker process 3299 exited on
> signal 11 (core dumped)
> 2011/07/15 08:04:59 [alert] 3294#0: worker process 3297 exited on
> signal 11 (core dumped)
> 2011/07/15 08:05:33 [alert] 3294#0: worker process 3298 exited on
> signal 11 (core dumped)
> 
> (also notice the times)
> 
> There were also different messages, like:
> *** glibc detected *** nginx: worker process: double free or
> corruption (!prev): 0x00000000009f9ad0 ***
> *** glibc detected *** nginx: worker process: free(): corrupted
> unsorted chunks: 0x00000000009872c0 ***
> 
> 
> After loading the last dumped core with gdb & doing "bt" I get this:
> 
> Core was generated by `nginx: worker pr'.
> Program terminated with signal 11, Segmentation fault.
> #0  0x000000000041ac42 in ngx_output_chain ()
> (gdb) bt
> #0  0x000000000041ac42 in ngx_output_chain ()
> #1  0x0000000000457723 in ?? ()
> #2  0x00000000004583dd in ?? ()
> #3  0x000000000044e965 in ngx_http_read_client_request_body ()
> #4  0x000000000047f7f9 in ?? ()
> #5  0x00000000004408d4 in ngx_http_core_content_phase ()
> #6  0x000000000043d0ed in ngx_http_core_run_phases ()
> #7  0x000000000043f6d5 in ngx_http_named_location ()
> #8  0x0000000000443467 in ngx_http_core_try_files_phase ()
> #9  0x000000000043d0ed in ngx_http_core_run_phases ()
> #10 0x0000000000447f25 in ?? ()
> #11 0x0000000000448bf2 in ?? ()
> #12 0x0000000000436424 in ?? ()
> #13 0x000000000042d54e in ngx_process_events_and_timers ()
> #14 0x0000000000434ad5 in ?? ()
> #15 0x0000000000432de4 in ngx_spawn_process ()
> #16 0x0000000000433f12 in ?? ()
> #17 0x0000000000435034 in ngx_master_process_cycle ()
> #18 0x0000000000417517 in main ()
> 
> 
> More info about my nginx version:
> nginx -V
> nginx: nginx version: nginx/1.0.4
> nginx: TLS SNI support enabled
> nginx: configure arguments: --prefix=/etc/nginx
> --conf-path=/etc/nginx/nginx.conf
> --error-log-path=/var/log/nginx/error.log
> --http-client-body-temp-path=/var/lib/nginx/body
> --http-fastcgi-temp-path=/var/lib/nginx/fastcgi
> --http-log-path=/var/log/nginx/access.log
> --http-proxy-temp-path=/var/lib/nginx/proxy
> --http-scgi-temp-path=/var/lib/nginx/scgi
> --http-uwsgi-temp-path=/var/lib/nginx/uwsgi
> --lock-path=/var/lock/nginx.lock --pid-path=/var/run/nginx.pid
> --with-debug --with-http_addition_module --with-http_dav_module
> --with-http_flv_module --with-http_geoip_module
> --with-http_gzip_static_module --with-http_image_filter_module
> --with-http_perl_module --with-http_random_index_module
> --with-http_realip_module --with-http_secure_link_module
> --with-http_stub_status_module --with-http_ssl_module
> --with-http_sub_module --with-http_xslt_module --with-ipv6
> --with-sha1=/usr/include/openssl --with-md5=/usr/include/openssl
> --with-mail --with-mail_ssl_module
> --add-module=/build/buildd/nginx-1.0.4/debian/modules/nginx-development-kit
> --add-module=/build/buildd/nginx-1.0.4/debian/modules/nginx-upstream-fair
> --add-module=/build/buildd/nginx-1.0.4/debian/modules/nginx-echo
> --add-module=/build/buildd/nginx-1.0.4/debian/modules/nginx-lua
> --add-module=/build/buildd/nginx-1.0.4/debian/modules/nginx-http-push
> --add-module=/build/buildd/nginx-1.0.4/debian/modules/nginx-upload-progress
> --add-module=/build/buildd/nginx-1.0.4/debian/modules/nginx-secure-download

Could you reproduce the crash without third-pary modules which are built
using --add-module ?


-- 
Igor Sysoev