nginx 1.0.4 crashes

[Q & A]

Hi,
I'm experiencing some random nginx crashes. My server is Ubuntu 10.04,
with glibc 2.11.1-0ubuntu7.8. Here is an excerpt from the log file:

2011/07/15 05:15:14 [error] 3299#0: *2788234 open() "[...]" failed (2:
No such file or directory), client: [...],
server: [...], request: "GET /robots.txt HTTP/1.0", host: "[...]"
*** glibc detected *** nginx: worker process: malloc(): memory
corruption: 0x0000000000c15dd0 ***
======= Backtrace: =========
/lib/libc.so.6(+0x775b6)[0x7fb125cf35b6]
/lib/libc.so.6(+0x7b6d8)[0x7fb125cf76d8]
/lib/libc.so.6(__libc_malloc+0x6e)[0x7fb125cf858e]
nginx: worker process(ngx_alloc+0x1e)[0x430bfe]
nginx: worker process[0x418405]
nginx: worker process(ngx_pcalloc+0x16)[0x4185f6]
nginx: worker process[0x44538e]
nginx: worker process[0x436424]
nginx: worker process(ngx_process_events_and_timers+0x5e)[0x42d54e]
nginx: worker process[0x434ad5]
nginx: worker process(ngx_spawn_process+0x194)[0x432de4]
nginx: worker process[0x433f12]
nginx: worker process(ngx_master_process_cycle+0x1b4)[0x435034]
nginx: worker process(main+0x947)[0x417517]
/lib/libc.so.6(__libc_start_main+0xfd)[0x7fb125c9ac4d]
nginx: worker process[0x415d09]
======= Memory map: ========
00400000-004d1000 r-xp 00000000 09:01 16612
  /usr/sbin/nginx
006d0000-006d1000 r--p 000d0000 09:01 16612
  /usr/sbin/nginx
006d1000-006e6000 rw-p 000d1000 09:01 16612
  /usr/sbin/nginx
006e6000-00e90000 rw-p 00000000 00:00 0                                  [heap]
[...]
2011/07/15 07:29:59 [alert] 3294#0: worker process 3295 exited on
signal 6 (core dumped)
2011/07/15 08:04:28 [alert] 3294#0: worker process 3299 exited on
signal 11 (core dumped)
2011/07/15 08:04:59 [alert] 3294#0: worker process 3297 exited on
signal 11 (core dumped)
2011/07/15 08:05:33 [alert] 3294#0: worker process 3298 exited on
signal 11 (core dumped)

(also notice the times)

There were also different messages, like:
*** glibc detected *** nginx: worker process: double free or
corruption (!prev): 0x00000000009f9ad0 ***
*** glibc detected *** nginx: worker process: free(): corrupted
unsorted chunks: 0x00000000009872c0 ***


After loading the last dumped core with gdb & doing "bt" I get this:

Core was generated by `nginx: worker pr'.
Program terminated with signal 11, Segmentation fault.
#0  0x000000000041ac42 in ngx_output_chain ()
(gdb) bt
#0  0x000000000041ac42 in ngx_output_chain ()
#1  0x0000000000457723 in ?? ()
#2  0x00000000004583dd in ?? ()
#3  0x000000000044e965 in ngx_http_read_client_request_body ()
#4  0x000000000047f7f9 in ?? ()
#5  0x00000000004408d4 in ngx_http_core_content_phase ()
#6  0x000000000043d0ed in ngx_http_core_run_phases ()
#7  0x000000000043f6d5 in ngx_http_named_location ()
#8  0x0000000000443467 in ngx_http_core_try_files_phase ()
#9  0x000000000043d0ed in ngx_http_core_run_phases ()
#10 0x0000000000447f25 in ?? ()
#11 0x0000000000448bf2 in ?? ()
#12 0x0000000000436424 in ?? ()
#13 0x000000000042d54e in ngx_process_events_and_timers ()
#14 0x0000000000434ad5 in ?? ()
#15 0x0000000000432de4 in ngx_spawn_process ()
#16 0x0000000000433f12 in ?? ()
#17 0x0000000000435034 in ngx_master_process_cycle ()
#18 0x0000000000417517 in main ()


More info about my nginx version:
nginx -V
nginx: nginx version: nginx/1.0.4
nginx: TLS SNI support enabled
nginx: configure arguments: --prefix=/etc/nginx
--conf-path=/etc/nginx/nginx.conf
--error-log-path=/var/log/nginx/error.log
--http-client-body-temp-path=/var/lib/nginx/body
--http-fastcgi-temp-path=/var/lib/nginx/fastcgi
--http-log-path=/var/log/nginx/access.log
--http-proxy-temp-path=/var/lib/nginx/proxy
--http-scgi-temp-path=/var/lib/nginx/scgi
--http-uwsgi-temp-path=/var/lib/nginx/uwsgi
--lock-path=/var/lock/nginx.lock --pid-path=/var/run/nginx.pid
--with-debug --with-http_addition_module --with-http_dav_module
--with-http_flv_module --with-http_geoip_module
--with-http_gzip_static_module --with-http_image_filter_module
--with-http_perl_module --with-http_random_index_module
--with-http_realip_module --with-http_secure_link_module
--with-http_stub_status_module --with-http_ssl_module
--with-http_sub_module --with-http_xslt_module --with-ipv6
--with-sha1=/usr/include/openssl --with-md5=/usr/include/openssl
--with-mail --with-mail_ssl_module
--add-module=/build/buildd/nginx-1.0.4/debian/modules/nginx-development-kit
--add-module=/build/buildd/nginx-1.0.4/debian/modules/nginx-upstream-fair
--add-module=/build/buildd/nginx-1.0.4/debian/modules/nginx-echo
--add-module=/build/buildd/nginx-1.0.4/debian/modules/nginx-lua
--add-module=/build/buildd/nginx-1.0.4/debian/modules/nginx-http-push
--add-module=/build/buildd/nginx-1.0.4/debian/modules/nginx-upload-progress
--add-module=/build/buildd/nginx-1.0.4/debian/modules/nginx-secure-download


The biggest problem is that when it happens, the server process is still alive,
but it no longer handles connections, i.e. when you connect, the
connection hangs infinitely.
It's necessary to restart nginx manually to make it work again. I
thought nginx had
a master/worker architecture to prevent such cases - a worker would
die, but the
rest should remain safe.

Anyway - is there any way to fix it? What can cause it?

-- 
Thanks
meris