Worker process exited on signal 7

[Maxim Dounin]

Hello!

On Fri, Jan 07, 2011 at 10:35:31AM +0100, Balazs Pocze wrote:

> Hello everyone,
> 
> I started to use nginx insted of lighttpd for a few days ago. It
> serves only static content (thumbnail pictures) and performs well, but
> yesterday it stopped working, and I got the following messages on the
> error.log.
> 
> 2011/01/06 17:54:30 [alert] 7542#0: worker process 31959 exited on signal 7
> 
> I didn't found anything about this error message on the net.

It looks like signal 7 on linux x86 is SIGBUS (oh, linux has 
architecture-dependant signal numbers... really intresting design 
decision).

It usually generated on an attempt to access memory which doesn't 
exists, and may be generated on various hardware problems as well.
Under linux this includes uncorrectable memory errors as seen via 
MCA and problems reading pages after page fault (e.g. swap disk 
died, or mmap'ed file read failed).

> The error
> logging level is "crit" because I don't have enough storage to keep
> all the "file not found" errors :) (The logging goes to a tmpfs
> partiton).

Just a side note: you may want to use "log_not_found off;" instead.

> So I restarted the lighttpd on that server, and it works now, so I
> didn't think that it is a hardware error.
> Here is th the output of nginx -V:
> 
> nginx version: nginx/0.8.54
> built by gcc 4.3.3 (Ubuntu 4.3.3-5ubuntu4)
> TLS SNI support enabled
> configure arguments: --prefix=/opt/nginx
> --error-log-path=var/log/error.log --http-log-path=var/log/access.log
> --http-client-body-temp-path=/opt/nginx/var/cache/client_body_temp
> --http-proxy-temp-path=/opt/nginx/var/cache/proxy_temp
> --http-fastcgi-temp-path=/opt/nginx/var/cache/fastcgi_temp
> --without-http_uwsgi_module --without-http_scgi_module
> --with-http_ssl_module --with-openssl=../openssl-0.9.8k
> --user=www-data --group=www-data --pid-
> path=/opt/nginx/var/run/nginx.pid
> --conf-path=/opt/nginx/etc/nginx.conf --with-http_stub_status_module
> 
> It is a x86_64 system.
> 
> At this machine the www data files are on two SSD-s, and one of those
> SSD-s are bad, bacause not all 'secors' (hey what is the correct term
> for sectors on SSD?) can be read. Can it cause this type of error?

Obviously it can if you also have swap on these disks and/or read 
files via mmap().  While nginx doesn't use mmap() to read files, 
something like may be used internally by OS for e.g. sendfile(), 
not sure.

This may also be a problem in nginx which causes it to access 
nonexisting memory regions.

To further debug the problem it should be helpful to obtain 
coredump and backtrace, as well as show full config.  See 
http://wiki.nginx.org/Debugging for basic instructions.

Maxim Dounin