Worker process exited on signal 7

[Balazs Pocze]

2011/1/10 Maxim Dounin <mdounin at mdounin.ru>:
> Hello!
>
> On Mon, Jan 10, 2011 at 01:38:53PM +0100, Balazs Pocze wrote:
>
>> 2011/1/7 Maxim Dounin <mdounin at mdounin.ru>:
>> > Hello!
>> >
>> > On Fri, Jan 07, 2011 at 10:35:31AM +0100, Balazs Pocze wrote:
>> >
>> >> Hello everyone,
>> >>
>> >> I started to use nginx insted of lighttpd for a few days ago. It
>> >> serves only static content (thumbnail pictures) and performs well, but
>> >> yesterday it stopped working, and I got the following messages on the
>> >> error.log.
>> >>
>> >> 2011/01/06 17:54:30 [alert] 7542#0: worker process 31959 exited on signal 7
>> >>
>> >> I didn't found anything about this error message on the net.
>> >
>> > It looks like signal 7 on linux x86 is SIGBUS (oh, linux has
>> > architecture-dependant signal numbers... really intresting design
>> > decision).
>> >
>> > It usually generated on an attempt to access memory which doesn't
>> > exists, and may be generated on various hardware problems as well.
>> > Under linux this includes uncorrectable memory errors as seen via
>> > MCA and problems reading pages after page fault (e.g. swap disk
>> > died, or mmap'ed file read failed).
>> >
>> Maybe yes. But I have no proof, and the problem is that: with lighttpd
>> i don't have this errors. So I don't know what to think about it.
>
> This may be hardware-related error.  No errors in lighttpd may
> indicate it's lucky (in case of memory errors) or using other
> system interfaces (in case of paging-related problems).
>
yeah, it can be. I'll put an strace process to one of the workers, and
look into the logs later.

>> >> The error
>> >> logging level is "crit" because I don't have enough storage to keep
>> >> all the "file not found" errors :) (The logging goes to a tmpfs
>> >> partiton).
>> >
>> > Just a side note: you may want to use "log_not_found off;" instead.
>> >
>> It is not a good sollution for me, because I need the 404 errors. (for
>> fact, on this server I _only_ need the 404 errors to get (re)generated
>> the thumbnail pictures, I don't need these errors on error log, just
>> in access.log (and a feature will be welcome, if I can direct the 404
>> logs into one log file, and 200's to an another.)
>
> "log_not_found off;" does exactly what you need - it just disables
> logging "file not found" errors to error_log.
>

yep, I was wrong, you have right. thx.
my bad.

>> >> So I restarted the lighttpd on that server, and it works now, so I
>> >> didn't think that it is a hardware error.
>> >> Here is th the output of nginx -V:
>> >>
>> >> nginx version: nginx/0.8.54
>> >> built by gcc 4.3.3 (Ubuntu 4.3.3-5ubuntu4)
>> >> TLS SNI support enabled
>> >> configure arguments: --prefix=/opt/nginx
>> >> --error-log-path=var/log/error.log --http-log-path=var/log/access.log
>> >> --http-client-body-temp-path=/opt/nginx/var/cache/client_body_temp
>> >> --http-proxy-temp-path=/opt/nginx/var/cache/proxy_temp
>> >> --http-fastcgi-temp-path=/opt/nginx/var/cache/fastcgi_temp
>> >> --without-http_uwsgi_module --without-http_scgi_module
>> >> --with-http_ssl_module --with-openssl=../openssl-0.9.8k
>> >> --user=www-data --group=www-data --pid-
>> >> path=/opt/nginx/var/run/nginx.pid
>> >> --conf-path=/opt/nginx/etc/nginx.conf --with-http_stub_status_module
>> >>
>> >> It is a x86_64 system.
>> >>
>> >> At this machine the www data files are on two SSD-s, and one of those
>> >> SSD-s are bad, bacause not all 'secors' (hey what is the correct term
>> >> for sectors on SSD?) can be read. Can it cause this type of error?
>> >
>> > Obviously it can if you also have swap on these disks and/or read
>> > files via mmap().  While nginx doesn't use mmap() to read files,
>> > something like may be used internally by OS for e.g. sendfile(),
>> > not sure.
>> >
>> > This may also be a problem in nginx which causes it to access
>> > nonexisting memory regions.
>> >
>> > To further debug the problem it should be helpful to obtain
>> > coredump and backtrace, as well as show full config.  See
>> > http://wiki.nginx.org/Debugging for basic instructions.
>> >
>> there's no swap space on this machine.
>
> It's impossible to tell anything without further info.  Please
> follow provided instructions to obtain coredump.
>

yep I'll do that.

thx.

-- 
"Overhead, without any fuss, the stars were going out."