nginx worker segfault, NULL pool

Sun Jun 10 10:49:36 UTC 2012

Hello Maxim,

On Sun, 10 June 2012 Maxim Dounin <mdounin at mdounin.ru> wrote:
> On Sat, Jun 09, 2012 at 04:54:01PM +0200, Bruno Prémont wrote:
> > On Sat, 09 June 2012 Maxim Dounin <mdounin at mdounin.ru> wrote:
> > > Anyway, please make sure you have aligment problems properly 
> > > reported by a kernel.  It looks like the linux kernel has an 
> > > unfortunate default to silently ignore alignment problems on arm, 
> > > which results in data corruption on unaligned accesses instead of 
> > > immediate exit on SIGBUS when unaligned access happens.  You may 
> > > get proper behaviour with
> > > 
> > > echo 4 > /proc/cpu/alignment 
> > > 
> > > This should allow to trace a root of your problems.
> > > 
> > > See http://lecs.cs.ucla.edu/wiki/index.php/XScale_alignment for 
> > > more details.
> > 
> > Thanks for the pointer, will read trough it!
> > 
> > Seems to be that one, after echoing 4 to /proc/cpu/alignment nginx
> > does not even start anymore (and `nginx -t` fails as well), each time with
> > SIGBUS.
> > 
> > e.g. for `nginx -t` the first SIGBUS happens at
> > 
> > #0  0x0000d64c in ngx_set_cpu_affinity (cf=0xbe892358, cmd=<optimized out>, conf=<optimized out>) at src/core/nginx.c:1275
> > #1  0x0001cafc in ngx_conf_handler (last=13909340, cf=0xbe892358) at src/core/ngx_conf_file.c:394
> > #2  ngx_conf_parse (cf=0xbe892358, filename=0xd43d70) at src/core/ngx_conf_file.c:244
> > #3  0x0001aba4 in ngx_init_cycle (old_cycle=0xbe8923c0) at src/core/ngx_cycle.c:268
> > #4  0x0000e29c in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:331
> > 
> > as backtraced with gdb.
> 
> Ok, this looks sensisble.
> 
> Could you please provide ./configure output and test if the 
> following patch fixes things for you?
> 
> diff --git a/auto/os/conf b/auto/os/conf
> --- a/auto/os/conf
> +++ b/auto/os/conf
> @@ -93,6 +93,7 @@ case "$NGX_MACHINE" in
>      ;;
>  
>      *)
> +        have=NGX_ALIGNMENT value=16 . auto/define
>          NGX_MACH_CACHE_LINE=32
>      ;;
>  

The patch seems to fix things, `nginx -t` does not die on SIGBUS anymore,
it also runs properly for the requests that made it fail (
/proc/cpu/alignment does not account any new alignment traps).

Thanks!
Bruno

Full configure output (as well as first few lines of make which shows
used CFLAGS -- compiler does not generate any warnings):

checking for OS
 + Linux 2.6.37-00003-g924cf4c armv5tel
checking for C compiler ... found
 + using GNU C compiler
checking for --with-ld-opt="-L/usr/lib" ... found
checking for gcc builtin atomic operations ... found
checking for C99 variadic macros ... found
checking for gcc variadic macros ... found
checking for unistd.h ... found
checking for inttypes.h ... found
checking for limits.h ... found
checking for sys/filio.h ... not found
checking for sys/param.h ... found
checking for sys/mount.h ... found
checking for sys/statvfs.h ... found
checking for crypt.h ... found
checking for Linux specific features
checking for epoll ... found
checking for sendfile() ... found
checking for sendfile64() ... found
checking for sys/prctl.h ... found
checking for prctl(PR_SET_DUMPABLE) ... found
checking for sched_setaffinity() ... found
checking for crypt_r() ... found
checking for sys/vfs.h ... found
checking for poll() ... found
checking for /dev/poll ... not found
checking for kqueue ... not found
checking for crypt() ... not found
checking for crypt() in libcrypt ... found
checking for F_READAHEAD ... not found
checking for posix_fadvise() ... found
checking for O_DIRECT ... found
checking for F_NOCACHE ... not found
checking for directio() ... not found
checking for statfs() ... found
checking for statvfs() ... found
checking for dlopen() ... not found
checking for dlopen() in libdl ... found
checking for sched_yield() ... found
checking for SO_SETFIB ... not found
checking for SO_ACCEPTFILTER ... not found
checking for TCP_DEFER_ACCEPT ... found
checking for TCP_KEEPIDLE, TCP_KEEPINTVL, TCP_KEEPCNT ... found
checking for TCP_INFO ... found
checking for accept4() ... found
checking for kqueue AIO support ... not found
checking for Linux AIO support ... found
checking for int size ... 4 bytes
checking for long size ... 4 bytes
checking for long long size ... 8 bytes
checking for void * size ... 4 bytes
checking for uint64_t ... found
checking for sig_atomic_t ... found
checking for sig_atomic_t size ... 4 bytes
checking for socklen_t ... found
checking for in_addr_t ... found
checking for in_port_t ... found
checking for rlim_t ... found
checking for uintptr_t ... uintptr_t found
checking for system endianess ... little endianess
checking for size_t size ... 4 bytes
checking for off_t size ... 8 bytes
checking for time_t size ... 4 bytes
checking for AF_INET6 ... found
checking for setproctitle() ... not found
checking for pread() ... found
checking for pwrite() ... found
checking for sys_nerr ... found
checking for localtime_r() ... found
checking for posix_memalign() ... found
checking for memalign() ... found
checking for mmap(MAP_ANON|MAP_SHARED) ... found
checking for mmap("/dev/zero", MAP_SHARED) ... found
checking for System V shared memory ... found
checking for POSIX semaphores ... not found
checking for POSIX semaphores in libpthread ... found
checking for struct msghdr.msg_control ... found
checking for ioctl(FIONBIO) ... found
checking for struct tm.tm_gmtoff ... found
checking for struct dirent.d_namlen ... not found
checking for struct dirent.d_type ... found
checking for sysconf(_SC_NPROCESSORS_ONLN) ... found
checking for openat(), fstatat() ... found
configuring additional modules
adding module in /var/tmp/portage/www-servers/nginx-1.2.1/work/agentzh-headers-more-nginx-module-3580526
 + ngx_http_headers_more_filter_module was configured
checking for PCRE library ... found
checking for PCRE JIT support ... found
checking for OpenSSL library ... found
checking for zlib library ... found
checking for libxslt ... found
checking for libexslt ... found
creating objs/Makefile

Configuration summary
  + using system PCRE library
  + using system OpenSSL library
  + md5: using OpenSSL library
  + sha1: using OpenSSL library
  + using system zlib library

  nginx path prefix: "/usr"  
  nginx binary file: "/usr/sbin/nginx"
  nginx configuration prefix: "/etc/nginx"
  nginx configuration file: "/etc/nginx/nginx.conf"
  nginx pid file: "/var/run/nginx.pid"
  nginx error log file: "/var/log/nginx/error_log"
  nginx http access log file: "/var/log/nginx/access_log"
  nginx http client request body temporary files: "/var/tmp/nginx/client"
  nginx http fastcgi temporary files: "/var/tmp/nginx/fastcgi"

make -j2 'LINK=armv5tel-softfloat-linux-gnueabi-gcc -Wl,-O1 -Wl,--as-needed' 'OTHERLDFLAGS=-Wl,-O1 -Wl,--as-needed'
make -f objs/Makefile
make[1]: Entering directory `/var/tmp/portage/www-servers/nginx-1.2.1/work/nginx-1.2.1'
armv5tel-softfloat-linux-gnueabi-gcc -c -O2 -march=armv5te -mtune=xscale -pipe -Wall -ggdb -I/usr/include -I src/core -I src/event -I src/event/modules -I src/os/unix -I /usr/include/libxml2 -I objs \
        -o objs/src/core/nginx.o \
        src/core/nginx.c
...