Strange Segfault

[Igor Sysoev]

On Tue, Sep 23, 2008 at 12:49:42PM -0500, Resicow wrote:

> Igor Sysoev wrote:
> >On Tue, Sep 23, 2008 at 11:49:05AM -0500, Resicow wrote:
> >
> >  
> >>Hello,
> >>
> >>I am receiving a strange segault in dmesg every few hours... and then 
> >>nginx orphans those connections, so after about a day there are 
> >>2000-3000 connections that are just "waiting".

These orphan connections leave from segfaulted workers.
This is wrong counter statistics only. It does not actually leak
OS resources.

> >>This isn't that bad, as the server has received over 1,228,503 
> >>connections since a restart this morning, but still those orphaned 
> >>connections pile up, and I have to restart nginx to clear them.
> >>
> >>Here is what I am receiving:
> >>
> >>nginx[27715]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
> >>error 4 in libkrb5.so.3.3[30d1000000+9f000]
> >>nginx[27679]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
> >>error 4 in libkrb5.so.3.3[30d1000000+9f000]
> >>nginx[27690]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
> >>error 4 in libkrb5.so.3.3[30d1000000+9f000]
> >>nginx[27727]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
> >>error 4 in libkrb5.so.3.3[30d1000000+9f000]
> >>nginx[27728]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
> >>error 4 in libkrb5.so.3.3[30d1000000+9f000]
> >>nginx[27677]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
> >>error 4 in libkrb5.so.3.3[30d1000000+9f000]
> >>nginx[27706]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
> >>error 4 in libkrb5.so.3.3[30d1000000+9f000]
> >>nginx[27680]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
> >>error 4 in libkrb5.so.3.3[30d1000000+9f000]
> >>nginx[5214]: segfault at c ip 00000030d106c942 sp 00007fffcd340798 error 
> >>4 in libkrb5.so.3.3[30d1000000+9f000]
> >>nginx[27686]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
> >>error 4 in libkrb5.so.3.3[30d1000000+9f000]
> >>nginx[27688]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
> >>error 4 in libkrb5.so.3.3[30d1000000+9f000]
> >>nginx[27678]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
> >>error 4 in libkrb5.so.3.3[30d1000000+9f000]
> >>nginx[27717]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
> >>error 4 in libkrb5.so.3.3[30d1000000+9f000]
> >>nginx[27682]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
> >>error 4 in libkrb5.so.3.3[30d1000000+9f000]
> >>nginx[27726]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
> >>error 4 in libkrb5.so.3.3[30d1000000+9f000]
> >>nginx[27724]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
> >>error 4 in libkrb5.so.3.3[30d1000000+9f000]
> >>__ratelimit: 34 callbacks suppressed
> >>nginx[27720]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
> >>error 4 in libkrb5.so.3.3[30d1000000+9f000]
> >>nginx[22566]: segfault at c ip 00000030d106c942 sp 00007fffcd340798 
> >>error 4 in libkrb5.so.3.3[30d1000000+9f000]
> >>nginx[27685]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
> >>error 4 in libkrb5.so.3.3[30d1000000+9f000]
> >>nginx[22587]: segfault at c ip 00000030d106c942 sp 00007fffcd340798 
> >>error 4 in libkrb5.so.3.3[30d1000000+9f000]
> >>nginx[27722]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
> >>error 4 in libkrb5.so.3.3[30d1000000+9f000]
> >>nginx[27696]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
> >>error 4 in libkrb5.so.3.3[30d1000000+9f000]
> >>nginx[27691]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
> >>error 4 in libkrb5.so.3.3[30d1000000+9f000]
> >>nginx[27701]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
> >>error 4 in libkrb5.so.3.3[30d1000000+9f000]
> >>nginx[23722]: segfault at c ip 00000030d106c942 sp 00007fffcd340798 
> >>error 4 in libkrb5.so.3.3[30d1000000+9f000]
> >>nginx[22562]: segfault at c ip 00000030d106c942 sp 00007fffcd340798 
> >>error 4 in libkrb5.so.3.3[30d1000000+9f000]
> >>nginx[27719]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
> >>error 4 in libkrb5.so.3.3[30d1000000+9f000]
> >>nginx[22578]: segfault at c ip 00000030d106c942 sp 00007fffcd340798 
> >>error 4 in libkrb5.so.3.3[30d1000000+9f000]
> >>nginx[22560]: segfault at c ip 00000030d106c942 sp 00007fffcd340798 
> >>error 4 in libkrb5.so.3.3[30d1000000+9f000]
> >>
> >>I know that libkrb5 is for kerberos... I don't think I am even using 
> >>that in my configuration. No DAV support, etc... So I don't understand 
> >>exactly why nginx is faulting.
> >>
> >>Please let me know what additional information I can send that would be 
> >>helpful.
> >>    
> >
> >What does "ldd /path/to/nginx" show ?
> >
> >  
> Here is the output:
> 
> ldd /usr/local/nginx/sbin/nginx
>        linux-vdso.so.1 =>  (0x00007fff745fe000)
>        libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00000030d5c00000)
>        libpcre.so.0 => /lib64/libpcre.so.0 (0x00000030d8800000)
>        libssl.so.7 => /lib64/libssl.so.7 (0x00000030d3c00000)
>        libcrypto.so.7 => /lib64/libcrypto.so.7 (0x00000030d0400000)
>        libz.so.1 => /lib64/libz.so.1 (0x00000030c4800000)
>        libc.so.6 => /lib64/libc.so.6 (0x00000030c3800000)
>        libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2 
> (0x00000030d2000000)
>        libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x00000030d1000000)
>        libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00000030d0000000)
>        libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x00000030d0c00000)
>        libdl.so.2 => /lib64/libdl.so.2 (0x00000030c4000000)
>        /lib64/ld-linux-x86-64.so.2 (0x00000030c2600000)
>        libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0 
> (0x00000030d0800000)
>        libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00000030d1400000)
>        libresolv.so.2 => /lib64/libresolv.so.2 (0x00000030cac00000)
>        libselinux.so.1 => /lib64/libselinux.so.1 (0x00000030c4c00000)

It seems that librkb5 was linked via libssl/etc. You may try to do two things:
1) build nginx without SSL,
2) build nginx without stripping debug info and allow to create coredump.
   Then I can invetsigate the bug.


-- 
Igor Sysoev
http://sysoev.ru/en/