Strange Segfault

[Resicow]

Igor Sysoev wrote:
> On Tue, Sep 23, 2008 at 12:49:42PM -0500, Resicow wrote:
>
>   
>> Igor Sysoev wrote:
>>     
>>> On Tue, Sep 23, 2008 at 11:49:05AM -0500, Resicow wrote:
>>>
>>>  
>>>       
>>>> Hello,
>>>>
>>>> I am receiving a strange segault in dmesg every few hours... and then 
>>>> nginx orphans those connections, so after about a day there are 
>>>> 2000-3000 connections that are just "waiting".
>>>>         
>
> These orphan connections leave from segfaulted workers.
> This is wrong counter statistics only. It does not actually leak
> OS resources.
>
>   
>>>> This isn't that bad, as the server has received over 1,228,503 
>>>> connections since a restart this morning, but still those orphaned 
>>>> connections pile up, and I have to restart nginx to clear them.
>>>>
>>>> Here is what I am receiving:
>>>>
>>>> nginx[27715]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
>>>> error 4 in libkrb5.so.3.3[30d1000000+9f000]
>>>> nginx[27679]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
>>>> error 4 in libkrb5.so.3.3[30d1000000+9f000]
>>>> nginx[27690]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
>>>> error 4 in libkrb5.so.3.3[30d1000000+9f000]
>>>> nginx[27727]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
>>>> error 4 in libkrb5.so.3.3[30d1000000+9f000]
>>>> nginx[27728]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
>>>> error 4 in libkrb5.so.3.3[30d1000000+9f000]
>>>> nginx[27677]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
>>>> error 4 in libkrb5.so.3.3[30d1000000+9f000]
>>>> nginx[27706]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
>>>> error 4 in libkrb5.so.3.3[30d1000000+9f000]
>>>> nginx[27680]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
>>>> error 4 in libkrb5.so.3.3[30d1000000+9f000]
>>>> nginx[5214]: segfault at c ip 00000030d106c942 sp 00007fffcd340798 error 
>>>> 4 in libkrb5.so.3.3[30d1000000+9f000]
>>>> nginx[27686]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
>>>> error 4 in libkrb5.so.3.3[30d1000000+9f000]
>>>> nginx[27688]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
>>>> error 4 in libkrb5.so.3.3[30d1000000+9f000]
>>>> nginx[27678]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
>>>> error 4 in libkrb5.so.3.3[30d1000000+9f000]
>>>> nginx[27717]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
>>>> error 4 in libkrb5.so.3.3[30d1000000+9f000]
>>>> nginx[27682]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
>>>> error 4 in libkrb5.so.3.3[30d1000000+9f000]
>>>> nginx[27726]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
>>>> error 4 in libkrb5.so.3.3[30d1000000+9f000]
>>>> nginx[27724]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
>>>> error 4 in libkrb5.so.3.3[30d1000000+9f000]
>>>> __ratelimit: 34 callbacks suppressed
>>>> nginx[27720]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
>>>> error 4 in libkrb5.so.3.3[30d1000000+9f000]
>>>> nginx[22566]: segfault at c ip 00000030d106c942 sp 00007fffcd340798 
>>>> error 4 in libkrb5.so.3.3[30d1000000+9f000]
>>>> nginx[27685]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
>>>> error 4 in libkrb5.so.3.3[30d1000000+9f000]
>>>> nginx[22587]: segfault at c ip 00000030d106c942 sp 00007fffcd340798 
>>>> error 4 in libkrb5.so.3.3[30d1000000+9f000]
>>>> nginx[27722]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
>>>> error 4 in libkrb5.so.3.3[30d1000000+9f000]
>>>> nginx[27696]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
>>>> error 4 in libkrb5.so.3.3[30d1000000+9f000]
>>>> nginx[27691]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
>>>> error 4 in libkrb5.so.3.3[30d1000000+9f000]
>>>> nginx[27701]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
>>>> error 4 in libkrb5.so.3.3[30d1000000+9f000]
>>>> nginx[23722]: segfault at c ip 00000030d106c942 sp 00007fffcd340798 
>>>> error 4 in libkrb5.so.3.3[30d1000000+9f000]
>>>> nginx[22562]: segfault at c ip 00000030d106c942 sp 00007fffcd340798 
>>>> error 4 in libkrb5.so.3.3[30d1000000+9f000]
>>>> nginx[27719]: segfault at c ip 00000030d106c942 sp 00007fffcd340728 
>>>> error 4 in libkrb5.so.3.3[30d1000000+9f000]
>>>> nginx[22578]: segfault at c ip 00000030d106c942 sp 00007fffcd340798 
>>>> error 4 in libkrb5.so.3.3[30d1000000+9f000]
>>>> nginx[22560]: segfault at c ip 00000030d106c942 sp 00007fffcd340798 
>>>> error 4 in libkrb5.so.3.3[30d1000000+9f000]
>>>>
>>>> I know that libkrb5 is for kerberos... I don't think I am even using 
>>>> that in my configuration. No DAV support, etc... So I don't understand 
>>>> exactly why nginx is faulting.
>>>>
>>>> Please let me know what additional information I can send that would be 
>>>> helpful.
>>>>    
>>>>         
>>> What does "ldd /path/to/nginx" show ?
>>>
>>>  
>>>       
>> Here is the output:
>>
>> ldd /usr/local/nginx/sbin/nginx
>>        linux-vdso.so.1 =>  (0x00007fff745fe000)
>>        libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00000030d5c00000)
>>        libpcre.so.0 => /lib64/libpcre.so.0 (0x00000030d8800000)
>>        libssl.so.7 => /lib64/libssl.so.7 (0x00000030d3c00000)
>>        libcrypto.so.7 => /lib64/libcrypto.so.7 (0x00000030d0400000)
>>        libz.so.1 => /lib64/libz.so.1 (0x00000030c4800000)
>>        libc.so.6 => /lib64/libc.so.6 (0x00000030c3800000)
>>        libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2 
>> (0x00000030d2000000)
>>        libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x00000030d1000000)
>>        libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00000030d0000000)
>>        libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x00000030d0c00000)
>>        libdl.so.2 => /lib64/libdl.so.2 (0x00000030c4000000)
>>        /lib64/ld-linux-x86-64.so.2 (0x00000030c2600000)
>>        libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0 
>> (0x00000030d0800000)
>>        libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00000030d1400000)
>>        libresolv.so.2 => /lib64/libresolv.so.2 (0x00000030cac00000)
>>        libselinux.so.1 => /lib64/libselinux.so.1 (0x00000030c4c00000)
>>     
>
> It seems that librkb5 was linked via libssl/etc. You may try to do two things:
> 1) build nginx without SSL,
> 2) build nginx without stripping debug info and allow to create coredump.
>    Then I can invetsigate the bug.
>
>
>   

I'm glad that the orphaned connections do not use system resources... at 
least the system won't be taken down and run out of available connections.

The faults are very strange, since I have a replica setup on replica 
hardware, and it doesn't produce faults. I'll have some time in a few 
days to take the system down and create a coredump, but SSL support is 
required and used in my setup. Also the faults come in waves, which is 
also strange.

Both servers have the same version of libkrb5 as well... so maybe there 
is some client (or attacker) trying to create SSL sessions in a bad way.

Is there anyway to find out what "error 4" is in libkrb5.so.3? Should I 
upgrade openssl and then rebuild?

Thanks Again,

John