Nginx run into Stat D

[Maxim Dounin]

Hello!

On Fri, Sep 26, 2014 at 03:45:40PM +0800, Zhang Qiang wrote:

> Hi community,
> 
> We are using Nginx as static http server to cache/send static smal file.
> The average file size is about 10k ~ 200k -, and nginx server can process
> ~10000 request per second. It runs well but some times most of nginx worker
> go into Stat D. and there's no way to kill them but restart system.
> 
> Here's the call stack for kernel/user space:
> 
> With AIO enabled
> -----Kernel Space Call Stack-----
>  0xffffffff8111f700 : sync_page+0x0/0x50 [kernel]
>  0xffffffff8111f75e : sync_page_killable+0xe/0x40 [kernel]
>  0xffffffff81529e7a : __wait_on_bit_lock+0x5a/0xc0 [kernel]
>  0xffffffff8111f667 : __lock_page_killable+0x67/0x70 [kernel]
>  0xffffffff81121394 : generic_file_aio_read+0x4b4/0x700 [kernel]
>  0xffffffff811d58d4 : aio_rw_vect_retry+0x84/0x200 [kernel]
>  0xffffffff811d7294 : aio_run_iocb+0x64/0x170 [kernel]
>  0xffffffff811d86c1 : do_io_submit+0x291/0x920 [kernel]
>  0xffffffff811d8d60 : sys_io_submit+0x10/0x20 [kernel]
>  0xffffffff8100b288 : tracesys+0xd9/0xde [kernel]
> -----User Space Call Stack-----
>  0x3c362e50c9 : syscall+0x19/0x40 [/lib64/libc-2.12.so]
>  0x4d2232 : ngx_linux_sendfile_chain+0xc2a/0xc2c
> [/opt/soft/nginx/sbin/nginx]
>  0x4d24ea : ngx_file_aio_read+0x2b6/0x528 [/opt/soft/nginx/sbin/nginx]
>  0x515247 : ngx_http_file_cache_open+0xbef/0x1437
> [/opt/soft/nginx/sbin/nginx]
>  0x514df6 : ngx_http_file_cache_open+0x79e/0x1437
> [/opt/soft/nginx/sbin/nginx]
>  0x514abb : ngx_http_file_cache_open+0x463/0x1437
> [/opt/soft/nginx/sbin/nginx]

Just a side note: it looks like there are very wrong symbols shown 
in the stack, likely due to optimizations used.

[...]

> Is anyone see these issue (pending on Stat D) before? how can I resolve it?
> it seems  that not the kernel issue.

What makes you think so?  It looks like the kernel issue from here - 
"unkillable" proceses just can't happen due to userland code 
unless there is a problem in the kernel.

-- 
Maxim Dounin
http://nginx.org/