nginx cache exceeded, cache manager busy

Tue Mar 15 12:36:35 MSK 2011

Hi, we're experiencing exactly the same issue using the local filesystem as cache store. We had the same issue with nginx 0.7.67 and now with 0.8.54. The cache grows outside the specified boundaries and nginx process uses the 100% of a single core. 

I tried to strace the malfunctioning running process but all I got is tons of gettimeofday
gettimeofday({1300181425, 698292}, NULL) = 0
gettimeofday({1300181425, 698360}, NULL) = 0
gettimeofday({1300181425, 698410}, NULL) = 0
gettimeofday({1300181425, 698459}, NULL) = 0
gettimeofday({1300181425, 698508}, NULL) = 0
...

Are you using CentOS 5 64bit? Can you check nginx error log and see if you have crashed workers please?

Thanks.

--

nginx error log:

2011/03/12 09:09:09 [notice] 5979#0: signal 17 (SIGCHLD) received
2011/03/12 09:09:09 [alert] 5979#0: worker process 5988 exited on signal 11
2011/03/12 09:09:09 [notice] 5979#0: start worker process 29906
2011/03/12 09:09:09 [notice] 5979#0: signal 29 (SIGIO) received
*** glibc detected *** nginx: worker process: free(): invalid next size (normal): 0x00000000006e13c0 ***
======= Backtrace: =========
/lib64/libc.so.6[0x2b197d5f630f]
/lib64/libc.so.6(cfree+0x4b)[0x2b197d5f676b]
nginx: worker process[0x404ce4]
nginx: worker process[0x42701b]
nginx: worker process[0x427065]
nginx: worker process[0x42929f]
nginx: worker process[0x4346aa]
nginx: worker process[0x435056]
nginx: worker process[0x435130]
nginx: worker process[0x436c03]
nginx: worker process[0x435237]
nginx: worker process[0x41732e]
nginx: worker process[0x4171f0]
nginx: worker process[0x41c951]
nginx: worker process[0x41b2a7]
nginx: worker process[0x41d139]
nginx: worker process[0x40419a]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x2b197d5a1994]
nginx: worker process[0x4029b9]
======= Memory map: ========
00400000-00468000 r-xp 00000000 ca:02 5210299                            /usr/local/nginx/sbin/nginx
00668000-00675000 rw-p 00068000 ca:02 5210299                            /usr/local/nginx/sbin/nginx
00675000-0074d000 rw-p 00675000 00:00 0                                  [heap]
2b197c9ad000-2b197c9c9000 r-xp 00000000 ca:02 4047069                    /lib64/ld-2.5.so
2b197c9c9000-2b197c9ca000 rw-p 2b197c9c9000 00:00 0 
2b197c9d1000-2b197c9ff000 rw-p 2b197c9d1000 00:00 0 
2b197cbc8000-2b197cbc9000 r--p 0001b000 ca:02 4047069                    /lib64/ld-2.5.so
2b197cbc9000-2b197cbca000 rw-p 0001c000 ca:02 4047069                    /lib64/ld-2.5.so
2b197cbca000-2b197cbd3000 r-xp 00000000 ca:02 606255                     /lib64/libcrypt-2.5.so
2b197cbd3000-2b197cdd2000 ---p 00009000 ca:02 606255                     /lib64/libcrypt-2.5.so
2b197cdd2000-2b197cdd3000 r--p 00008000 ca:02 606255                     /lib64/libcrypt-2.5.so
2b197cdd3000-2b197cdd4000 rw-p 00009000 ca:02 606255                     /lib64/libcrypt-2.5.so
2b197cdd4000-2b197ce02000 rw-p 2b197cdd4000 00:00 0 
2b197ce02000-2b197ce1d000 r-xp 00000000 ca:02 606375                     /lib64/libpcre.so.0.0.1
2b197ce1d000-2b197d01d000 ---p 0001b000 ca:02 606375                     /lib64/libpcre.so.0.0.1
2b197d01d000-2b197d01e000 rw-p 0001b000 ca:02 606375                     /lib64/libpcre.so.0.0.1
2b197d01e000-2b197d14b000 r-xp 00000000 ca:02 4047961                    /lib64/libcrypto.so.0.9.8e
2b197d14b000-2b197d34a000 ---p 0012d000 ca:02 4047961                    /lib64/libcrypto.so.0.9.8e
2b197d34a000-2b197d36b000 rw-p 0012c000 ca:02 4047961                    /lib64/libcrypto.so.0.9.8e
2b197d36b000-2b197d370000 rw-p 2b197d36b000 00:00 0 
2b197d370000-2b197d384000 r-xp 00000000 ca:02 5033221                    /usr/lib64/libz.so.1.2.3
2b197d384000-2b197d583000 ---p 00014000 ca:02 5033221                    /usr/lib64/libz.so.1.2.3
2b197d583000-2b197d584000 rw-p 00013000 ca:02 5033221                    /usr/lib64/libz.so.1.2.3
2b197d584000-2b197d6d2000 r-xp 00000000 ca:02 606251                     /lib64/libc-2.5.so
2b197d6d2000-2b197d8d1000 ---p 0014e000 ca:02 606251                     /lib64/libc-2.5.so
2b197d8d1000-2b197d8d5000 r--p 0014d000 ca:02 606251                     /lib64/libc-2.5.so
2b197d8d5000-2b197d8d6000 rw-p 00151000 ca:02 606251                     /lib64/libc-2.5.so
2b197d8d6000-2b197d8db000 rw-p 2b197d8d6000 00:00 0 
2b197d8db000-2b197d8dd000 r-xp 00000000 ca:02 606257                     /lib64/libdl-2.5.so
2b197d8dd000-2b197dadd000 ---p 00002000 ca:02 606257                     /lib64/libdl-2.5.so
2b197dadd000-2b197dade000 r--p 00002000 ca:02 606257                     /lib64/libdl-2.5.so
2b197dade000-2b197dadf000 rw-p 00003000 ca:02 606257                     /lib64/libdl-2.5.so
2b197dadf000-2b197dae1000 rw-p 2b197dadf000 00:00 0 
2b197dae1000-2b197daeb000 r-xp 00000000 ca:02 606383                     /lib64/libnss_files-2.5.so
2b197daeb000-2b197dcea000 ---p 0000a000 ca:02 606383                     /lib64/libnss_files-2.5.so
2b197dcea000-2b197dceb000 r--p 00009000 ca:02 606383                     /lib64/libnss_files-2.5.so
2b197dceb000-2b197dcec000 rw-p 0000a000 ca:02 606383                     /lib64/libnss_files-2.5.so
2b197dcec000-2b198a4ec000 rw-s 00000000 00:08 1855417355                 /dev/zero (deleted)
2b198a4ec000-2b198acec000 rw-s 00000000 00:08 1855417356                 /dev/zero (deleted)
2b198acec000-2b198aced000 rw-s 00000000 00:08 1855417359                 /dev/zero (deleted)
2b198aced000-2b198acfa000 r-xp 00000000 ca:02 4047067                    /lib64/libgcc_s-4.1.2-20080825.so.1
2b198acfa000-2b198aefa000 ---p 0000d000 ca:02 4047067                    /lib64/libgcc_s-4.1.2-20080825.so.1
2b198aefa000-2b198aefb000 rw-p 0000d000 ca:02 4047067                    /lib64/libgcc_s-4.1.2-20080825.so.1
7fff2e0e7000-7fff2e0fd000 rw-p 7fff2e0e7000 00:00 0                      [stack]
ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0                  [vdso]

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,182995,183019#msg-183019