[NGINX] cache process manager consume a lot of CPU at reload and has to be restarted

[Maxim Dounin]

Hello!

On Wed, Apr 06, 2011 at 05:55:58PM +0200, Srebrenko Šehić wrote:

> 2011/4/5 Srebrenko Šehić <ssehic at gmail.com>:
> 
> > I will try to get a gdb trace of running wild process.
> 
> Some more info about this problem. I had cache manager in a spinlock
> using 100% CPU. I did a ktrace (this is on OpenBSD 4.8/amd64 running
> 0.8.50) and here is the output (edited for brevity):
>
> # ps auxww | grep 28055 | grep -v grep
> _proxy   28055 99.1  1.9 646012 39096 ??  R/0    2:28PM   75:07.06
> nginx: cache manager process (nginx)
> 
> # kdump -R | head -50
>  28055 nginx    1302104178.942729 EMUL  "native"
>  28055 nginx    0.000051 RET   gettimeofday 0
>  28055 nginx    0.000031 CALL  gettimeofday(0x7f7ffffea2a0,0)
>  28055 nginx    0.000011 RET   gettimeofday 0

No, this isn't not in spinlock.  While looping in spinlock nginx 
only runs and calls sched_yeild(), nothing more.  This is another 
problem.

[...]

> I also found out that setting "proxy_cache_path .... max_size=10k" (a
> ridiculously small number) while there is traffic on the virtualhost
> triggers the problem even *without* nginx receiving a SIGHUP.

I was able to reproduce the problem.  With max_size=0 it starts 
hogging cpu right on startup.

Try the attached patch.  It's against 0.9.7, but probably will 
apply cleanly or with minor modifications to 0.8.* as well.

Maxim Dounin
-------------- next part --------------
# HG changeset patch
# User Maxim Dounin <mdounin at mdounin.ru>
# Date 1302113238 -14400
# Node ID de5be49305c6ae5c71ffbed522d9feb383e2d81a
# Parent  c4e0de226b0b4292c120f85ef3d509df1d6db47f
Cache: fix cpu hog in cache manager.

If cache manager can't make any progress (e.g. due to locked entries, or
max_size being effectively set to 0, or no memory) it was looping in
ngx_http_file_cache_manager() consuming cpu.  Make sure to loop only if
some progress was made.

diff --git a/src/http/ngx_http_file_cache.c b/src/http/ngx_http_file_cache.c
--- a/src/http/ngx_http_file_cache.c
+++ b/src/http/ngx_http_file_cache.c
@@ -1027,6 +1027,7 @@ ngx_http_file_cache_forced_expire(ngx_ht
 
         if (fcn->count == 0) {
             ngx_http_file_cache_delete(cache, q, name);
+            wait = 0;
 
         } else {
             if (--tries) {
@@ -1191,7 +1192,7 @@ ngx_http_file_cache_manager(void *data)
     ngx_http_file_cache_t  *cache = data;
 
     off_t   size;
-    time_t  next;
+    time_t  next, wait;
 
     next = ngx_http_file_cache_expire(cache);
 
@@ -1212,7 +1213,11 @@ ngx_http_file_cache_manager(void *data)
             return next;
         }
 
-        next = ngx_http_file_cache_forced_expire(cache);
+        wait = ngx_http_file_cache_forced_expire(cache);
+
+        if (wait > 0) {
+            return wait;
+        }
 
         if (ngx_http_file_cache_manager_sleep(cache) != NGX_OK) {
             return next;