Workers CPU leak [epoll_wait,epoll_ctl]

[Maxim Dounin]

Hello!

On Wed, Mar 09, 2016 at 09:28:20AM -0500, vizl wrote:

> Debug log regarding to hanged  PID 7479 http://dev.vizl.org/debug.log.txt

This looks like a threads + sendfile() loop due to non-atomic updates of 
underlying file, similar to one recently reported on the Russian 
mailing list.

Correct solution would be to fix your system to update files 
atomically instead of overwriting them in place.

The patch below will resolve CPU hog and will log an alert 
instead:

# HG changeset patch
# User Maxim Dounin <mdounin at mdounin.ru>
# Date 1457536139 -10800
#      Wed Mar 09 18:08:59 2016 +0300
# Node ID e96e5dfe4ff8ffe301264c3eb2771596fae24d38
# Parent  93049710cb7f6ea91fa9bd707e88fbe79d82d0ef
Truncation detection in sendfile() on Linux.

This addresses connection hangs as observed in ticket #504, and
CPU hogs with "aio threads; sendfile on" as reported in the mailing list,
see http://mailman.nginx.org/pipermail/nginx-ru/2016-March/057638.html.

The alert is identical to one used on FreeBSD.

diff --git a/src/os/unix/ngx_linux_sendfile_chain.c b/src/os/unix/ngx_linux_sendfile_chain.c
--- a/src/os/unix/ngx_linux_sendfile_chain.c
+++ b/src/os/unix/ngx_linux_sendfile_chain.c
@@ -292,6 +292,19 @@ eintr:
         }
     }
 
+    if (n == 0) {
+        /*
+         * if sendfile returns zero, then someone has truncated the file,
+         * so the offset became beyond the end of the file
+         */
+
+        ngx_log_error(NGX_LOG_ALERT, c->log, 0,
+                      "sendfile() reported that \"%s\" was truncated at %O",
+                      file->file->name.data, file->file_pos);
+
+        return NGX_ERROR;
+    }
+
     ngx_log_debug3(NGX_LOG_DEBUG_EVENT, c->log, 0, "sendfile: %z of %uz @%O",
                    n, size, file->file_pos);
 
@@ -349,6 +362,19 @@ ngx_linux_sendfile_thread(ngx_connection
             return NGX_ERROR;
         }
 
+        if (ctx->err != NGX_AGAIN && ctx->sent == 0) {
+            /*
+             * if sendfile returns zero, then someone has truncated the file,
+             * so the offset became beyond the end of the file
+             */
+
+            ngx_log_error(NGX_LOG_ALERT, c->log, 0,
+                          "sendfile() reported that \"%s\" was truncated at %O",
+                          file->file->name.data, file->file_pos);
+ 
+            return NGX_ERROR;
+        }
+
         *sent = ctx->sent;
 
         return (ctx->sent == ctx->size) ? NGX_DONE : NGX_AGAIN;

-- 
Maxim Dounin
http://nginx.org/