Nginx writing to Cephfs

[Valentin V. Bartenev]

On Friday 24 July 2015 09:55:04 vedranf wrote:
> Valentin V. Bartenev Wrote:
> -------------------------------------------------------
> > On Thursday 23 July 2015 14:51:58 vedranf wrote:
> > > Valentin V. Bartenev Wrote:
> > > 
> > > > It more looks like a bug in cephfs.  writev() should never return
> > > > ERESTARTSYS.
> > > 
> > > 
> > > I've talked to the ceph people, they say ERESTARTSYS shows up in strace
> > > output but it is handled by the kernel and that writev(2) is interrupted by
> > > the SIGALRM, which actually appears in the strace output just after writev
> > > fails.
> > > 
> > > I also failed to get this error by doing the same this as nginx using dd, dd
> > > always succeeded so it happens due to combination of nginx and cephfs.
> > > 
> > > Here's full strace output (2 examples from 2 differently configured
> > > servers):
> > > 
> > > http://pastebin.com/wUAAcdT7
> > > 
> > > http://pastebin.com/wHyWc9U5
> > > 
> > 
> > Do you have timer_resolution configured?
> 
> Yes, it's:
> 
> timer_resolution 50ms;
> 

This is the root cause of interrupts.  Every 50ms it signals nginx
and can interrupt any interruptible syscall (writing to file is
usually not, but it seems different for Cephfs).

You should avoid using timer_resolution, or try this patch:

diff -r be8d8b1dad78 src/os/unix/ngx_files.c
--- a/src/os/unix/ngx_files.c   Fri Jul 24 17:18:20 2015 +0300
+++ b/src/os/unix/ngx_files.c   Fri Jul 24 17:18:37 2015 +0300
@@ -264,6 +264,7 @@ ngx_write_chain_to_file(ngx_file_t *file
     u_char        *prev;
     size_t         size;
     ssize_t        total, n;
+    ngx_err_t      err;
     ngx_array_t    vec;
     struct iovec  *iov, iovs[NGX_IOVS];
 
@@ -335,10 +336,20 @@ ngx_write_chain_to_file(ngx_file_t *file
             file->sys_offset = offset;
         }
 
+eintr:
+
         n = writev(file->fd, vec.elts, vec.nelts);
 
         if (n == -1) {
-            ngx_log_error(NGX_LOG_CRIT, file->log, ngx_errno,
+            err = ngx_errno;
+
+            if (err == NGX_EINTR) {
+                ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, err,
+                               "writev() was interrupted");
+                goto eintr;
+            }
+
+            ngx_log_error(NGX_LOG_CRIT, file->log, err,
                           "writev() \"%s\" failed", file->name.data);
             return NGX_ERROR;
         }