Lost connection after reading 2147479552 bytes with sendfile

[Maxim Dounin]

Hello!

On Fri, Jul 10, 2015 at 06:01:30PM +0200, Mathias Andre wrote:

> I've bumped into a strange problem lately, and though I've found a
> workaround I thought I'd post here to try and understand what the
> underlying issue can be in case it affects anyone else.
> 
> The short version is:
> Running Nginx 1.9.2 on Scientific Linux 6 with a
> 2.6.32-504.8.1.el6.x86_64 kernel, nginx is configured to serve a large
> file (4GB) with sendfile enabled, the file download hangs exactly after
> 2147479552 bytes, after ~ 60s the connection is dropped (presumably
> because of the default send_timeout).
> 
> Setting sendfile_max_chunk to 1G fixes the problem

[...]

> I tried the same thing on an Ubuntu 14.04.2 without problem, however, in
> this case the sendfile system call always reads much fewer bytes at a
> time https://gist.githubusercontent.com/mathiasuk/018e0462d8788288d0d3/raw/error-ubuntu-14.04.02.log :
> 
> $ grep sendfile /tmp/error-ubuntu-14.04.02.log|head
> 2015/07/10 17:08:51 [debug] 15680#0: *1 sendfile: @0 2147479552
> 2015/07/10 17:08:51 [debug] 15680#0: *1 sendfile: 3557719 of 2147479552 @0
> 2015/07/10 17:08:51 [debug] 15680#0: *1 sendfile: @3557719 2147481257
> 2015/07/10 17:08:51 [debug] 15680#0: *1 sendfile: 7792477 of 2147481257 @3557719
> 2015/07/10 17:08:51 [debug] 15680#0: *1 sendfile: @11350196 2147483468
> 2015/07/10 17:08:51 [debug] 15680#0: *1 sendfile: 1833524 of 2147483468 @11350196
> 2015/07/10 17:08:51 [debug] 15680#0: *1 sendfile: @13183720 2147480856
> 2015/07/10 17:08:51 [debug] 15680#0: *1 sendfile: 1899007 of 2147480856 @13183720
> 2015/07/10 17:08:51 [debug] 15680#0: *1 sendfile: @15082727 2147482393
> 2015/07/10 17:08:51 [debug] 15680#0: *1 sendfile: 1964490 of 2147482393 @15082727
> 
> 
> Whereas on the SL6 box sendfile tries to read 2147479552 in one go and
> nginx then seems to stop there:
> 
> $ grep sendfile /tmp/error-sc6.5.log 
> 2015/07/10 17:02:22 [debug] 22749#0: *1 sendfile: @0 2147479552
> 2015/07/10 17:02:22 [debug] 22749#0: *1 sendfile: 2147479552 of 2147479552 @0

The 2147479552 is a limit applied by default to allow sendfile() 
to work with larger files on Linux up to 2.6.16 (see 
src/os/unix/ngx_linux_sendfile_chain.c for some comments).  You can see the 
same limit on the first sendfile() call in the Ubuntu log as well.

The strange thing here is that on Scientific Linux 6 the call 
pretends it send all the bytes in a single non-blocking call.  
This is not nginx expects to ever happen, and this is what causes 
the problem to appear.  It would be interesting to dig further to 
understand what causes this SL6 behaviour.

Using sendfile_max_chunk with some large value is a correct 
workaround and expected to work fine.

-- 
Maxim Dounin
http://nginx.org/