sendfile EAGAIN support broken?

[Maxim Dounin]

Hello!

On Fri, Dec 24, 2010 at 05:01:39PM -0500, steveh wrote:

> I've been trying figure out a problem we are seeing in passenger module
> where by the full POST data isnt passed to the underlying application.
> 
> After much digging I believe this may actually be a problem with the
> http_upstream / sendfile support in the core nginx code.
> 
> The following log snippet shows a trace of this happening:-
> 
> 2010/12/24 18:04:08 [debug] 27169#0: *21 connected
> 2010/12/24 18:04:08 [debug] 27169#0: *21 http upstream connect: 0
> 2010/12/24 18:04:08 [debug] 27169#0: *21 http upstream send request
> 2010/12/24 18:04:08 [debug] 27169#0: *21 chain writer buf fl:0 s:2405
> 2010/12/24 18:04:08 [debug] 27169#0: *21 chain writer buf fl:0 s:15939
> 2010/12/24 18:04:08 [debug] 27169#0: *21 chain writer buf fl:1
> s:23658449
> 2010/12/24 18:04:08 [debug] 27169#0: *21 chain writer in:
> 0000000801885F78
> 2010/12/24 18:04:08 [debug] 27169#0: *21 sendfile() sent only 18344
> bytes (35: Resource temporarily unavailable)
> 2010/12/24 18:04:08 [debug] 27169#0: *21 sendfile: -1, 1, 0, @0
> 18344:23658449
> 2010/12/24 18:04:08 [debug] 27169#0: *21 chain writer out:
> 0000000801885F98

Sendfile returned EAGAIN - it's somewhat normal, though exact 
number of bytes sent looks suspicious (two memory buffers, not a 
single byte from file).

> 2010/12/24 18:04:08 [debug] 27169#0: *21 event timer add: 23:
> 600000:1293214448446

Timeout on send to upstream set (proxy_send_timeout/fastcgi_send_timeout/...) 
set, it's 10 minutes.

> 2010/12/24 18:04:08 [debug] 27169#0: *21 kevent set event: 23: ft:-2
> fl:0025

Write event enabled and expected to be triggered by OS as long as 
write will be again possible on the connection in question.

> 2010/12/24 18:04:08 [debug] 27169#0: *21 http run request:
> "/uploadfile"
> 2010/12/24 18:04:08 [debug] 27169#0: *21 http upstream check client,
> write event:1, "/uploadfile"
> 2010/12/24 18:06:08 [debug] 27169#0: *21 http run request:
> "/uploadfile"
> 
> The last line is where the 2 minute request timeout triggered.

By "2 minute request timeout" you mean something in your client?
The only timeout nginx will maintain in this state is upstream 
send_timeout (just set to 10 mins).

> It seems that even though EAGAIN is returned there is never an attempt
> to try to send the remaining data.
> 
> Any ideas?

Looks like OS problem.  So the questions are: which OS? which 
filesystem used for client_body_temp_path?

Assuming OS is FreeBSD: make sure filesystem isn't tmpfs and/or 
zfs, both had known problems with sendfile().  These problems are 
believed to be fixed, but I doubt you're running recent enough 
9-current or 8-stable.

Removing passenger out of the picture for further debugging may be 
a good idea, too.

Maxim Dounin