Proxying large downloads from s3

[Maxim Dounin]

Hello!

On Wed, Apr 02, 2014 at 05:05:36PM +0200, Sudara Williams wrote:

[...]

> With regards to early terminated / truncated large files — is this 
> something you or anyone else has seen before?
> 
> I'll see if I can get some better logging going on and report back. 
> Might be tough to correlate failed requests in production with log 
> entries, but I'll do my best :)

Upstream timeouts certainly may result in truncated responses sent 
to clients - if a timeout happens in the middle of a response, 
there is no way how nginx can handle this.

(The log line you provided is "... while reading a response header 
from upstream ..." though, and it should result in 504 returned to a 
client or next upstream tried if there are multiple upstream 
servers.)

Use of "chunked_transfer_encoding off;" is expected to make things 
worse, as it makes truncation undetectable if Content-Length isn't 
known.  It should not be used unless there are good reasons too - 
e.g., you have to support broken clients which use HTTP/1.1 but do 
not understand chunked transfer encoding (see 
http://nginx.org/r/chunked_transfer_encoding).

-- 
Maxim Dounin
http://nginx.org/