client wrongly closed keepalive connection problem

[Maxim Dounin]

Hello!

On Mon, Nov 28, 2011 at 10:17:07AM -0500, freshsun wrote:

> Hi all, we are some college students maintaining a linux distribution
> mirror. We are not experienced but now meet with
> this problem.
> There is a pxe system in our college to help every computer boot a linux
> system or install one on Ethernet. Now most users 
> cannot install, cause most of the packages will time out when being
> downloaded.
> We used lftp to mirror the directories, trying to find out the problem.
> On this directory (on our mirror with same content)
> http://ftp.debian.org/debian/pool/main/a/abiword/
> lftp hints:
> 
> <--- Accept-Ranges: bytes
> <--- 
> ---> HEAD /debian/pool/main/a/abiword/abiword_2.8.2.orig.tar.gz
> HTTP/1.1
> ---> Host: mirrors.ustc.edu.cn
> ---> User-Agent: lftp/4.0.6
> ---> Accept: */*
> ---> Connection: keep-alive
> ---> 
> <--- HTTP/1.1 200 OK
> **** recv: <input-socket>: Connection reset by peer
> ---- Closing HTTP connection
> 
> sometimes might be 
> "send:<output-socket>: Connection reset by peer"
> 
> and the nginx error.log shows"[info] 8864#0: *1661715 client xxxx closed
> keepalive connection"
> 
> After trying mirror with lftp many times, we found that after request
> around 100 files in this directory,
> the server starts to send data back. And around receiving each 15-17
> files, the reset error will show once,
> until the server start to send out bigger files in this directory.
> 
> At first we suspect that it might because there are too many small files
> at the start in this directory,
> but when we move this directory to another host and mirror it, it is
> just ok.

It looks like lftp uses pipelining excessively, you may try 
setting

    lingering_close always;

to see if it helps (available in nginx 1.1.0+, 1.0.6+).

Maxim Dounin