accept() failed (53: Software caused connection abort)

[Adam PAPAI]

Maxim Dounin wrote:
> Hello!
>
> On Wed, Jul 13, 2011 at 11:07:03AM +0200, Adam PAPAI wrote:
>
>> Dear List,
>>
>> We've switched to Nginx because it rocks!
>>
>> I have a problem with a really heavy loaded webserver:
>>
>> It has 100,000,000 request per day and this number is increasing.
>> (Probably 200-300M would be the peak) We are using it with php-fcgi
>> backend.
>>
>> We have 2 serious constant error messages, can be found below  [1]
>>
>> [1] http://pastebin.com/6nQuEqkT
>>
>> writev() failed (54: Connection reset by peer)
>> accept() failed (53: Software caused connection abort)
>>
>> When this error occurs, the 50x page is shown OR totally disconnects
>> and "could not connect to server" happens.
>>
>> Our system is a:
>>
>> FreeBSD iridium 8.2-RELEASE FreeBSD 8.2-RELEASE #0: Thu Feb 17
>> 02:41:51 UTC 2011
>> root at mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  amd64
>>
>> HP Proliant G7,
>> - 32GB memory
>> - CPU: Intel(R) Xeon(R) CPU  @ 2.40GHz x 6 ( x 2 ) = 24 CPU
>> - SAS disks
>>
>> What do the error messages above mean? Has it problem with the
>> nginx, or the PHP, or kernel limits? (TCP limits?)
>
> In no particular order:
>
> 2011/07/13 10:20:58 [error] 38061#0: accept() failed (53: Software caused connection abort)
>
> Client closed connection before nginx was able to accept() it.
> This may be normal (i.e. user just closed a browser page while images
> was still loading) and may be not (i.e. nginx wasn't able to
> accept() for a long time due to some problems and client bored
> waiting and closed a page).
>
> Some number of such messages are expected to appear.  High number
> of such errors may indicate problems, try looking at listen queues
> at first (netstat -Lan).
>
> 2011/07/13 10:26:56 [error] 38059#0: *73523 writev() failed (54: Connection reset by peer) while sending request to upstream ...
>
> Your upstream reset connection.  This is a problem with your
> backend (most likely it just died).  Inspect backend logs.
>
> 2011/07/13 10:31:33 [error] 38060#0: *113640 writev() failed (32: Broken pipe) while sending request to upstream ...
>
> The same as above, but in slightly different moment.
>
>> What should I tuning? What should I set up?
>
>> From here it looks like you have problems with backend(s).


netstat -Lan shows:

Current listen queue sizes (qlen/incqlen/maxqlen)
Proto Listen         Local Address
tcp4  0/0/4096       *.80
tcp4  27/0/128       127.0.0.1.9002
tcp4  32/0/128       127.0.0.1.9001
tcp4  26/0/128       127.0.0.1.9000
tcp4  0/0/128        *.2818
tcp6  0/0/128        *.2818
tcp4  0/0/10         127.0.0.1.25
Some tcp sockets may have been created.
unix  0/0/4          /var/run/devd.pipe

but sometimes the qlen over 128 for the 3 php backend. It should be the 
problem I guess.. hmmm.

-- 
Adam PAPAI
Szoftverfejlesztési igazgató
Grapes Communication Ltd.
http://www.grapes.hu
E-mail: wooh at wooh.hu
Phone: +36 30 33-55-735 (Hungary)