nginx, php-fpm and 502 errors

[Jure Pečar]

Hello,

I'm trying to understand why some of our production nginx/php-fpm servers frequently return 502 errors. At that time "writev() failed (107: Transport endpoint is not connected) while sending request to upstream" is logged into error log.

Runnng strace -e connect,writev on nginx worker process frequently shows:

connect(75, {sa_family=AF_FILE, path="/tmp/php-fpm.sock"}, 110) = 0
writev(75, [{"\1\1\0\1\0\10\0\0\0\1\0\0\0\0\0\0\1\4\0\1\6\260\0\0\17"..., 1752}], 1) = 1752
connect(759, {sa_family=AF_FILE, path="/tmp/php-fpm.sock"}, 110) = 0
writev(759, [{"\1\1\0\1\0\10\0\0\0\1\0\0\0\0\0\0\1\4\0\1\5\377\1\0\17"..., 1576}], 1) = 1576
connect(940, {sa_family=AF_FILE, path="/tmp/php-fpm.sock"}, 110) = -1 EAGAIN (Resource temporarily unavailable)
connect(996, {sa_family=AF_FILE, path="/tmp/php-fpm.sock"}, 110) = -1 EAGAIN (Resource temporarily unavailable)
connect(391, {sa_family=AF_FILE, path="/tmp/php-fpm.sock"}, 110) = -1 EAGAIN (Resource temporarily unavailable)
connect(1120, {sa_family=AF_FILE, path="/tmp/php-fpm.sock"}, 110) = -1 EAGAIN (Resource temporarily unavailable)
writev(996, [{"\1\1\0\1\0\10\0\0\0\1\0\0\0\0\0\0\1\4\0\1\6O\1\0\0172S"..., 1656}], 1) = -1 ENOTCONN (Transport endpoint is not connected)
writev(758, [{"HTTP/1.1 502 Bad Gateway\r\nServer"..., 157}], 1) = 157
writev(940, [{"\1\1\0\1\0\10\0\0\0\1\0\0\0\0\0\0\1\4\0\1\6\320\0\0\17"..., 1784}], 1) = -1 ENOTCONN (Transport endpoint is not connected)
writev(658, [{"HTTP/1.1 502 Bad Gateway\r\nServer"..., 157}], 1) = 157

Could these EAGAIN from connect be related to ENOTCONN from writev?

The frequency of these increase if I decrease the number of php-cgi processes and vice-versa. If I have enough php-cgi processes, they do not occur at all. But this "enough" number is way too high for my taste (I ended with 64).

What exactly happens that writev looses connection in the middle of the writing?



-- 

Jure Pečar
http://jure.pecar.org