104: Connection reset by peer

Пт Мар 20 09:00:36 UTC 2020

nginx работает в качестве tcp lb
Периодически получаю 104: Connection reset by peer.
---
Если причинно следственная связь в системных вызовах?
writev() not ready (11: Resource temporarily unavailable)
recv() failed (104: Connection reset by peer)
или 104 ошибку получаем из-за того что не получили сообщений от сокета для
файлового дескриптора?
recv: fd:96 -1 of 16384
---
2020/03/20 04:54:21 [debug] 37456#0: *98446240 accept: 188.187.126.40:40513
fd:96
2020/03/20 04:54:21 [info] 37456#0: *98446240 client 188.187.126.40:40513
connected to 8.8.157.49:80
2020/03/20 04:54:21 [debug] 37456#0: *98446240 posix_memalign:
00005597192FA4F0:256 @16
2020/03/20 04:54:21 [debug] 37456#0: *98446240 generic phase: 0
2020/03/20 04:54:21 [debug] 37456#0: *98446240 generic phase: 1
2020/03/20 04:54:21 [debug] 37456#0: *98446240 generic phase: 2
2020/03/20 04:54:21 [debug] 37456#0: *98446240 ssl preread handler
2020/03/20 04:54:21 [debug] 37456#0: *98446240 tcp_nodelay
2020/03/20 04:54:21 [debug] 37456#0: *98446240 proxy connection handler
2020/03/20 04:54:21 [debug] 37456#0: *98446240 malloc: 00005597192FA600:416
2020/03/20 04:54:21 [debug] 37456#0: *98446240 malloc:
000055971947AB00:16384
2020/03/20 04:54:21 [debug] 37456#0: *98446240 get rr peer, try: 8
2020/03/20 04:54:21 [debug] 37456#0: *98446240 get rr peer, current:
000055971928E448 0
2020/03/20 04:54:21 [debug] 37456#0: *98446240 stream socket 97
2020/03/20 04:54:21 [debug] 37456#0: *98446240 epoll add connection: fd:97
ev:80002005
2020/03/20 04:54:21 [debug] 37456#0: *98446240 connect to
10.121.15.75:31001, fd:97 #98446241
2020/03/20 04:54:21 [debug] 37456#0: *98446240 proxy connect: -2
2020/03/20 04:54:21 [debug] 37456#0: *98446240 event timer add: 97:
1000:155123895
2020/03/20 04:54:21 [debug] 37456#0: *98446240 event timer del: 97:
155123895
2020/03/20 04:54:21 [debug] 37456#0: *98446240 stream proxy connect
upstream
2020/03/20 04:54:21 [debug] 37456#0: *98446240 tcp_nodelay
2020/03/20 04:54:21 [info] 37456#0: *98446240 proxy 10.121.15.65:3564
connected to 10.121.15.75:31001
2020/03/20 04:54:21 [debug] 37456#0: *98446240 malloc:
000055971947EB10:16384
2020/03/20 04:54:21 [debug] 37456#0: *98446240 stream proxy add PROXY
protocol header
2020/03/20 04:54:21 [debug] 37456#0: *98446240 posix_memalign:
00005597192FA7B0:256 @16
2020/03/20 04:54:21 [debug] 37456#0: *98446240 write new buf t:1 f:0
0000000000000000, pos 00005597192FA820, size: 51 file: 0, size: 0
2020/03/20 04:54:21 [debug] 37456#0: *98446240 stream write filter: l:0 f:0
s:51
2020/03/20 04:54:21 [debug] 37456#0: *98446240 writev: 51 of 51
2020/03/20 04:54:21 [debug] 37456#0: *98446240 stream write filter
0000000000000000
2020/03/20 04:54:21 [debug] 37456#0: *98446240 epoll add event: fd:96 op:1
ev:80002001
2020/03/20 04:54:21 [debug] 37456#0: *98446240 event timer add: 96:
600000:155722896
2020/03/20 04:54:21 [debug] 37456#0: *98446240 recv: eof:0, avail:1
2020/03/20 04:54:21 [debug] 37456#0: *98446240 recv: fd:96 259 of 16384
2020/03/20 04:54:21 [debug] 37456#0: *98446240 write new buf t:1 f:0
0000000000000000, pos 000055971947AB00, size: 259 file: 0, size: 0
2020/03/20 04:54:21 [debug] 37456#0: *98446240 stream write filter: l:0 f:1
s:259
2020/03/20 04:54:21 [debug] 37456#0: *98446240 writev: 259 of 259
2020/03/20 04:54:21 [debug] 37456#0: *98446240 stream write filter
0000000000000000
2020/03/20 04:54:21 [debug] 37456#0: *98446240 event timer: 96, old:
155722896, new: 155722896
2020/03/20 04:54:21 [debug] 37456#0: *98446240 recv: eof:0, avail:1
2020/03/20 04:54:21 [debug] 37456#0: *98446240 recv: fd:97 8388 of 16384
2020/03/20 04:54:21 [debug] 37456#0: *98446240 write new buf t:1 f:0
0000000000000000, pos 000055971947EB10, size: 8388 file: 0, size: 0
2020/03/20 04:54:21 [debug] 37456#0: *98446240 stream write filter: l:0 f:1
s:8388
2020/03/20 04:54:21 [debug] 37456#0: *98446240 writev: 8388 of 8388
2020/03/20 04:54:21 [debug] 37456#0: *98446240 stream write filter
0000000000000000
2020/03/20 04:54:21 [debug] 37456#0: *98446240 event timer: 96, old:
155722896, new: 155722897
2020/03/20 04:54:21 [debug] 37456#0: *98446240 event timer: 96, old:
155722896, new: 155722897
2020/03/20 04:54:21 [debug] 37456#0: *98446240 recv: eof:0, avail:1
2020/03/20 04:54:21 [debug] 37456#0: *98446240 recv: fd:97 5592 of 16384
2020/03/20 04:54:21 [debug] 37456#0: *98446240 write new buf t:1 f:0
0000000000000000, pos 000055971947EB10, size: 5592 file: 0, size: 0
2020/03/20 04:54:21 [debug] 37456#0: *98446240 stream write filter: l:0 f:1
s:5592
2020/03/20 04:54:21 [debug] 37456#0: *98446240 writev: 5592 of 5592
2020/03/20 04:54:21 [debug] 37456#0: *98446240 stream write filter
0000000000000000
2020/03/20 04:54:21 [debug] 37456#0: *98446240 event timer: 96, old:
155722896, new: 155722897
2020/03/20 04:54:21 [debug] 37456#0: *98446240 event timer: 96, old:
155722896, new: 155722897
2020/03/20 04:54:21 [debug] 37456#0: *98446240 recv: eof:0, avail:1
2020/03/20 04:54:21 [debug] 37456#0: *98446240 recv: fd:97 6990 of 16384
2020/03/20 04:54:21 [debug] 37456#0: *98446240 write new buf t:1 f:0
0000000000000000, pos 000055971947EB10, size: 6990 file: 0, size: 0
2020/03/20 04:54:21 [debug] 37456#0: *98446240 stream write filter: l:0 f:1
s:6990
2020/03/20 04:54:21 [debug] 37456#0: *98446240 writev: 6990 of 6990
2020/03/20 04:54:21 [debug] 37456#0: *98446240 stream write filter
0000000000000000
2020/03/20 04:54:21 [debug] 37456#0: *98446240 event timer: 96, old:
155722896, new: 155722897
2020/03/20 04:54:21 [debug] 37456#0: *98446240 event timer: 96, old:
155722896, new: 155722897
2020/03/20 04:54:21 [debug] 37456#0: *98446240 recv: eof:0, avail:1
2020/03/20 04:54:21 [debug] 37456#0: *98446240 recv: fd:97 6990 of 16384
2020/03/20 04:54:21 [debug] 37456#0: *98446240 write new buf t:1 f:0
0000000000000000, pos 000055971947EB10, size: 6990 file: 0, size: 0
2020/03/20 04:54:21 [debug] 37456#0: *98446240 stream write filter: l:0 f:1
s:6990
2020/03/20 04:54:21 [debug] 37456#0: *98446240 writev: 6990 of 6990
2020/03/20 04:54:21 [debug] 37456#0: *98446240 stream write filter
0000000000000000
2020/03/20 04:54:21 [debug] 37456#0: *98446240 event timer: 96, old:
155722896, new: 155722897
2020/03/20 04:54:21 [debug] 37456#0: *98446240 event timer: 96, old:
155722896, new: 155722897
2020/03/20 04:54:21 [debug] 37456#0: *98446240 recv: eof:0, avail:1
2020/03/20 04:54:21 [debug] 37456#0: *98446240 recv: fd:97 5592 of 16384
2020/03/20 04:54:21 [debug] 37456#0: *98446240 write new buf t:1 f:0
0000000000000000, pos 000055971947EB10, size: 5592 file: 0, size: 0
2020/03/20 04:54:21 [debug] 37456#0: *98446240 stream write filter: l:0 f:1
s:5592
2020/03/20 04:54:21 [debug] 37456#0: *98446240 writev: 4160 of 5592
2020/03/20 04:54:21 [debug] 37456#0: *98446240 writev: -1 of 1432
2020/03/20 04:54:21 [debug] 37456#0: *98446240 writev() not ready (11:
Resource temporarily unavailable)
2020/03/20 04:54:21 [debug] 37456#0: *98446240 stream write filter
00005597192FA890
2020/03/20 04:54:21 [debug] 37456#0: *98446240 epoll add event: fd:96 op:3
ev:80002005
2020/03/20 04:54:21 [debug] 37456#0: *98446240 event timer: 96, old:
155722896, new: 155722898
2020/03/20 04:54:21 [debug] 37456#0: *98446240 event timer: 96, old:
155722896, new: 155722898
2020/03/20 04:54:21 [debug] 37456#0: *98446240 recv: eof:0, avail:1
2020/03/20 04:54:21 [debug] 37456#0: *98446240 recv: fd:97 9786 of 10792
2020/03/20 04:54:21 [debug] 37456#0: *98446240 posix_memalign:
0000559719349B70:256 @16
2020/03/20 04:54:21 [debug] 37456#0: *98446240 write old buf t:1 f:0
0000000000000000, pos 000055971947FB50, size: 1432 file: 0, size: 0
2020/03/20 04:54:21 [debug] 37456#0: *98446240 write new buf t:1 f:0
0000000000000000, pos 00005597194800E8, size: 9786 file: 0, size: 0
2020/03/20 04:54:21 [debug] 37456#0: *98446240 stream write filter: l:0 f:1
s:11218
2020/03/20 04:54:21 [debug] 37456#0: *98446240 stream write filter
00005597192FA890
2020/03/20 04:54:21 [debug] 37456#0: *98446240 event timer: 96, old:
155722896, new: 155722898
2020/03/20 04:54:21 [debug] 37456#0: *98446240 event timer: 96, old:
155722896, new: 155722898
2020/03/20 04:54:21 [debug] 37456#0: *98446240 recv: eof:0, avail:1
2020/03/20 04:54:21 [debug] 37456#0: *98446240 recv: fd:97 1006 of 1006
2020/03/20 04:54:21 [debug] 37456#0: *98446240 write old buf t:1 f:0
0000000000000000, pos 000055971947FB50, size: 1432 file: 0, size: 0
2020/03/20 04:54:21 [debug] 37456#0: *98446240 write old buf t:1 f:0
0000000000000000, pos 00005597194800E8, size: 9786 file: 0, size: 0
2020/03/20 04:54:21 [debug] 37456#0: *98446240 write new buf t:1 f:0
0000000000000000, pos 0000559719482722, size: 1006 file: 0, size: 0
2020/03/20 04:54:21 [debug] 37456#0: *98446240 stream write filter: l:0 f:1
s:12224
2020/03/20 04:54:21 [debug] 37456#0: *98446240 stream write filter
00005597192FA890
2020/03/20 04:54:21 [debug] 37456#0: *98446240 event timer: 96, old:
155722896, new: 155722898
2020/03/20 04:54:21 [debug] 37456#0: *98446240 event timer: 96, old:
155722896, new: 155722898
2020/03/20 04:54:21 [debug] 37456#0: *98446240 event timer: 96, old:
155722896, new: 155722898
2020/03/20 04:54:21 [debug] 37456#0: *98446240 event timer: 96, old:
155722896, new: 155722898
2020/03/20 04:54:21 [debug] 37456#0: *98446240 recv: eof:1, avail:1
2020/03/20 04:54:21 [debug] 37456#0: *98446240 recv: fd:96 -1 of 16384
2020/03/20 04:54:21 [info] 37456#0: *98446240 recv() failed (104: Connection
reset by peer) while proxying and reading from client, client:
188.187.126.40, server: 8.8.157.49:80, upstream: "10.121.15.75:31001", bytes
from/to client:259/32120, bytes from/to upstream:44344/310
2020/03/20 04:54:21 [debug] 37456#0: *98446240 posix_memalign:
0000559719349C80:256 @16
2020/03/20 04:54:21 [debug] 37456#0: *98446240 write new buf t:0 f:0
0000000000000000, pos 000055971947AB00, size: 0 file: 0, size: 0
2020/03/20 04:54:21 [debug] 37456#0: *98446240 stream write filter: l:1 f:1
s:0
2020/03/20 04:54:21 [info] 37456#0: *98446240 client disconnected, bytes
from/to client:259/32120, bytes from/to upstream:44344/310
2020/03/20 04:54:21 [debug] 37456#0: *98446240 finalize stream proxy: 200
2020/03/20 04:54:21 [debug] 37456#0: *98446240 free rr peer 8 0
2020/03/20 04:54:21 [debug] 37456#0: *98446240 close stream proxy upstream
connection: 97
2020/03/20 04:54:21 [debug] 37456#0: *98446240 reusable connection: 0
2020/03/20 04:54:21 [debug] 37456#0: *98446240 finalize stream session: 200
2020/03/20 04:54:21 [debug] 37456#0: *98446240 stream log handler
2020/03/20 04:54:21 [debug] 37456#0: *98446240 close stream connection: 96
2020/03/20 04:54:21 [debug] 37456#0: *98446240 event timer del: 96:
155722896
2020/03/20 04:54:21 [debug] 37456#0: *98446240 reusable connection: 0
2020/03/20 04:54:21 [debug] 37456#0: *98446240 free: 000055971947EB10
2020/03/20 04:54:21 [debug] 37456#0: *98446240 free: 000055971947AB00
2020/03/20 04:54:21 [debug] 37456#0: *98446240 free: 00005597192FA600
2020/03/20 04:54:21 [debug] 37456#0: *98446240 free: 0000559719352190,
unused: 8
2020/03/20 04:54:21 [debug] 37456#0: *98446240 free: 00005597193522A0,
unused: 0
2020/03/20 04:54:21 [debug] 37456#0: *98446240 free: 00005597192FA4F0,
unused: 0
2020/03/20 04:54:21 [debug] 37456#0: *98446240 free: 00005597192FA7B0,
unused: 0
2020/03/20 04:54:21 [debug] 37456#0: *98446240 free: 0000559719349B70,
unused: 0
2020/03/20 04:54:21 [debug] 37456#0: *98446240 free: 0000559719349C80,
unused: 128
---
буду благодарен за объяснение для слабых.

Posted at Nginx Forum: https://forum.nginx.org/read.php?21,287396,287396#msg-287396