Too many workers conenction on udp stream

[Maxim Dounin]

Hello!

On Tue, Jan 02, 2018 at 02:42:53PM -0300, Agus wrote:

> Hi guys,
> 
> I have an nginx proxying udp/streams to another proxy which handles the
> connection to the backend.
> 
> The same proxy proxying the udp streams to another proxy is working ok.
> But when it proxies it to the other one, it fills with the worker error. I
> turned on debugging and what i see, is that nginx aint releasing the udp
> connections...
> I could use a hand  as I cant get it to work.
> 
> in the first proxy i have:
> 
>   server {
>     listen *:8330 udp;
>     proxy_responses 1;
>     proxy_pass second-proxy:8330;
>     error_log /var/log/nginx/8330udp.log debug;
>   }
> 
> 
> 
> in the second that is the main which receives from various proxies:
> 
> server {
>     listen *:8330 udp;
>     proxy_responses 1;
>     proxy_pass server:8302;
>     error_log /var/log/nginx/udp8330.log debug;
>   }
> 
> 
> This same config in another "third" proxy for a differnet set of backends
> works ok.
> 
> 
> 
> The main proxy for the working requests logs is like this, it is ending the
> connections:
> 
> 2018/01/02 17:08:01 [debug] 6158#6158: *13 recv: fd:70 183 of 16384
> 2018/01/02 17:08:01 [debug] 6158#6158: *13 write new buf t:1 f:0
> 00000000012A6D00, pos 00000000012A7270, size: 183 file: 0, size: 0
> 2018/01/02 17:08:01 [debug] 6158#6158: *13 stream write filter: l:1 f:1
> s:183
> 2018/01/02 17:08:01 [debug] 6158#6158: *13 sendmsg: 183 of 183
> 2018/01/02 17:08:01 [debug] 6158#6158: *13 stream write filter
> 0000000000000000
> 2018/01/02 17:08:01 [info] 6158#6158: *13 udp upstream disconnected, bytes
> from/to client:122/183, bytes from/to upstream:183/122

Here nginx got an UDP response, and based on "proxy_responses 1" 
in your configuration closes the session.

[...]

> The same proxy for the other non working one is: NO finalize, nor closing
> connection
> 2018/01/02 17:06:30 [debug] 6101#6101: *291 recvmsg: 52.200.231.253:13129
> fd:51 n:313
> 2018/01/02 17:06:30 [info] 6101#6101: *291 udp client 52.200.231.253:13129
> connected to 0.0.0.0:8330

[...]

> 2018/01/02 17:06:30 [info] 6101#6101: *291 udp proxy 10.13.11.74:48173
> connected to 52.44.235.174:8330
> 2018/01/02 17:06:30 [debug] 6101#6101: *291 malloc: 00000000025DE7D0:16384
> 2018/01/02 17:06:30 [debug] 6101#6101: *291 stream proxy add preread
> buffer: 313
> 2018/01/02 17:06:30 [debug] 6101#6101: *291 posix_memalign:
> 00000000025E27E0:256 @16
> 2018/01/02 17:06:30 [debug] 6101#6101: *291 write new buf t:1 f:0
> 00000000025DE2C0, pos 00000000025DE2C0, size: 313 file: 0, size: 0
> 2018/01/02 17:06:30 [debug] 6101#6101: *291 stream write filter: l:1 f:1
> s:313
> 2018/01/02 17:06:30 [debug] 6101#6101: *291 sendmsg: 313 of 313
> 2018/01/02 17:06:30 [debug] 6101#6101: *291 stream write filter
> 0000000000000000
> 2018/01/02 17:06:30 [debug] 6101#6101: *291 event timer add: 51:
> 600000:1514913390811
> 2018/01/02 17:06:30 [debug] 6101#6101: *291 event timer: 51, old:
> 1514913390811, new: 1514913390811
> 2018/01/02 17:06:30 [debug] 6101#6101: *291 event timer: 51, old:
> 1514913390811, new: 1514913390811

Here nginx got a new UDP client, sent the packet to the upstream 
server and started to wait for a response.  Once the response is 
received, nginx will close the session much like in the above 
case.

How long nginx will wait for a response can be controlled using 
the "proxy_timeout" directive"

http://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_timeout

In your configuration it seems to be set to 600 seconds, which is 
10 times longer than the default.  If you want nginx to better 
tolerate non-responding UDP backends, you may want to configure 
shorter timeouts instead.  Alternatively, consider configuring 
more worker connections.

-- 
Maxim Dounin
http://mdounin.ru/