performance guide for nginx L4 stream

[solomon]

Hi,

Probably an issue associated with backlog and/or timewait. If the
backlog queue or the timewait bucket is full, the new connection request
will be dropped by tcp, before Nginx even accepting the connection, so you
can't see error log in Nginx.

To increase the backlog queue, you can increase the system tcp
configuration `net.core.somaxconn`, `net.ipv4.tcp_max_syn_backlog` and the
nginx config `listen backlog=xxxx`. The former is the system-level limit,
the latter is the process-level limit.

To avoid timewait bucket overflow, you can increase the system tcp
configuration `net.ipv4.tcp_max_tw_buckets` or enable the
`net.ipv4.tcp_tw_reuse`.

In addition, enlarging the ip/port range of the client side may also be
helpful.

BTW, you can use `ss` or `netstat` to observe the backlog of the listening
socket which corresponds to the `Send-Q` and `Recv-Q` fields of the output.
The kernel log can also gives you some information. If I remember
correctly, there is a kernel log when timewait bucket overflow occurs.

Hope it helps.

Reinis Rozitis <r at roze.lv> 于2022年10月27日周四 19:46写道：

> > We are using the hey (https://github.com/rakyll/hey) tool to pump 50k
> requests per second and are seeing only 40k requests being received on the
> backend application side.
> > Any other tcp configuration that needs to be tuned ?
>
> I am not familiar with the tool but per documentation it should have some
> sort of error status report for the failed requests What it is for the 10k
> "missing" requests?
>
> Are they "missing" (already) on nginx or just on the proxied backend(s)?
> (in the provided nginx configuration I don't see any access/error log
> configuration - you could enable both to see if you actually get those 50k
> requests to nginx).
>
> Are you testing from a single client (same server) or multiple?
> Do you use keepalive or new connection per request (in the case of later
> might come close to the ephemeral port limit (~65k) depending on if
> tcp_tw_reuse is or isn’t configured)?
>
> Have you tried with other tools like 'ab', 'httperf' or 'siege' to see if
> you get the same results/problems?
>
> rr
>
> _______________________________________________
> nginx mailing list -- nginx at nginx.org
> To unsubscribe send an email to nginx-leave at nginx.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20221104/1c1c06f5/attachment.htm>