<!DOCTYPE html>
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hello,</p>
<p>I have been experimenting with the HTTP/3 performance (using
nginx build from the current master branch) for the past several
days and I am not satisfied with the performance. I was trying to
fiddle firstly with the nginx settings, systems settings, and
lately also with the algorithm parameters in the nginx code
itself, but without success.<br>
</p>
<p>There are several benchmarks to present the problem.</p>
<p>I used the nginx build from the current master branch (git commit
145b228530, hg revision 9270:d1b8568f3042) with the curl from
Debian testing (with http3 support) or with a custom build of the
curl using OpenSSL (same numbers were achieved with both builds of
the curl, so I combined both measurements into one). The
nginx.conf is attached.</p>
<p>The same server was used for all tests (Intel E5-2683 with 40Gbit
NIC). Multiple clients were tried with similar results (Hetzner
VPS, home server connected to 500Mbps network, laptop over WiFi).
Results:</p>
<ul>
<li>VPS:<br>
</li>
<ul>
<li>iperf3: 1.68 Gbps</li>
<li>iperf3 --udp:1.66 Gbps<br>
</li>
<li>ping: 13 ms</li>
<li>avg download speed of 1GiB file served by nginx over HTTP/2:
214.4 MiB/s (1.7 Gbps)</li>
<li>avg download speed of 1GiB file served by nginx over HTTP/3:
4.82 MiB/s (38.5 Mbps)<br>
<br>
</li>
</ul>
<li>Home server with 500 Mbit connection</li>
<ul>
<li>iperf3: 498 Mbps<br>
</li>
<li>iperf3 --udp: 508 Mbps<br>
</li>
<li>ping: 6 ms</li>
<li>avg download speed of 1GiB file served by nginx over HTTP/2:
58.8 MiB/s (470 Mbps)</li>
<li>avg download speed of 1GiB file served by nginx over HTTP/3:
6.1 MiB/s (48.8 Mbps)<br>
<br>
</li>
</ul>
<li>My laptop over WiFi on work network:</li>
<ul>
<li>iperf3: 351 Mbps<br>
</li>
<li>iperf3 --udp: 395 Mbps</li>
<li>ping: 4 ms</li>
<li>avg download speed of 1GiB file served by nginx over HTTP/2:
40.8 MiB/s (326 Mbps)</li>
<li>avg download speed of 1GiB file served by nginx over HTTP/3:
13.3 MiB/s (106.4 Mbps)</li>
</ul>
</ul>
<p>Similar download speeds were achieved using browsers (Firefox
115, Chromium 126).<br>
</p>
<p>I was trying to to experiment with some directives firstly:</p>
<ul>
<li>quic_bpf - no change, even using the eBPF worker sockets patch
from Roman Arutyunyan</li>
<li>quic_gso - effect of this could be observed in tcpdumps or
debug nginx log, but no effect on the speed</li>
</ul>
<p>Also, I captured tcpdumps of this traffic and it seems like there
are very few lost packets, so it shouldn't be caused by waiting
for lost packets and retransmission of them. My thoughts were that
there could be something wrong with the congestion control or the
ACK processing, so I went roughly through the ngx_event_quic_ack.c
code, trying to modify some of the constants, but also with no
success in increasing the throughput of the HTTP/3 nginx
implementation.</p>
<p>The strange thing I noticed in the debug log (also attached), is
that there are multiple cycles over the same data ranges when the
ACK frame is received, which is probably triggered by overlapping
ACK ranges from the curl, but it yields into many times cycling
over the same data. Besides that, I am not aware of any other
performance penalty in the nginx code, which may yield in slow
HTTP/3 speed.</p>
<p>I would like to solve this mystery and help with optimizing the
HTTP/3 performance, but I think I am at a dead end with my ideas.
Any idea how to improve this would be appreciated. Thanks.<br>
</p>
<p>Jiří Setnička<br>
</p>
</body>
</html>