HTTP3 speed tuning

[Jiří Setnička]

Hello,

I have been experimenting with the HTTP/3 performance (using nginx build 
from the current master branch) for the past several days and I am not 
satisfied with the performance. I was trying to fiddle firstly with the 
nginx settings, systems settings, and lately also with the algorithm 
parameters in the nginx code itself, but without success.

There are several benchmarks to present the problem.

I used the nginx build from the current master branch (git commit 
145b228530, hg revision 9270:d1b8568f3042) with the curl from Debian 
testing (with http3 support) or with a custom build of the curl using 
OpenSSL (same numbers were achieved with both builds of the curl, so I 
combined both measurements into one). The nginx.conf is attached.

The same server was used for all tests (Intel E5-2683 with 40Gbit NIC). 
Multiple clients were tried with similar results (Hetzner VPS, home 
server connected to 500Mbps network, laptop over WiFi). Results:

  * VPS:
      o iperf3: 1.68 Gbps
      o iperf3 --udp:1.66 Gbps
      o ping: 13 ms
      o avg download speed of 1GiB file served by nginx over HTTP/2:  
        214.4 MiB/s (1.7 Gbps)
      o avg download speed of 1GiB file served by nginx over HTTP/3:  
        4.82 MiB/s (38.5 Mbps)

  * Home server with 500 Mbit connection
      o iperf3: 498 Mbps
      o iperf3 --udp: 508 Mbps
      o ping: 6 ms
      o avg download speed of 1GiB file served by nginx over HTTP/2:  
        58.8 MiB/s (470 Mbps)
      o avg download speed of 1GiB file served by nginx over HTTP/3:  
        6.1 MiB/s (48.8 Mbps)

  * My laptop over WiFi on work network:
      o iperf3: 351 Mbps
      o iperf3 --udp: 395 Mbps
      o ping: 4 ms
      o avg download speed of 1GiB file served by nginx over HTTP/2:  
        40.8 MiB/s (326 Mbps)
      o avg download speed of 1GiB file served by nginx over HTTP/3:  
        13.3 MiB/s (106.4 Mbps)

Similar download speeds were achieved using browsers (Firefox 115, 
Chromium 126).

I was trying to to experiment with some directives firstly:

  * quic_bpf - no change, even using the eBPF worker sockets patch from
    Roman Arutyunyan
  * quic_gso - effect of this could be observed in tcpdumps or debug
    nginx log, but no effect on the speed

Also, I captured tcpdumps of this traffic and it seems like there are 
very few lost packets, so it shouldn't be caused by waiting for lost 
packets and retransmission of them. My thoughts were that there could be 
something wrong with the congestion control or the ACK processing, so I 
went roughly through the ngx_event_quic_ack.c code, trying to modify 
some of the constants, but also with no success in increasing the 
throughput of the HTTP/3 nginx implementation.

The strange thing I noticed in the debug log (also attached), is that 
there are multiple cycles over the same data ranges when the ACK frame 
is received, which is probably triggered by overlapping ACK ranges from 
the curl, but it yields into many times cycling over the same data. 
Besides that, I am not aware of any other performance penalty in the 
nginx code, which may yield in slow HTTP/3 speed.

I would like to solve this mystery and help with optimizing the HTTP/3 
performance, but I think I am at a dead end with my ideas. Any idea how 
to improve this would be appreciated. Thanks.

Jiří Setnička
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20240809/0e44e6cf/attachment-0001.htm>
-------------- next part --------------
daemon off;

worker_processes auto;
worker_cpu_affinity auto;

error_log /tmp/nginx-quic/error.log info;
pid /tmp/nginx-quic/nginx.pid;

quic_bpf on;

events {
    worker_connections 768;
}

http {
    error_log /tmp/nginx-quic/error.log info;
    access_log /tmp/nginx-quic/access.log;

    sendfile on;
    quic_gso on;

    client_body_temp_path  /tmp/nginx-quic/client_body_temp;
    proxy_temp_path        /tmp/nginx-quic/proxy_temp;
    fastcgi_temp_path      /tmp/nginx-quic/fastcgi_temp;
    uwsgi_temp_path        /tmp/nginx-quic/uwsgi_temp;
    scgi_temp_path         /tmp/nginx-quic/scgi_temp;

    server {
        # for better compatibility it's recommended
        # to use the same port for quic and https
        listen 443 quic reuseport;
        listen 443 ssl;

        http2 on;
        http3 on;

        ssl_certificate     /opt/mc/nginx_ssl/test_cert.pem;
        ssl_certificate_key /opt/mc/nginx_ssl/test_cert.key;

        root /tmp/nginx-quic/www;

        location / {
            # required for browsers to direct them to quic port
            add_header Alt-Svc 'h3=":443"; ma=86400';

            try_files $uri $uri/ =404;
        }
    }
}
-------------- next part --------------
A non-text attachment was scrubbed...
Name: debug_log.log
Type: text/x-log
Size: 183414 bytes
Desc: not available
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20240809/0e44e6cf/attachment-0001.bin>