Inquiry Regarding Handling of QUIC Connections During Nginx Reload

[上勾拳]

Hello,

I hope this email finds you well. I am writing to inquire about the status
of an issue I have encountered regarding the handling of QUIC connections
when Nginx is reloaded.

Recently, I delved into the Nginx eBPF implementation, specifically
examining how QUIC connection packets are handled, especially during Nginx
reloads. My focus was on ensuring that existing QUIC connection packets are
correctly routed to the appropriate worker after a reload, and the Nginx
eBPF prog have done this job perfectly.

However, I observed that during a reload, new QUIC connections might be
directed to the old worker. The underlying problem stems from the fact that
new QUIC connections fail to match the eBPF reuseport socket map. The
kernel default logic then routes UDP packets based on the hash UDP 4-tuple
in the reuseport group socket array. Since the old worker's listen FDs
persist in the reuseport group socket array (reuse->socks), there is a
possibility that the old worker may still be tasked with handling new QUIC
connections.

Given that the old worker should not process new requests, it results in
the old worker not responding to the QUIC new connection. Consequently,
clients have to endure the handshake timeout and retry the connection,
potentially encountering the old worker again, leading to an ineffective
cycle.

In my investigation, I came across a thread in the nginx-devel mailing list
[https://www.mail-archive.com/nginx-devel@nginx.org/msg10627.html], where
it was mentioned that there would be some work addressing this issue.

Considering my limited experience with eBPF, I propose a potential
solution. The eBPF program could maintain another eBPF map containing only
the listen sockets of the new worker. When the eBPF program calls
`bpf_sk_select_reuseport` and receives `-ENOENT`, it could utilize this new
eBPF map with the hash UDP 4-tuple to route the new QUIC connection to the
new worker. This approach aims to circumvent the kernel logic routing the
packet to the shutting down worker since removing the old worker's listen
socket from the reuseport group socket array seems unfeasible. Not sure
about whether this solution is a good idea and I also wonder if there are
other solutions for this. I would appreciate any insights or updates you
could provide on this matter. Thank you for your time and consideration.

Best regards,
Zhenzhong
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20240225/04dd8c30/attachment.htm>