Running nginx-quic in a real environment

[Sergey Kandaurov]

> On 28 Sep 2022, at 01:04, Maxim Dounin <mdounin at mdounin.ru> wrote:
> 
> Hello!
> 
> On Tue, Sep 27, 2022 at 04:05:54PM +0400, Sergey Kandaurov wrote:
> 
>>> On 27 Sep 2022, at 14:11, João Sousa Andrade <joao.andrade at protocol.ai> wrote:
>>> 
>>> Thank you for the clarification Sergey. 
>>> 
>>> We have been running http3 in production for the past couple of weeks. There's something we have noticed which I'm not entirely sure as to what is causing it.
>>> 
>>> We have been getting lots of errors of the form: "[error] 34#34: *338736 quic no available client ids for new path while handling decrypted packet, client: $IP, server: 0.0.0.0:443". I tried looking through the code to little avail. I'm wondering what's causing these errors. Is it something which could be tweaked through configuration?
>> 
>> This happens when QUIC connection continues over a new network path,
>> known as QUIC connection migration, which should by done by switching
>> to a new connection ID, but the client didn't previously supply enough
>> Connection ID to chose from.  Normally both peers maintain a set of
>> unused Connection IDs, which may be needed not only for migration,
>> but also due to NAT rebinding.  Assuming that peers that initiate
>> connection migration maintain enough connection IDs, a likely reason
>> of the network path change failure as seen in the above error can be
>> due to NAT rebinding with the client implementation that doesn't a notion
>> of connection migration so it didn't sent NEW_CONNECTION_ID frames.
>> In the case of NAT rebinding the server should send probing frames
>> over a new network path using next available client Connection ID,
>> but there were no any, as seen in the above error.
>> Hope that helps.
> 

At least Firefox doesn't send NEW_CONNECTION_ID, which normally
happens after handshake completion, and, as additionally received
in another, private report, this leads to such errors behind NAT.

> Shouldn't it be at the "info" level, much like other 
> client-related errors?
> 

So indeed it may have sense to change logging level.

# HG changeset patch
# User Sergey Kandaurov <pluknet at nginx.com>
# Date 1664459599 -14400
#      Thu Sep 29 17:53:19 2022 +0400
# Branch quic
# Node ID 7d78208f141b382a55bea3f7c1e66471b0c53937
# Parent  a931e690475ee59387af517de60845b4b4307d28
QUIC: "info" logging level on insufficient client connection ids.

Apparently, this error is reported on NAT rebinding if client didn't
previously send NEW_CONNECTION_ID to supply additional connection ids.

diff --git a/src/event/quic/ngx_event_quic_migration.c b/src/event/quic/ngx_event_quic_migration.c
--- a/src/event/quic/ngx_event_quic_migration.c
+++ b/src/event/quic/ngx_event_quic_migration.c
@@ -309,7 +309,7 @@ ngx_quic_set_path(ngx_connection_t *c, n
     /* new path requires new client id */
     cid = ngx_quic_next_client_id(c);
     if (cid == NULL) {
-        ngx_log_error(NGX_LOG_ERR, c->log, 0,
+        ngx_log_error(NGX_LOG_INFO, c->log, 0,
                       "quic no available client ids for new path");
         /* stop processing of this datagram */
         return NGX_DONE;

-- 
Sergey Kandaurov