From a.bavshin at nginx.com Wed Feb 1 01:36:55 2023 From: a.bavshin at nginx.com (=?iso-8859-1?q?Aleksei_Bavshin?=) Date: Tue, 31 Jan 2023 17:36:55 -0800 Subject: [PATCH 0 of 6] Upstream: re-resolvable servers. Message-ID: The series is a compilation of patches with the upstream re-resolve feature from the Nginx Plus. The original commits were rebased on top of the current OSS code, grouped by features introduced and squashed. Some formatting quirks and other minor oddities could be attributed to a conscious effort to reduce divergence with the source branch. The last couple of patches in the series is a new code that allows sharing name resolution tasks between all the workers. Known issues and TODOs: === - The whole series is known to be broken on win32 with multiple worker processes, as it relies on the ngx_worker value to keep track of the locality of data. Initializing ngx_worker to a correct value should address that. 'noreuse' zones also seem to be unsupported on this platform, so configuration reload may fail. - The functionality requires shared zone of a sufficient size to be configured in the upstream block. A rough estimation is 2k for a configured server entry + 2k for each resolved address. The zone requirement could be lifted with local allocation of the resolved peer data, but implementing that was out of scope. - Resolved peer addresses are not carried over to a new generation of workers during configuration reload (see below). - Tests still require some cleanup and will be published later. Peer list population delay === In the cases of a cold start, a reload or a binary upgrade, the upstreams that contain only resolvable servers will have an empty list of peers. This leads to a short delay before Nginx is able to send the traffic to upstream. There's no perfect solution for that: if the server list in the configuration has changed, it's no longer compatible with the data we collected for a previous config. If the resolver parameters were modified, we may get an entirely different set of servers. The following options were considered: - Publishing the preresolve code from the Nginx Plus as is. The solution involves copying peer states from the non-reusable zone of a previous generation of workers. This only addresses the reload case and may result in a stale peer data if the configuration changes. The advantage of this code is that it is heavily tested and has been running in multiple production environments for many years. - Sharing the zone between all generations of workers. This requires some changes in the code, notably improving reference counting and cleanup for peer data in the shared zone (as we're no longer able to discard the old zone with all the allocated data) and tracking the upstream configuration compatibility. It also doesn't work when the zone size has changed in the config. The approach leads to increased memory requirements: zone size should be configured to accomodate multiple generations of workers, and we are aware of deployments that have lots of those due to long-living connections. Nginx OSS does not offer any means to monitor shared memory usage at the moment, so I fear this approach will hurt a lot of unsuspecting users who haven't reserved enough memory. There are also performance concerns, as access to the same list of peers from multiple generations of workers would increase lock contention (and the situation is already not looking well with round-robin lb). We can copy the peers instead of attempting to reuse, but that prevents us from optimizing the memory usage. - Queueing the requests until we finish the initial cycle of name resolution ('queue' directive of the ngx_http_upstream_module). This option adds a latency spike at the moment of configuration reload. There's also an issue with propagation of the upstream readiness state to all the worker processes - we need an event passing channel to be able to resume queued requests immediately. On the positive side, this would mitigate downtime for all 3 scenarios, as long as the queue capacity is sufficient. Given the latency spike, it doesn't seem to be a good standalone solution. But it might be a nice addition to one of the options above. Alternatives like pre-resolving servers during configuration load were not considered due to complexity and significant disadvantages. Maxim, from the list archives I understand that you had a negative opinion on the current approach with noreuse zones and pre-resolve, but I'm afraid there wasn't enough context to understand all the sides of that discussion. I'd appreciate if you share your thoughts on the problem and on the approach you consider architecturally correct. From a.bavshin at nginx.com Wed Feb 1 01:36:56 2023 From: a.bavshin at nginx.com (=?iso-8859-1?q?Aleksei_Bavshin?=) Date: Tue, 31 Jan 2023 17:36:56 -0800 Subject: [PATCH 1 of 6] Upstream: re-resolvable servers In-Reply-To: References: Message-ID: # HG changeset patch # User Ruslan Ermilov # Date 1392462754 -14400 # Sat Feb 15 15:12:34 2014 +0400 # Node ID b6e13aa0330c8434add55c88598480016b80fc0e # Parent 106328a70f4ecb32f828d33e5cd66c861e455f92 Upstream: re-resolvable servers. Specifying the upstream server by a hostname together with the "resolve" parameter will make the hostname to be periodically resolved, and upstream servers added/removed as necessary. This requires a "resolver" at the "http" configuration block. The "resolver_timeout" parameter also affects when the failed DNS requests will be attempted again. Responses with NXDOMAIN will be attempted again in 10 seconds. Upstream has a configuration generation number that is incremented each time servers are added/removed to the primary/backup list. This number is remembered by the peer.init method, and if peer.get detects a change in configuration, it returns NGX_BUSY. Each server has a reference counter. It is incremented by peer.get and decremented by peer.free. When a server is removed, it is removed from the list of servers and is marked as "zombie". The memory allocated by a zombie peer is freed only when its reference count becomes zero. Re-resolvable servers utilize timers that also hold a reference. A reference is also held while upstream keepalive caches an idle connection. Co-authored-by: Roman Arutyunyan Co-authored-by: Sergey Kandaurov Co-authored-by: Vladimir Homutov diff --git a/src/http/modules/ngx_http_upstream_hash_module.c b/src/http/modules/ngx_http_upstream_hash_module.c --- a/src/http/modules/ngx_http_upstream_hash_module.c +++ b/src/http/modules/ngx_http_upstream_hash_module.c @@ -24,6 +24,9 @@ typedef struct { typedef struct { ngx_http_complex_value_t key; +#if (NGX_HTTP_UPSTREAM_ZONE) + ngx_uint_t config; +#endif ngx_http_upstream_chash_points_t *points; } ngx_http_upstream_hash_srv_conf_t; @@ -49,6 +52,8 @@ static ngx_int_t ngx_http_upstream_get_h static ngx_int_t ngx_http_upstream_init_chash(ngx_conf_t *cf, ngx_http_upstream_srv_conf_t *us); +static ngx_int_t ngx_http_upstream_update_chash(ngx_pool_t *pool, + ngx_http_upstream_srv_conf_t *us); static int ngx_libc_cdecl ngx_http_upstream_chash_cmp_points(const void *one, const void *two); static ngx_uint_t ngx_http_upstream_find_chash_point( @@ -178,11 +183,18 @@ ngx_http_upstream_get_hash_peer(ngx_peer ngx_http_upstream_rr_peers_rlock(hp->rrp.peers); - if (hp->tries > 20 || hp->rrp.peers->single || hp->key.len == 0) { + if (hp->tries > 20 || hp->rrp.peers->number < 2 || hp->key.len == 0) { ngx_http_upstream_rr_peers_unlock(hp->rrp.peers); return hp->get_rr_peer(pc, &hp->rrp); } +#if (NGX_HTTP_UPSTREAM_ZONE) + if (hp->rrp.peers->config && hp->rrp.config != *hp->rrp.peers->config) { + ngx_http_upstream_rr_peers_unlock(hp->rrp.peers); + return hp->get_rr_peer(pc, &hp->rrp); + } +#endif + now = ngx_time(); pc->cached = 0; @@ -262,6 +274,7 @@ ngx_http_upstream_get_hash_peer(ngx_peer } hp->rrp.current = peer; + ngx_http_upstream_rr_peer_ref(hp->rrp.peers, peer); pc->sockaddr = peer->sockaddr; pc->socklen = peer->socklen; @@ -285,6 +298,26 @@ ngx_http_upstream_get_hash_peer(ngx_peer static ngx_int_t ngx_http_upstream_init_chash(ngx_conf_t *cf, ngx_http_upstream_srv_conf_t *us) { + if (ngx_http_upstream_init_round_robin(cf, us) != NGX_OK) { + return NGX_ERROR; + } + + us->peer.init = ngx_http_upstream_init_chash_peer; + +#if (NGX_HTTP_UPSTREAM_ZONE) + if (us->shm_zone) { + return NGX_OK; + } +#endif + + return ngx_http_upstream_update_chash(cf->pool, us); +} + + +static ngx_int_t +ngx_http_upstream_update_chash(ngx_pool_t *pool, + ngx_http_upstream_srv_conf_t *us) +{ u_char *host, *port, c; size_t host_len, port_len, size; uint32_t hash, base_hash; @@ -299,25 +332,32 @@ ngx_http_upstream_init_chash(ngx_conf_t u_char byte[4]; } prev_hash; - if (ngx_http_upstream_init_round_robin(cf, us) != NGX_OK) { - return NGX_ERROR; + hcf = ngx_http_conf_upstream_srv_conf(us, ngx_http_upstream_hash_module); + + if (hcf->points) { + ngx_free(hcf->points); + hcf->points = NULL; } - us->peer.init = ngx_http_upstream_init_chash_peer; - peers = us->peer.data; npoints = peers->total_weight * 160; size = sizeof(ngx_http_upstream_chash_points_t) - + sizeof(ngx_http_upstream_chash_point_t) * (npoints - 1); + - sizeof(ngx_http_upstream_chash_point_t) + + sizeof(ngx_http_upstream_chash_point_t) * npoints; - points = ngx_palloc(cf->pool, size); + points = pool ? ngx_palloc(pool, size) : ngx_alloc(size, ngx_cycle->log); if (points == NULL) { return NGX_ERROR; } points->number = 0; + if (npoints == 0) { + hcf->points = points; + return NGX_OK; + } + for (peer = peers->peer; peer; peer = peer->next) { server = &peer->server; @@ -401,7 +441,6 @@ ngx_http_upstream_init_chash(ngx_conf_t points->number = i + 1; - hcf = ngx_http_conf_upstream_srv_conf(us, ngx_http_upstream_hash_module); hcf->points = points; return NGX_OK; @@ -481,7 +520,22 @@ ngx_http_upstream_init_chash_peer(ngx_ht ngx_http_upstream_rr_peers_rlock(hp->rrp.peers); - hp->hash = ngx_http_upstream_find_chash_point(hcf->points, hash); +#if (NGX_HTTP_UPSTREAM_ZONE) + if (hp->rrp.peers->config + && (hcf->points == NULL || hcf->config != *hp->rrp.peers->config)) + { + if (ngx_http_upstream_update_chash(NULL, us) != NGX_OK) { + ngx_http_upstream_rr_peers_unlock(hp->rrp.peers); + return NGX_ERROR; + } + + hcf->config = *hp->rrp.peers->config; + } +#endif + + if (hcf->points->number) { + hp->hash = ngx_http_upstream_find_chash_point(hcf->points, hash); + } ngx_http_upstream_rr_peers_unlock(hp->rrp.peers); @@ -517,6 +571,20 @@ ngx_http_upstream_get_chash_peer(ngx_pee pc->cached = 0; pc->connection = NULL; + if (hp->rrp.peers->number == 0) { + pc->name = hp->rrp.peers->name; + ngx_http_upstream_rr_peers_unlock(hp->rrp.peers); + return NGX_BUSY; + } + +#if (NGX_HTTP_UPSTREAM_ZONE) + if (hp->rrp.peers->config && hp->rrp.config != *hp->rrp.peers->config) { + pc->name = hp->rrp.peers->name; + ngx_http_upstream_rr_peers_unlock(hp->rrp.peers); + return NGX_BUSY; + } +#endif + now = ngx_time(); hcf = hp->conf; @@ -597,6 +665,7 @@ ngx_http_upstream_get_chash_peer(ngx_pee found: hp->rrp.current = best; + ngx_http_upstream_rr_peer_ref(hp->rrp.peers, best); pc->sockaddr = best->sockaddr; pc->socklen = best->socklen; @@ -664,6 +733,7 @@ ngx_http_upstream_hash(ngx_conf_t *cf, n } uscf->flags = NGX_HTTP_UPSTREAM_CREATE + |NGX_HTTP_UPSTREAM_MODIFY |NGX_HTTP_UPSTREAM_WEIGHT |NGX_HTTP_UPSTREAM_MAX_CONNS |NGX_HTTP_UPSTREAM_MAX_FAILS diff --git a/src/http/modules/ngx_http_upstream_ip_hash_module.c b/src/http/modules/ngx_http_upstream_ip_hash_module.c --- a/src/http/modules/ngx_http_upstream_ip_hash_module.c +++ b/src/http/modules/ngx_http_upstream_ip_hash_module.c @@ -163,11 +163,19 @@ ngx_http_upstream_get_ip_hash_peer(ngx_p ngx_http_upstream_rr_peers_rlock(iphp->rrp.peers); - if (iphp->tries > 20 || iphp->rrp.peers->single) { + if (iphp->tries > 20 || iphp->rrp.peers->number < 2) { ngx_http_upstream_rr_peers_unlock(iphp->rrp.peers); return iphp->get_rr_peer(pc, &iphp->rrp); } +#if (NGX_HTTP_UPSTREAM_ZONE) + if (iphp->rrp.peers->config && iphp->rrp.config != *iphp->rrp.peers->config) + { + ngx_http_upstream_rr_peers_unlock(iphp->rrp.peers); + return iphp->get_rr_peer(pc, &iphp->rrp); + } +#endif + now = ngx_time(); pc->cached = 0; @@ -232,6 +240,7 @@ ngx_http_upstream_get_ip_hash_peer(ngx_p } iphp->rrp.current = peer; + ngx_http_upstream_rr_peer_ref(iphp->rrp.peers, peer); pc->sockaddr = peer->sockaddr; pc->socklen = peer->socklen; @@ -268,6 +277,7 @@ ngx_http_upstream_ip_hash(ngx_conf_t *cf uscf->peer.init_upstream = ngx_http_upstream_init_ip_hash; uscf->flags = NGX_HTTP_UPSTREAM_CREATE + |NGX_HTTP_UPSTREAM_MODIFY |NGX_HTTP_UPSTREAM_WEIGHT |NGX_HTTP_UPSTREAM_MAX_CONNS |NGX_HTTP_UPSTREAM_MAX_FAILS diff --git a/src/http/modules/ngx_http_upstream_least_conn_module.c b/src/http/modules/ngx_http_upstream_least_conn_module.c --- a/src/http/modules/ngx_http_upstream_least_conn_module.c +++ b/src/http/modules/ngx_http_upstream_least_conn_module.c @@ -124,6 +124,12 @@ ngx_http_upstream_get_least_conn_peer(ng ngx_http_upstream_rr_peers_wlock(peers); +#if (NGX_HTTP_UPSTREAM_ZONE) + if (peers->config && rrp->config != *peers->config) { + goto busy; + } +#endif + best = NULL; total = 0; @@ -244,6 +250,7 @@ ngx_http_upstream_get_least_conn_peer(ng best->conns++; rrp->current = best; + ngx_http_upstream_rr_peer_ref(peers, best); n = p / (8 * sizeof(uintptr_t)); m = (uintptr_t) 1 << p % (8 * sizeof(uintptr_t)); @@ -278,8 +285,18 @@ failed: } ngx_http_upstream_rr_peers_wlock(peers); + +#if (NGX_HTTP_UPSTREAM_ZONE) + if (peers->config && rrp->config != *peers->config) { + goto busy; + } +#endif } +#if (NGX_HTTP_UPSTREAM_ZONE) +busy: +#endif + ngx_http_upstream_rr_peers_unlock(peers); pc->name = peers->name; @@ -303,6 +320,7 @@ ngx_http_upstream_least_conn(ngx_conf_t uscf->peer.init_upstream = ngx_http_upstream_init_least_conn; uscf->flags = NGX_HTTP_UPSTREAM_CREATE + |NGX_HTTP_UPSTREAM_MODIFY |NGX_HTTP_UPSTREAM_WEIGHT |NGX_HTTP_UPSTREAM_MAX_CONNS |NGX_HTTP_UPSTREAM_MAX_FAILS diff --git a/src/http/modules/ngx_http_upstream_random_module.c b/src/http/modules/ngx_http_upstream_random_module.c --- a/src/http/modules/ngx_http_upstream_random_module.c +++ b/src/http/modules/ngx_http_upstream_random_module.c @@ -17,6 +17,9 @@ typedef struct { typedef struct { ngx_uint_t two; +#if (NGX_HTTP_UPSTREAM_ZONE) + ngx_uint_t config; +#endif ngx_http_upstream_random_range_t *ranges; } ngx_http_upstream_random_srv_conf_t; @@ -127,6 +130,11 @@ ngx_http_upstream_update_random(ngx_pool rcf = ngx_http_conf_upstream_srv_conf(us, ngx_http_upstream_random_module); + if (rcf->ranges) { + ngx_free(rcf->ranges); + rcf->ranges = NULL; + } + peers = us->peer.data; size = peers->number * sizeof(ngx_http_upstream_random_range_t); @@ -186,11 +194,15 @@ ngx_http_upstream_init_random_peer(ngx_h ngx_http_upstream_rr_peers_rlock(rp->rrp.peers); #if (NGX_HTTP_UPSTREAM_ZONE) - if (rp->rrp.peers->shpool && rcf->ranges == NULL) { + if (rp->rrp.peers->config + && (rcf->ranges == NULL || rcf->config != *rp->rrp.peers->config)) + { if (ngx_http_upstream_update_random(NULL, us) != NGX_OK) { ngx_http_upstream_rr_peers_unlock(rp->rrp.peers); return NGX_ERROR; } + + rcf->config = *rp->rrp.peers->config; } #endif @@ -220,11 +232,18 @@ ngx_http_upstream_get_random_peer(ngx_pe ngx_http_upstream_rr_peers_rlock(peers); - if (rp->tries > 20 || peers->single) { + if (rp->tries > 20 || peers->number < 2) { ngx_http_upstream_rr_peers_unlock(peers); return ngx_http_upstream_get_round_robin_peer(pc, rrp); } +#if (NGX_HTTP_UPSTREAM_ZONE) + if (peers->config && rrp->config != *peers->config) { + ngx_http_upstream_rr_peers_unlock(peers); + return ngx_http_upstream_get_round_robin_peer(pc, rrp); + } +#endif + pc->cached = 0; pc->connection = NULL; @@ -274,6 +293,7 @@ ngx_http_upstream_get_random_peer(ngx_pe } rrp->current = peer; + ngx_http_upstream_rr_peer_ref(peers, peer); if (now - peer->checked > peer->fail_timeout) { peer->checked = now; @@ -314,11 +334,18 @@ ngx_http_upstream_get_random2_peer(ngx_p ngx_http_upstream_rr_peers_wlock(peers); - if (rp->tries > 20 || peers->single) { + if (rp->tries > 20 || peers->number < 2) { ngx_http_upstream_rr_peers_unlock(peers); return ngx_http_upstream_get_round_robin_peer(pc, rrp); } +#if (NGX_HTTP_UPSTREAM_ZONE) + if (peers->config && rrp->config != *peers->config) { + ngx_http_upstream_rr_peers_unlock(peers); + return ngx_http_upstream_get_round_robin_peer(pc, rrp); + } +#endif + pc->cached = 0; pc->connection = NULL; @@ -384,6 +411,7 @@ ngx_http_upstream_get_random2_peer(ngx_p } rrp->current = peer; + ngx_http_upstream_rr_peer_ref(peers, peer); if (now - peer->checked > peer->fail_timeout) { peer->checked = now; @@ -467,6 +495,7 @@ ngx_http_upstream_random(ngx_conf_t *cf, uscf->peer.init_upstream = ngx_http_upstream_init_random; uscf->flags = NGX_HTTP_UPSTREAM_CREATE + |NGX_HTTP_UPSTREAM_MODIFY |NGX_HTTP_UPSTREAM_WEIGHT |NGX_HTTP_UPSTREAM_MAX_CONNS |NGX_HTTP_UPSTREAM_MAX_FAILS diff --git a/src/http/modules/ngx_http_upstream_zone_module.c b/src/http/modules/ngx_http_upstream_zone_module.c --- a/src/http/modules/ngx_http_upstream_zone_module.c +++ b/src/http/modules/ngx_http_upstream_zone_module.c @@ -18,6 +18,10 @@ static ngx_http_upstream_rr_peers_t *ngx ngx_slab_pool_t *shpool, ngx_http_upstream_srv_conf_t *uscf); static ngx_http_upstream_rr_peer_t *ngx_http_upstream_zone_copy_peer( ngx_http_upstream_rr_peers_t *peers, ngx_http_upstream_rr_peer_t *src); +static void ngx_http_upstream_zone_set_single( + ngx_http_upstream_srv_conf_t *uscf); +static void ngx_http_upstream_zone_remove_peer_locked( + ngx_http_upstream_rr_peers_t *peers, ngx_http_upstream_rr_peer_t *peer); static ngx_command_t ngx_http_upstream_zone_commands[] = { @@ -33,6 +37,11 @@ static ngx_command_t ngx_http_upstream_ }; +static ngx_int_t ngx_http_upstream_zone_init_worker(ngx_cycle_t *cycle); +static void ngx_http_upstream_zone_resolve_timer(ngx_event_t *event); +static void ngx_http_upstream_zone_resolve_handler(ngx_resolver_ctx_t *ctx); + + static ngx_http_module_t ngx_http_upstream_zone_module_ctx = { NULL, /* preconfiguration */ NULL, /* postconfiguration */ @@ -55,7 +64,7 @@ ngx_module_t ngx_http_upstream_zone_mod NGX_HTTP_MODULE, /* module type */ NULL, /* init master */ NULL, /* init module */ - NULL, /* init process */ + ngx_http_upstream_zone_init_worker, /* init process */ NULL, /* init thread */ NULL, /* exit thread */ NULL, /* exit process */ @@ -188,9 +197,15 @@ ngx_http_upstream_zone_copy_peers(ngx_sl ngx_http_upstream_srv_conf_t *uscf) { ngx_str_t *name; + ngx_uint_t *config; ngx_http_upstream_rr_peer_t *peer, **peerp; ngx_http_upstream_rr_peers_t *peers, *backup; + config = ngx_slab_calloc(shpool, sizeof(ngx_uint_t)); + if (config == NULL) { + return NULL; + } + peers = ngx_slab_alloc(shpool, sizeof(ngx_http_upstream_rr_peers_t)); if (peers == NULL) { return NULL; @@ -214,6 +229,7 @@ ngx_http_upstream_zone_copy_peers(ngx_sl peers->name = name; peers->shpool = shpool; + peers->config = config; for (peerp = &peers->peer; *peerp; peerp = &peer->next) { /* pool is unlocked */ @@ -223,6 +239,17 @@ ngx_http_upstream_zone_copy_peers(ngx_sl } *peerp = peer; + peer->id = (*peers->config)++; + } + + for (peerp = &peers->resolve; *peerp; peerp = &peer->next) { + peer = ngx_http_upstream_zone_copy_peer(peers, *peerp); + if (peer == NULL) { + return NULL; + } + + *peerp = peer; + peer->id = (*peers->config)++; } if (peers->next == NULL) { @@ -239,6 +266,7 @@ ngx_http_upstream_zone_copy_peers(ngx_sl backup->name = name; backup->shpool = shpool; + backup->config = config; for (peerp = &backup->peer; *peerp; peerp = &peer->next) { /* pool is unlocked */ @@ -248,6 +276,17 @@ ngx_http_upstream_zone_copy_peers(ngx_sl } *peerp = peer; + peer->id = (*backup->config)++; + } + + for (peerp = &backup->resolve; *peerp; peerp = &peer->next) { + peer = ngx_http_upstream_zone_copy_peer(backup, *peerp); + if (peer == NULL) { + return NULL; + } + + *peerp = peer; + peer->id = (*backup->config)++; } peers->next = backup; @@ -279,6 +318,7 @@ ngx_http_upstream_zone_copy_peer(ngx_htt dst->sockaddr = NULL; dst->name.data = NULL; dst->server.data = NULL; + dst->host = NULL; } dst->sockaddr = ngx_slab_calloc_locked(pool, sizeof(ngx_sockaddr_t)); @@ -301,12 +341,37 @@ ngx_http_upstream_zone_copy_peer(ngx_htt } ngx_memcpy(dst->server.data, src->server.data, src->server.len); + + if (src->host) { + dst->host = ngx_slab_calloc_locked(pool, + sizeof(ngx_http_upstream_host_t)); + if (dst->host == NULL) { + goto failed; + } + + dst->host->name.data = ngx_slab_alloc_locked(pool, + src->host->name.len); + if (dst->host->name.data == NULL) { + goto failed; + } + + dst->host->peers = peers; + dst->host->peer = dst; + + dst->host->name.len = src->host->name.len; + ngx_memcpy(dst->host->name.data, src->host->name.data, + src->host->name.len); + } } return dst; failed: + if (dst->host) { + ngx_slab_free_locked(pool, dst->host); + } + if (dst->server.data) { ngx_slab_free_locked(pool, dst->server.data); } @@ -323,3 +388,338 @@ failed: return NULL; } + + +static void +ngx_http_upstream_zone_set_single(ngx_http_upstream_srv_conf_t *uscf) +{ + ngx_http_upstream_rr_peers_t *peers; + + peers = uscf->peer.data; + + if (peers->number == 1 + && (peers->next == NULL || peers->next->number == 0)) + { + peers->single = 1; + + } else { + peers->single = 0; + } +} + + +static void +ngx_http_upstream_zone_remove_peer_locked(ngx_http_upstream_rr_peers_t *peers, + ngx_http_upstream_rr_peer_t *peer) +{ + peers->total_weight -= peer->weight; + peers->number--; + peers->tries -= (peer->down == 0); + (*peers->config)++; + peers->weighted = (peers->total_weight != peers->number); + + ngx_http_upstream_rr_peer_free(peers, peer); +} + + +static ngx_int_t +ngx_http_upstream_zone_init_worker(ngx_cycle_t *cycle) +{ + ngx_uint_t i; + ngx_event_t *event; + ngx_http_upstream_rr_peer_t *peer; + ngx_http_upstream_rr_peers_t *peers; + ngx_http_upstream_srv_conf_t *uscf, **uscfp; + ngx_http_upstream_main_conf_t *umcf; + + if (ngx_process != NGX_PROCESS_WORKER + && ngx_process != NGX_PROCESS_SINGLE) + { + return NGX_OK; + } + + umcf = ngx_http_cycle_get_module_main_conf(cycle, ngx_http_upstream_module); + + if (umcf == NULL) { + return NGX_OK; + } + + uscfp = umcf->upstreams.elts; + + for (i = 0; i < umcf->upstreams.nelts; i++) { + + uscf = uscfp[i]; + + if (uscf->shm_zone == NULL) { + continue; + } + + peers = uscf->peer.data; + + do { + ngx_http_upstream_rr_peers_wlock(peers); + + for (peer = peers->resolve; peer; peer = peer->next) { + + if (peer->host->worker != ngx_worker) { + continue; + } + + event = &peer->host->event; + ngx_memzero(event, sizeof(ngx_event_t)); + + event->data = uscf; + event->handler = ngx_http_upstream_zone_resolve_timer; + event->log = cycle->log; + event->cancelable = 1; + + ngx_http_upstream_rr_peer_ref(peers, peer); + ngx_add_timer(event, 1); + } + + ngx_http_upstream_rr_peers_unlock(peers); + + peers = peers->next; + + } while (peers); + } + + return NGX_OK; +} + + +static void +ngx_http_upstream_zone_resolve_timer(ngx_event_t *event) +{ + ngx_resolver_ctx_t *ctx; + ngx_http_upstream_host_t *host; + ngx_http_upstream_rr_peer_t *template; + ngx_http_upstream_rr_peers_t *peers; + ngx_http_upstream_srv_conf_t *uscf; + + host = (ngx_http_upstream_host_t *) event; + uscf = event->data; + peers = host->peers; + template = host->peer; + + if (template->zombie) { + (void) ngx_http_upstream_rr_peer_unref(peers, template); + + ngx_shmtx_lock(&peers->shpool->mutex); + + if (host->service.len) { + ngx_slab_free_locked(peers->shpool, host->service.data); + } + + ngx_slab_free_locked(peers->shpool, host->name.data); + ngx_slab_free_locked(peers->shpool, host); + ngx_shmtx_unlock(&peers->shpool->mutex); + + return; + } + + ctx = ngx_resolve_start(uscf->resolver, NULL); + if (ctx == NULL) { + goto retry; + } + + if (ctx == NGX_NO_RESOLVER) { + ngx_log_error(NGX_LOG_ERR, event->log, 0, + "no resolver defined to resolve %V", &host->name); + return; + } + + ctx->name = host->name; + ctx->handler = ngx_http_upstream_zone_resolve_handler; + ctx->data = host; + ctx->timeout = uscf->resolver_timeout; + ctx->cancelable = 1; + + if (ngx_resolve_name(ctx) == NGX_OK) { + return; + } + +retry: + + ngx_add_timer(event, ngx_max(uscf->resolver_timeout, 1000)); +} + + +static void +ngx_http_upstream_zone_resolve_handler(ngx_resolver_ctx_t *ctx) +{ + time_t now; + in_port_t port; + ngx_msec_t timer; + ngx_uint_t i, j; + ngx_event_t *event; + ngx_resolver_addr_t *addr; + ngx_http_upstream_host_t *host; + ngx_http_upstream_rr_peer_t *peer, *template, **peerp; + ngx_http_upstream_rr_peers_t *peers; + ngx_http_upstream_srv_conf_t *uscf; + + host = ctx->data; + event = &host->event; + uscf = event->data; + peers = host->peers; + template = host->peer; + + ngx_http_upstream_rr_peers_wlock(peers); + + if (template->zombie) { + (void) ngx_http_upstream_rr_peer_unref(peers, template); + + ngx_http_upstream_rr_peers_unlock(peers); + + ngx_shmtx_lock(&peers->shpool->mutex); + ngx_slab_free_locked(peers->shpool, host->name.data); + ngx_slab_free_locked(peers->shpool, host); + ngx_shmtx_unlock(&peers->shpool->mutex); + + ngx_resolve_name_done(ctx); + + return; + } + + now = ngx_time(); + + if (ctx->state) { + ngx_log_error(NGX_LOG_ERR, event->log, 0, + "%V could not be resolved (%i: %s)", + &ctx->name, ctx->state, + ngx_resolver_strerror(ctx->state)); + + if (ctx->state != NGX_RESOLVE_NXDOMAIN) { + ngx_http_upstream_rr_peers_unlock(peers); + + ngx_resolve_name_done(ctx); + + ngx_add_timer(event, ngx_max(uscf->resolver_timeout, 1000)); + return; + } + + /* NGX_RESOLVE_NXDOMAIN */ + + ctx->naddrs = 0; + } + +#if (NGX_DEBUG) + { + u_char text[NGX_SOCKADDR_STRLEN]; + size_t len; + + for (i = 0; i < ctx->naddrs; i++) { + len = ngx_sock_ntop(ctx->addrs[i].sockaddr, ctx->addrs[i].socklen, + text, NGX_SOCKADDR_STRLEN, 0); + + ngx_log_debug3(NGX_LOG_DEBUG_HTTP, event->log, 0, + "name %V was resolved to %*s", &host->name, len, text); + } + } +#endif + + for (peerp = &peers->peer; *peerp; /* void */ ) { + peer = *peerp; + + if (peer->host != host) { + goto next; + } + + for (j = 0; j < ctx->naddrs; j++) { + + addr = &ctx->addrs[j]; + + if (addr->name.len == 0 + && ngx_cmp_sockaddr(peer->sockaddr, peer->socklen, + addr->sockaddr, addr->socklen, 0) + == NGX_OK) + { + addr->name.len = 1; + goto next; + } + } + + *peerp = peer->next; + ngx_http_upstream_zone_remove_peer_locked(peers, peer); + + ngx_http_upstream_zone_set_single(uscf); + + continue; + + next: + + peerp = &peer->next; + } + + for (i = 0; i < ctx->naddrs; i++) { + + addr = &ctx->addrs[i]; + + if (addr->name.len == 1) { + addr->name.len = 0; + continue; + } + + ngx_shmtx_lock(&peers->shpool->mutex); + peer = ngx_http_upstream_zone_copy_peer(peers, NULL); + ngx_shmtx_unlock(&peers->shpool->mutex); + + if (peer == NULL) { + ngx_log_error(NGX_LOG_ERR, event->log, 0, + "cannot add new server to upstream \"%V\", " + "memory exhausted", peers->name); + break; + } + + ngx_memcpy(peer->sockaddr, addr->sockaddr, addr->socklen); + + port = ((struct sockaddr_in *) template->sockaddr)->sin_port; + + switch (peer->sockaddr->sa_family) { +#if (NGX_HAVE_INET6) + case AF_INET6: + ((struct sockaddr_in6 *) peer->sockaddr)->sin6_port = port; + break; +#endif + default: /* AF_INET */ + ((struct sockaddr_in *) peer->sockaddr)->sin_port = port; + } + + peer->socklen = addr->socklen; + + peer->name.len = ngx_sock_ntop(peer->sockaddr, peer->socklen, + peer->name.data, NGX_SOCKADDR_STRLEN, 1); + + peer->host = template->host; + peer->server = template->server; + + peer->weight = template->weight; + peer->effective_weight = peer->weight; + peer->max_conns = template->max_conns; + peer->max_fails = template->max_fails; + peer->fail_timeout = template->fail_timeout; + peer->slow_start = template->slow_start; + peer->down = template->down; + + *peerp = peer; + peerp = &peer->next; + + peers->number++; + peers->tries += (peer->down == 0); + peers->total_weight += peer->weight; + peers->weighted = (peers->total_weight != peers->number); + peer->id = (*peers->config)++; + + ngx_http_upstream_zone_set_single(uscf); + } + + ngx_http_upstream_rr_peers_unlock(peers); + + timer = (ngx_msec_t) 1000 * (ctx->valid > now ? ctx->valid - now + 1 : 1); + timer = ngx_min(timer, uscf->resolver_timeout); + + ngx_resolve_name_done(ctx); + + ngx_add_timer(event, timer); +} diff --git a/src/http/ngx_http_upstream.c b/src/http/ngx_http_upstream.c --- a/src/http/ngx_http_upstream.c +++ b/src/http/ngx_http_upstream.c @@ -1545,6 +1545,26 @@ ngx_http_upstream_connect(ngx_http_reque u->state->peer = u->peer.name; +#if (NGX_HTTP_UPSTREAM_ZONE) + if (u->upstream && u->upstream->shm_zone + && (u->upstream->flags & NGX_HTTP_UPSTREAM_MODIFY) + ) { + u->state->peer = ngx_palloc(r->pool, + sizeof(ngx_str_t) + u->peer.name->len); + if (u->state->peer == NULL) { + ngx_http_upstream_finalize_request(r, u, + NGX_HTTP_INTERNAL_SERVER_ERROR); + return; + } + + u->state->peer->len = u->peer.name->len; + u->state->peer->data = (u_char *) (u->state->peer + 1); + ngx_memcpy(u->state->peer->data, u->peer.name->data, u->peer.name->len); + + u->peer.name = u->state->peer; + } +#endif + if (rc == NGX_BUSY) { ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, "no live upstreams"); ngx_http_upstream_next(r, u, NGX_HTTP_UPSTREAM_FT_NOLIVE); @@ -6029,6 +6049,7 @@ ngx_http_upstream(ngx_conf_t *cf, ngx_co u.no_port = 1; uscf = ngx_http_upstream_add(cf, &u, NGX_HTTP_UPSTREAM_CREATE + |NGX_HTTP_UPSTREAM_MODIFY |NGX_HTTP_UPSTREAM_WEIGHT |NGX_HTTP_UPSTREAM_MAX_CONNS |NGX_HTTP_UPSTREAM_MAX_FAILS @@ -6114,7 +6135,11 @@ ngx_http_upstream(ngx_conf_t *cf, ngx_co return rv; } - if (uscf->servers->nelts == 0) { + if (uscf->servers->nelts == 0 +#if (NGX_HTTP_UPSTREAM_ZONE) + && uscf->shm_zone == NULL +#endif + ) { ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, "no servers are inside upstream"); return NGX_CONF_ERROR; @@ -6134,6 +6159,9 @@ ngx_http_upstream_server(ngx_conf_t *cf, ngx_url_t u; ngx_int_t weight, max_conns, max_fails; ngx_uint_t i; +#if (NGX_HTTP_UPSTREAM_ZONE) + ngx_uint_t resolve; +#endif ngx_http_upstream_server_t *us; us = ngx_array_push(uscf->servers); @@ -6149,6 +6177,9 @@ ngx_http_upstream_server(ngx_conf_t *cf, max_conns = 0; max_fails = 1; fail_timeout = 10; +#if (NGX_HTTP_UPSTREAM_ZONE) + resolve = 0; +#endif for (i = 2; i < cf->args->nelts; i++) { @@ -6237,6 +6268,13 @@ ngx_http_upstream_server(ngx_conf_t *cf, continue; } +#if (NGX_HTTP_UPSTREAM_ZONE) + if (ngx_strcmp(value[i].data, "resolve") == 0) { + resolve = 1; + continue; + } +#endif + goto invalid; } @@ -6245,6 +6283,13 @@ ngx_http_upstream_server(ngx_conf_t *cf, u.url = value[1]; u.default_port = 80; +#if (NGX_HTTP_UPSTREAM_ZONE) + if (resolve) { + /* resolve at run time */ + u.no_resolve = 1; + } +#endif + if (ngx_parse_url(cf->pool, &u) != NGX_OK) { if (u.err) { ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, @@ -6255,8 +6300,45 @@ ngx_http_upstream_server(ngx_conf_t *cf, } us->name = u.url; + +#if (NGX_HTTP_UPSTREAM_ZONE) + + if (resolve && u.naddrs == 0) { + ngx_addr_t *addr; + + /* save port */ + + addr = ngx_pcalloc(cf->pool, sizeof(ngx_addr_t)); + if (addr == NULL) { + return NGX_CONF_ERROR; + } + + addr->sockaddr = ngx_palloc(cf->pool, u.socklen); + if (addr->sockaddr == NULL) { + return NGX_CONF_ERROR; + } + + ngx_memcpy(addr->sockaddr, &u.sockaddr, u.socklen); + + addr->socklen = u.socklen; + + us->addrs = addr; + us->naddrs = 1; + + us->host = u.host; + + } else { + us->addrs = u.addrs; + us->naddrs = u.naddrs; + } + +#else + us->addrs = u.addrs; us->naddrs = u.naddrs; + +#endif + us->weight = weight; us->max_conns = max_conns; us->max_fails = max_fails; diff --git a/src/http/ngx_http_upstream.h b/src/http/ngx_http_upstream.h --- a/src/http/ngx_http_upstream.h +++ b/src/http/ngx_http_upstream.h @@ -104,7 +104,11 @@ typedef struct { unsigned backup:1; - NGX_COMPAT_BEGIN(6) +#if (NGX_HTTP_UPSTREAM_ZONE) + ngx_str_t host; +#endif + + NGX_COMPAT_BEGIN(4) NGX_COMPAT_END } ngx_http_upstream_server_t; @@ -115,6 +119,7 @@ typedef struct { #define NGX_HTTP_UPSTREAM_FAIL_TIMEOUT 0x0008 #define NGX_HTTP_UPSTREAM_DOWN 0x0010 #define NGX_HTTP_UPSTREAM_BACKUP 0x0020 +#define NGX_HTTP_UPSTREAM_MODIFY 0x0040 #define NGX_HTTP_UPSTREAM_MAX_CONNS 0x0100 @@ -133,6 +138,8 @@ struct ngx_http_upstream_srv_conf_s { #if (NGX_HTTP_UPSTREAM_ZONE) ngx_shm_zone_t *shm_zone; + ngx_resolver_t *resolver; + ngx_msec_t resolver_timeout; #endif }; diff --git a/src/http/ngx_http_upstream_round_robin.c b/src/http/ngx_http_upstream_round_robin.c --- a/src/http/ngx_http_upstream_round_robin.c +++ b/src/http/ngx_http_upstream_round_robin.c @@ -32,10 +32,15 @@ ngx_http_upstream_init_round_robin(ngx_c ngx_http_upstream_srv_conf_t *us) { ngx_url_t u; - ngx_uint_t i, j, n, w, t; + ngx_uint_t i, j, n, r, w, t; ngx_http_upstream_server_t *server; ngx_http_upstream_rr_peer_t *peer, **peerp; ngx_http_upstream_rr_peers_t *peers, *backup; +#if (NGX_HTTP_UPSTREAM_ZONE) + ngx_uint_t resolve; + ngx_http_core_loc_conf_t *clcf; + ngx_http_upstream_rr_peer_t **rpeerp; +#endif us->peer.init = ngx_http_upstream_init_round_robin_peer; @@ -43,23 +48,99 @@ ngx_http_upstream_init_round_robin(ngx_c server = us->servers->elts; n = 0; + r = 0; w = 0; t = 0; +#if (NGX_HTTP_UPSTREAM_ZONE) + resolve = 0; +#endif + for (i = 0; i < us->servers->nelts; i++) { + +#if (NGX_HTTP_UPSTREAM_ZONE) + if (server[i].host.len) { + resolve = 1; + } +#endif + if (server[i].backup) { continue; } +#if (NGX_HTTP_UPSTREAM_ZONE) + if (server[i].host.len) { + r++; + + } else { + n += server[i].naddrs; + w += server[i].naddrs * server[i].weight; + + if (!server[i].down) { + t += server[i].naddrs; + } + } +#else n += server[i].naddrs; w += server[i].naddrs * server[i].weight; if (!server[i].down) { t += server[i].naddrs; } +#endif } - if (n == 0) { +#if (NGX_HTTP_UPSTREAM_ZONE) + if (us->shm_zone) { + + if (resolve && !(us->flags & NGX_HTTP_UPSTREAM_MODIFY)) { + ngx_log_error(NGX_LOG_EMERG, cf->log, 0, + "load balancing method does not support" + " resolving names at run time in" + " upstream \"%V\" in %s:%ui", + &us->host, us->file_name, us->line); + return NGX_ERROR; + } + + clcf = ngx_http_conf_get_module_loc_conf(cf, ngx_http_core_module); + + us->resolver = clcf->resolver; + us->resolver_timeout = clcf->resolver_timeout; + + /* + * Without "resolver_timeout" in http{}, the value is unset. + * Even if we set it in ngx_http_core_merge_loc_conf(), it's + * still dependent on the module order and unreliable. + */ + ngx_conf_init_msec_value(us->resolver_timeout, 30000); + + if (resolve + && (us->resolver == NULL + || us->resolver->connections.nelts == 0)) + { + ngx_log_error(NGX_LOG_EMERG, cf->log, 0, + "no resolver defined to resolve names" + " at run time in upstream \"%V\" in %s:%ui", + &us->host, us->file_name, us->line); + return NGX_ERROR; + } + + } else if (resolve) { + + ngx_log_error(NGX_LOG_EMERG, cf->log, 0, + "resolving names at run time requires" + " upstream \"%V\" in %s:%ui" + " to be in shared memory", + &us->host, us->file_name, us->line); + return NGX_ERROR; + } +#endif + + if (n == 0 +#if (NGX_HTTP_UPSTREAM_ZONE) + && us->shm_zone == NULL +#endif + ) { ngx_log_error(NGX_LOG_EMERG, cf->log, 0, "no servers in upstream \"%V\" in %s:%ui", &us->host, us->file_name, us->line); @@ -71,7 +152,8 @@ ngx_http_upstream_init_round_robin(ngx_c return NGX_ERROR; } - peer = ngx_pcalloc(cf->pool, sizeof(ngx_http_upstream_rr_peer_t) * n); + peer = ngx_pcalloc(cf->pool, sizeof(ngx_http_upstream_rr_peer_t) + * (n + r)); if (peer == NULL) { return NGX_ERROR; } @@ -86,11 +168,47 @@ ngx_http_upstream_init_round_robin(ngx_c n = 0; peerp = &peers->peer; +#if (NGX_HTTP_UPSTREAM_ZONE) + rpeerp = &peers->resolve; +#endif + for (i = 0; i < us->servers->nelts; i++) { if (server[i].backup) { continue; } +#if (NGX_HTTP_UPSTREAM_ZONE) + if (server[i].host.len) { + + peer[n].host = ngx_pcalloc(cf->pool, + sizeof(ngx_http_upstream_host_t)); + if (peer[n].host == NULL) { + return NGX_ERROR; + } + + peer[n].host->name = server[i].host; + + peer[n].sockaddr = server[i].addrs[0].sockaddr; + peer[n].socklen = server[i].addrs[0].socklen; + peer[n].name = server[i].addrs[0].name; + peer[n].weight = server[i].weight; + peer[n].effective_weight = server[i].weight; + peer[n].current_weight = 0; + peer[n].max_conns = server[i].max_conns; + peer[n].max_fails = server[i].max_fails; + peer[n].fail_timeout = server[i].fail_timeout; + peer[n].slow_start = server[i].slow_start; + peer[n].down = server[i].down; + peer[n].server = server[i].name; + + *rpeerp = &peer[n]; + rpeerp = &peer[n].next; + n++; + + continue; + } +#endif + for (j = 0; j < server[i].naddrs; j++) { peer[n].sockaddr = server[i].addrs[j].sockaddr; peer[n].socklen = server[i].addrs[j].socklen; @@ -115,6 +233,7 @@ ngx_http_upstream_init_round_robin(ngx_c /* backup servers */ n = 0; + r = 0; w = 0; t = 0; @@ -123,15 +242,37 @@ ngx_http_upstream_init_round_robin(ngx_c continue; } +#if (NGX_HTTP_UPSTREAM_ZONE) + if (server[i].host.len) { + r++; + + } else { + n += server[i].naddrs; + w += server[i].naddrs * server[i].weight; + + if (!server[i].down) { + t += server[i].naddrs; + } + } +#else n += server[i].naddrs; w += server[i].naddrs * server[i].weight; if (!server[i].down) { t += server[i].naddrs; } +#endif } - if (n == 0) { + if (n == 0 +#if (NGX_HTTP_UPSTREAM_ZONE) + && us->shm_zone == NULL +#endif + ) { + return NGX_OK; + } + + if (n + r == 0 && !(us->flags & NGX_HTTP_UPSTREAM_BACKUP)) { return NGX_OK; } @@ -140,12 +281,16 @@ ngx_http_upstream_init_round_robin(ngx_c return NGX_ERROR; } - peer = ngx_pcalloc(cf->pool, sizeof(ngx_http_upstream_rr_peer_t) * n); + peer = ngx_pcalloc(cf->pool, sizeof(ngx_http_upstream_rr_peer_t) + * (n + r)); if (peer == NULL) { return NGX_ERROR; } - peers->single = 0; + if (n > 0) { + peers->single = 0; + } + backup->single = 0; backup->number = n; backup->weighted = (w != n); @@ -156,11 +301,47 @@ ngx_http_upstream_init_round_robin(ngx_c n = 0; peerp = &backup->peer; +#if (NGX_HTTP_UPSTREAM_ZONE) + rpeerp = &backup->resolve; +#endif + for (i = 0; i < us->servers->nelts; i++) { if (!server[i].backup) { continue; } +#if (NGX_HTTP_UPSTREAM_ZONE) + if (server[i].host.len) { + + peer[n].host = ngx_pcalloc(cf->pool, + sizeof(ngx_http_upstream_host_t)); + if (peer[n].host == NULL) { + return NGX_ERROR; + } + + peer[n].host->name = server[i].host; + + peer[n].sockaddr = server[i].addrs[0].sockaddr; + peer[n].socklen = server[i].addrs[0].socklen; + peer[n].name = server[i].addrs[0].name; + peer[n].weight = server[i].weight; + peer[n].effective_weight = server[i].weight; + peer[n].current_weight = 0; + peer[n].max_conns = server[i].max_conns; + peer[n].max_fails = server[i].max_fails; + peer[n].fail_timeout = server[i].fail_timeout; + peer[n].slow_start = server[i].slow_start; + peer[n].down = server[i].down; + peer[n].server = server[i].name; + + *rpeerp = &peer[n]; + rpeerp = &peer[n].next; + n++; + + continue; + } +#endif + for (j = 0; j < server[i].naddrs; j++) { peer[n].sockaddr = server[i].addrs[j].sockaddr; peer[n].socklen = server[i].addrs[j].socklen; @@ -273,7 +454,12 @@ ngx_http_upstream_init_round_robin_peer( rrp->peers = us->peer.data; rrp->current = NULL; - rrp->config = 0; + + ngx_http_upstream_rr_peers_rlock(rrp->peers); + +#if (NGX_HTTP_UPSTREAM_ZONE) + rrp->config = rrp->peers->config ? *rrp->peers->config : 0; +#endif n = rrp->peers->number; @@ -281,6 +467,10 @@ ngx_http_upstream_init_round_robin_peer( n = rrp->peers->next->number; } + r->upstream->peer.tries = ngx_http_upstream_tries(rrp->peers); + + ngx_http_upstream_rr_peers_unlock(rrp->peers); + if (n <= 8 * sizeof(uintptr_t)) { rrp->tried = &rrp->data; rrp->data = 0; @@ -296,7 +486,6 @@ ngx_http_upstream_init_round_robin_peer( r->upstream->peer.get = ngx_http_upstream_get_round_robin_peer; r->upstream->peer.free = ngx_http_upstream_free_round_robin_peer; - r->upstream->peer.tries = ngx_http_upstream_tries(rrp->peers); #if (NGX_HTTP_SSL) r->upstream->peer.set_session = ngx_http_upstream_set_round_robin_peer_session; @@ -446,6 +635,12 @@ ngx_http_upstream_get_round_robin_peer(n peers = rrp->peers; ngx_http_upstream_rr_peers_wlock(peers); +#if (NGX_HTTP_UPSTREAM_ZONE) + if (peers->config && rrp->config != *peers->config) { + goto busy; + } +#endif + if (peers->single) { peer = peers->peer; @@ -458,6 +653,7 @@ ngx_http_upstream_get_round_robin_peer(n } rrp->current = peer; + ngx_http_upstream_rr_peer_ref(peers, peer); } else { @@ -508,8 +704,18 @@ failed: } ngx_http_upstream_rr_peers_wlock(peers); + +#if (NGX_HTTP_UPSTREAM_ZONE) + if (peers->config && rrp->config != *peers->config) { + goto busy; + } +#endif } +#if (NGX_HTTP_UPSTREAM_ZONE) +busy: +#endif + ngx_http_upstream_rr_peers_unlock(peers); pc->name = peers->name; @@ -580,6 +786,7 @@ ngx_http_upstream_get_peer(ngx_http_upst } rrp->current = best; + ngx_http_upstream_rr_peer_ref(rrp->peers, best); n = p / (8 * sizeof(uintptr_t)); m = (uintptr_t) 1 << p % (8 * sizeof(uintptr_t)); @@ -617,9 +824,16 @@ ngx_http_upstream_free_round_robin_peer( if (rrp->peers->single) { + if (peer->fails) { + peer->fails = 0; + } + peer->conns--; - ngx_http_upstream_rr_peer_unlock(rrp->peers, peer); + if (ngx_http_upstream_rr_peer_unref(rrp->peers, peer) == NGX_OK) { + ngx_http_upstream_rr_peer_unlock(rrp->peers, peer); + } + ngx_http_upstream_rr_peers_unlock(rrp->peers); pc->tries = 0; @@ -661,7 +875,10 @@ ngx_http_upstream_free_round_robin_peer( peer->conns--; - ngx_http_upstream_rr_peer_unlock(rrp->peers, peer); + if (ngx_http_upstream_rr_peer_unref(rrp->peers, peer) == NGX_OK) { + ngx_http_upstream_rr_peer_unlock(rrp->peers, peer); + } + ngx_http_upstream_rr_peers_unlock(rrp->peers); if (pc->tries) { diff --git a/src/http/ngx_http_upstream_round_robin.h b/src/http/ngx_http_upstream_round_robin.h --- a/src/http/ngx_http_upstream_round_robin.h +++ b/src/http/ngx_http_upstream_round_robin.h @@ -14,8 +14,23 @@ #include +typedef struct ngx_http_upstream_rr_peers_s ngx_http_upstream_rr_peers_t; typedef struct ngx_http_upstream_rr_peer_s ngx_http_upstream_rr_peer_t; + +#if (NGX_HTTP_UPSTREAM_ZONE) + +typedef struct { + ngx_event_t event; /* must be first */ + ngx_uint_t worker; + ngx_str_t name; + ngx_http_upstream_rr_peers_t *peers; + ngx_http_upstream_rr_peer_t *peer; +} ngx_http_upstream_host_t; + +#endif + + struct ngx_http_upstream_rr_peer_s { struct sockaddr *sockaddr; socklen_t socklen; @@ -46,7 +61,12 @@ struct ngx_http_upstream_rr_peer_s { #endif #if (NGX_HTTP_UPSTREAM_ZONE) + unsigned zombie:1; + ngx_atomic_t lock; + ngx_uint_t id; + ngx_uint_t refs; + ngx_http_upstream_host_t *host; #endif ngx_http_upstream_rr_peer_t *next; @@ -56,8 +76,6 @@ struct ngx_http_upstream_rr_peer_s { }; -typedef struct ngx_http_upstream_rr_peers_s ngx_http_upstream_rr_peers_t; - struct ngx_http_upstream_rr_peers_s { ngx_uint_t number; @@ -78,6 +96,12 @@ struct ngx_http_upstream_rr_peers_s { ngx_http_upstream_rr_peers_t *next; ngx_http_upstream_rr_peer_t *peer; + +#if (NGX_HTTP_UPSTREAM_ZONE) + ngx_uint_t *config; + ngx_http_upstream_rr_peer_t *resolve; + ngx_uint_t zombies; +#endif }; @@ -114,6 +138,67 @@ struct ngx_http_upstream_rr_peers_s { ngx_rwlock_unlock(&peer->lock); \ } + +#define ngx_http_upstream_rr_peer_ref(peers, peer) \ + (peer)->refs++; + + +static ngx_inline void +ngx_http_upstream_rr_peer_free_locked(ngx_http_upstream_rr_peers_t *peers, + ngx_http_upstream_rr_peer_t *peer) +{ + if (peer->refs) { + peer->zombie = 1; + peers->zombies++; + return; + } + + ngx_slab_free_locked(peers->shpool, peer->sockaddr); + ngx_slab_free_locked(peers->shpool, peer->name.data); + + if (peer->server.data && (peer->host == NULL || peer->host->peer == peer)) { + ngx_slab_free_locked(peers->shpool, peer->server.data); + } + +#if (NGX_HTTP_SSL) + if (peer->ssl_session) { + ngx_slab_free_locked(peers->shpool, peer->ssl_session); + } +#endif + + ngx_slab_free_locked(peers->shpool, peer); +} + + +static ngx_inline void +ngx_http_upstream_rr_peer_free(ngx_http_upstream_rr_peers_t *peers, + ngx_http_upstream_rr_peer_t *peer) +{ + ngx_shmtx_lock(&peers->shpool->mutex); + ngx_http_upstream_rr_peer_free_locked(peers, peer); + ngx_shmtx_unlock(&peers->shpool->mutex); +} + + +static ngx_inline ngx_int_t +ngx_http_upstream_rr_peer_unref(ngx_http_upstream_rr_peers_t *peers, + ngx_http_upstream_rr_peer_t *peer) +{ + peer->refs--; + + if (peers->shpool == NULL) { + return NGX_OK; + } + + if (peer->refs == 0 && peer->zombie) { + ngx_http_upstream_rr_peer_free(peers, peer); + peers->zombies--; + return NGX_DONE; + } + + return NGX_OK; +} + #else #define ngx_http_upstream_rr_peers_rlock(peers) @@ -121,6 +206,8 @@ struct ngx_http_upstream_rr_peers_s { #define ngx_http_upstream_rr_peers_unlock(peers) #define ngx_http_upstream_rr_peer_lock(peers, peer) #define ngx_http_upstream_rr_peer_unlock(peers, peer) +#define ngx_http_upstream_rr_peer_ref(peers, peer) +#define ngx_http_upstream_rr_peer_unref(peers, peer) NGX_OK #endif From a.bavshin at nginx.com Wed Feb 1 01:36:57 2023 From: a.bavshin at nginx.com (=?iso-8859-1?q?Aleksei_Bavshin?=) Date: Tue, 31 Jan 2023 17:36:57 -0800 Subject: [PATCH 2 of 6] Stream: re-resolvable servers In-Reply-To: References: Message-ID: <09954272153fb0ad62a8.1675215417@fedora-wsl.> # HG changeset patch # User Roman Arutyunyan # Date 1423128416 -10800 # Thu Feb 05 12:26:56 2015 +0300 # Node ID 09954272153fb0ad62a80799a854a22fd9ebcdc9 # Parent b6e13aa0330c8434add55c88598480016b80fc0e Stream: re-resolvable servers. Co-authored-by: Ruslan Ermilov Co-authored-by: Sergey Kandaurov Co-authored-by: Vladimir Homutov diff --git a/src/stream/ngx_stream_proxy_module.c b/src/stream/ngx_stream_proxy_module.c --- a/src/stream/ngx_stream_proxy_module.c +++ b/src/stream/ngx_stream_proxy_module.c @@ -742,6 +742,25 @@ ngx_stream_proxy_connect(ngx_stream_sess u->state->peer = u->peer.name; +#if (NGX_STREAM_UPSTREAM_ZONE) + if (u->upstream && u->upstream->shm_zone + && (u->upstream->flags & NGX_STREAM_UPSTREAM_MODIFY) + ) { + u->state->peer = ngx_palloc(s->connection->pool, + sizeof(ngx_str_t) + u->peer.name->len); + if (u->state->peer == NULL) { + ngx_stream_proxy_finalize(s, NGX_STREAM_INTERNAL_SERVER_ERROR); + return; + } + + u->state->peer->len = u->peer.name->len; + u->state->peer->data = (u_char *) (u->state->peer + 1); + ngx_memcpy(u->state->peer->data, u->peer.name->data, u->peer.name->len); + + u->peer.name = u->state->peer; + } +#endif + if (rc == NGX_BUSY) { ngx_log_error(NGX_LOG_ERR, c->log, 0, "no live upstreams"); ngx_stream_proxy_finalize(s, NGX_STREAM_BAD_GATEWAY); diff --git a/src/stream/ngx_stream_upstream.c b/src/stream/ngx_stream_upstream.c --- a/src/stream/ngx_stream_upstream.c +++ b/src/stream/ngx_stream_upstream.c @@ -319,6 +319,7 @@ ngx_stream_upstream(ngx_conf_t *cf, ngx_ u.no_port = 1; uscf = ngx_stream_upstream_add(cf, &u, NGX_STREAM_UPSTREAM_CREATE + |NGX_STREAM_UPSTREAM_MODIFY |NGX_STREAM_UPSTREAM_WEIGHT |NGX_STREAM_UPSTREAM_MAX_CONNS |NGX_STREAM_UPSTREAM_MAX_FAILS @@ -388,7 +389,11 @@ ngx_stream_upstream(ngx_conf_t *cf, ngx_ return rv; } - if (uscf->servers->nelts == 0) { + if (uscf->servers->nelts == 0 +#if (NGX_STREAM_UPSTREAM_ZONE) + && uscf->shm_zone == NULL +#endif + ) { ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, "no servers are inside upstream"); return NGX_CONF_ERROR; @@ -408,6 +413,9 @@ ngx_stream_upstream_server(ngx_conf_t *c ngx_url_t u; ngx_int_t weight, max_conns, max_fails; ngx_uint_t i; +#if (NGX_STREAM_UPSTREAM_ZONE) + ngx_uint_t resolve; +#endif ngx_stream_upstream_server_t *us; us = ngx_array_push(uscf->servers); @@ -423,6 +431,9 @@ ngx_stream_upstream_server(ngx_conf_t *c max_conns = 0; max_fails = 1; fail_timeout = 10; +#if (NGX_STREAM_UPSTREAM_ZONE) + resolve = 0; +#endif for (i = 2; i < cf->args->nelts; i++) { @@ -511,6 +522,13 @@ ngx_stream_upstream_server(ngx_conf_t *c continue; } +#if (NGX_STREAM_UPSTREAM_ZONE) + if (ngx_strcmp(value[i].data, "resolve") == 0) { + resolve = 1; + continue; + } +#endif + goto invalid; } @@ -518,6 +536,13 @@ ngx_stream_upstream_server(ngx_conf_t *c u.url = value[1]; +#if (NGX_STREAM_UPSTREAM_ZONE) + if (resolve) { + /* resolve at run time */ + u.no_resolve = 1; + } +#endif + if (ngx_parse_url(cf->pool, &u) != NGX_OK) { if (u.err) { ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, @@ -534,8 +559,45 @@ ngx_stream_upstream_server(ngx_conf_t *c } us->name = u.url; + +#if (NGX_STREAM_UPSTREAM_ZONE) + + if (resolve && u.naddrs == 0) { + ngx_addr_t *addr; + + /* save port */ + + addr = ngx_pcalloc(cf->pool, sizeof(ngx_addr_t)); + if (addr == NULL) { + return NGX_CONF_ERROR; + } + + addr->sockaddr = ngx_palloc(cf->pool, u.socklen); + if (addr->sockaddr == NULL) { + return NGX_CONF_ERROR; + } + + ngx_memcpy(addr->sockaddr, &u.sockaddr, u.socklen); + + addr->socklen = u.socklen; + + us->addrs = addr; + us->naddrs = 1; + + us->host = u.host; + + } else { + us->addrs = u.addrs; + us->naddrs = u.naddrs; + } + +#else + us->addrs = u.addrs; us->naddrs = u.naddrs; + +#endif + us->weight = weight; us->max_conns = max_conns; us->max_fails = max_fails; diff --git a/src/stream/ngx_stream_upstream.h b/src/stream/ngx_stream_upstream.h --- a/src/stream/ngx_stream_upstream.h +++ b/src/stream/ngx_stream_upstream.h @@ -21,6 +21,7 @@ #define NGX_STREAM_UPSTREAM_FAIL_TIMEOUT 0x0008 #define NGX_STREAM_UPSTREAM_DOWN 0x0010 #define NGX_STREAM_UPSTREAM_BACKUP 0x0020 +#define NGX_STREAM_UPSTREAM_MODIFY 0x0040 #define NGX_STREAM_UPSTREAM_MAX_CONNS 0x0100 @@ -62,7 +63,11 @@ typedef struct { unsigned backup:1; - NGX_COMPAT_BEGIN(4) +#if (NGX_STREAM_UPSTREAM_ZONE) + ngx_str_t host; +#endif + + NGX_COMPAT_BEGIN(2) NGX_COMPAT_END } ngx_stream_upstream_server_t; @@ -83,6 +88,8 @@ struct ngx_stream_upstream_srv_conf_s { #if (NGX_STREAM_UPSTREAM_ZONE) ngx_shm_zone_t *shm_zone; + ngx_resolver_t *resolver; + ngx_msec_t resolver_timeout; #endif }; diff --git a/src/stream/ngx_stream_upstream_hash_module.c b/src/stream/ngx_stream_upstream_hash_module.c --- a/src/stream/ngx_stream_upstream_hash_module.c +++ b/src/stream/ngx_stream_upstream_hash_module.c @@ -23,6 +23,9 @@ typedef struct { typedef struct { +#if (NGX_STREAM_UPSTREAM_ZONE) + ngx_uint_t config; +#endif ngx_stream_complex_value_t key; ngx_stream_upstream_chash_points_t *points; } ngx_stream_upstream_hash_srv_conf_t; @@ -49,6 +52,8 @@ static ngx_int_t ngx_stream_upstream_get static ngx_int_t ngx_stream_upstream_init_chash(ngx_conf_t *cf, ngx_stream_upstream_srv_conf_t *us); +static ngx_int_t ngx_stream_upstream_update_chash(ngx_pool_t *pool, + ngx_stream_upstream_srv_conf_t *us); static int ngx_libc_cdecl ngx_stream_upstream_chash_cmp_points(const void *one, const void *two); static ngx_uint_t ngx_stream_upstream_find_chash_point( @@ -178,11 +183,18 @@ ngx_stream_upstream_get_hash_peer(ngx_pe ngx_stream_upstream_rr_peers_rlock(hp->rrp.peers); - if (hp->tries > 20 || hp->rrp.peers->single || hp->key.len == 0) { + if (hp->tries > 20 || hp->rrp.peers->number < 2 || hp->key.len == 0) { ngx_stream_upstream_rr_peers_unlock(hp->rrp.peers); return hp->get_rr_peer(pc, &hp->rrp); } +#if (NGX_STREAM_UPSTREAM_ZONE) + if (hp->rrp.peers->config && hp->rrp.config != *hp->rrp.peers->config) { + ngx_stream_upstream_rr_peers_unlock(hp->rrp.peers); + return hp->get_rr_peer(pc, &hp->rrp); + } +#endif + now = ngx_time(); pc->connection = NULL; @@ -261,6 +273,7 @@ ngx_stream_upstream_get_hash_peer(ngx_pe } hp->rrp.current = peer; + ngx_stream_upstream_rr_peer_ref(hp->rrp.peers, peer); pc->sockaddr = peer->sockaddr; pc->socklen = peer->socklen; @@ -285,6 +298,26 @@ static ngx_int_t ngx_stream_upstream_init_chash(ngx_conf_t *cf, ngx_stream_upstream_srv_conf_t *us) { + if (ngx_stream_upstream_init_round_robin(cf, us) != NGX_OK) { + return NGX_ERROR; + } + + us->peer.init = ngx_stream_upstream_init_chash_peer; + +#if (NGX_STREAM_UPSTREAM_ZONE) + if (us->shm_zone) { + return NGX_OK; + } +#endif + + return ngx_stream_upstream_update_chash(cf->pool, us); +} + + +static ngx_int_t +ngx_stream_upstream_update_chash(ngx_pool_t *pool, + ngx_stream_upstream_srv_conf_t *us) +{ u_char *host, *port, c; size_t host_len, port_len, size; uint32_t hash, base_hash; @@ -299,25 +332,33 @@ ngx_stream_upstream_init_chash(ngx_conf_ u_char byte[4]; } prev_hash; - if (ngx_stream_upstream_init_round_robin(cf, us) != NGX_OK) { - return NGX_ERROR; + hcf = ngx_stream_conf_upstream_srv_conf(us, + ngx_stream_upstream_hash_module); + + if (hcf->points) { + ngx_free(hcf->points); + hcf->points = NULL; } - us->peer.init = ngx_stream_upstream_init_chash_peer; - peers = us->peer.data; npoints = peers->total_weight * 160; size = sizeof(ngx_stream_upstream_chash_points_t) - + sizeof(ngx_stream_upstream_chash_point_t) * (npoints - 1); + - sizeof(ngx_stream_upstream_chash_point_t) + + sizeof(ngx_stream_upstream_chash_point_t) * npoints; - points = ngx_palloc(cf->pool, size); + points = pool ? ngx_palloc(pool, size) : ngx_alloc(size, ngx_cycle->log); if (points == NULL) { return NGX_ERROR; } points->number = 0; + if (npoints == 0) { + hcf->points = points; + return NGX_OK; + } + for (peer = peers->peer; peer; peer = peer->next) { server = &peer->server; @@ -401,8 +442,6 @@ ngx_stream_upstream_init_chash(ngx_conf_ points->number = i + 1; - hcf = ngx_stream_conf_upstream_srv_conf(us, - ngx_stream_upstream_hash_module); hcf->points = points; return NGX_OK; @@ -483,7 +522,22 @@ ngx_stream_upstream_init_chash_peer(ngx_ ngx_stream_upstream_rr_peers_rlock(hp->rrp.peers); - hp->hash = ngx_stream_upstream_find_chash_point(hcf->points, hash); +#if (NGX_STREAM_UPSTREAM_ZONE) + if (hp->rrp.peers->config + && (hcf->points == NULL || hcf->config != *hp->rrp.peers->config)) + { + if (ngx_stream_upstream_update_chash(NULL, us) != NGX_OK) { + ngx_stream_upstream_rr_peers_unlock(hp->rrp.peers); + return NGX_ERROR; + } + + hcf->config = *hp->rrp.peers->config; + } +#endif + + if (hcf->points->number) { + hp->hash = ngx_stream_upstream_find_chash_point(hcf->points, hash); + } ngx_stream_upstream_rr_peers_unlock(hp->rrp.peers); @@ -518,6 +572,20 @@ ngx_stream_upstream_get_chash_peer(ngx_p pc->connection = NULL; + if (hp->rrp.peers->number == 0) { + pc->name = hp->rrp.peers->name; + ngx_stream_upstream_rr_peers_unlock(hp->rrp.peers); + return NGX_BUSY; + } + +#if (NGX_STREAM_UPSTREAM_ZONE) + if (hp->rrp.peers->config && hp->rrp.config != *hp->rrp.peers->config) { + pc->name = hp->rrp.peers->name; + ngx_stream_upstream_rr_peers_unlock(hp->rrp.peers); + return NGX_BUSY; + } +#endif + now = ngx_time(); hcf = hp->conf; @@ -596,6 +664,7 @@ ngx_stream_upstream_get_chash_peer(ngx_p } hp->rrp.current = best; + ngx_stream_upstream_rr_peer_ref(hp->rrp.peers, best); pc->sockaddr = best->sockaddr; pc->socklen = best->socklen; @@ -663,6 +732,7 @@ ngx_stream_upstream_hash(ngx_conf_t *cf, } uscf->flags = NGX_STREAM_UPSTREAM_CREATE + |NGX_STREAM_UPSTREAM_MODIFY |NGX_STREAM_UPSTREAM_WEIGHT |NGX_STREAM_UPSTREAM_MAX_CONNS |NGX_STREAM_UPSTREAM_MAX_FAILS diff --git a/src/stream/ngx_stream_upstream_least_conn_module.c b/src/stream/ngx_stream_upstream_least_conn_module.c --- a/src/stream/ngx_stream_upstream_least_conn_module.c +++ b/src/stream/ngx_stream_upstream_least_conn_module.c @@ -120,6 +120,12 @@ ngx_stream_upstream_get_least_conn_peer( ngx_stream_upstream_rr_peers_wlock(peers); +#if (NGX_STREAM_UPSTREAM_ZONE) + if (peers->config && rrp->config != *peers->config) { + goto busy; + } +#endif + best = NULL; total = 0; @@ -240,6 +246,7 @@ ngx_stream_upstream_get_least_conn_peer( best->conns++; rrp->current = best; + ngx_stream_upstream_rr_peer_ref(peers, best); n = p / (8 * sizeof(uintptr_t)); m = (uintptr_t) 1 << p % (8 * sizeof(uintptr_t)); @@ -274,8 +281,18 @@ failed: } ngx_stream_upstream_rr_peers_wlock(peers); + +#if (NGX_STREAM_UPSTREAM_ZONE) + if (peers->config && rrp->config != *peers->config) { + goto busy; + } +#endif } +#if (NGX_STREAM_UPSTREAM_ZONE) +busy: +#endif + ngx_stream_upstream_rr_peers_unlock(peers); pc->name = peers->name; @@ -299,6 +316,7 @@ ngx_stream_upstream_least_conn(ngx_conf_ uscf->peer.init_upstream = ngx_stream_upstream_init_least_conn; uscf->flags = NGX_STREAM_UPSTREAM_CREATE + |NGX_STREAM_UPSTREAM_MODIFY |NGX_STREAM_UPSTREAM_WEIGHT |NGX_STREAM_UPSTREAM_MAX_CONNS |NGX_STREAM_UPSTREAM_MAX_FAILS diff --git a/src/stream/ngx_stream_upstream_random_module.c b/src/stream/ngx_stream_upstream_random_module.c --- a/src/stream/ngx_stream_upstream_random_module.c +++ b/src/stream/ngx_stream_upstream_random_module.c @@ -17,6 +17,9 @@ typedef struct { typedef struct { ngx_uint_t two; +#if (NGX_STREAM_UPSTREAM_ZONE) + ngx_uint_t config; +#endif ngx_stream_upstream_random_range_t *ranges; } ngx_stream_upstream_random_srv_conf_t; @@ -125,6 +128,11 @@ ngx_stream_upstream_update_random(ngx_po rcf = ngx_stream_conf_upstream_srv_conf(us, ngx_stream_upstream_random_module); + if (rcf->ranges) { + ngx_free(rcf->ranges); + rcf->ranges = NULL; + } + peers = us->peer.data; size = peers->number * sizeof(ngx_stream_upstream_random_range_t); @@ -186,11 +194,15 @@ ngx_stream_upstream_init_random_peer(ngx ngx_stream_upstream_rr_peers_rlock(rp->rrp.peers); #if (NGX_STREAM_UPSTREAM_ZONE) - if (rp->rrp.peers->shpool && rcf->ranges == NULL) { + if (rp->rrp.peers->config + && (rcf->ranges == NULL || rcf->config != *rp->rrp.peers->config)) + { if (ngx_stream_upstream_update_random(NULL, us) != NGX_OK) { ngx_stream_upstream_rr_peers_unlock(rp->rrp.peers); return NGX_ERROR; } + + rcf->config = *rp->rrp.peers->config; } #endif @@ -220,11 +232,18 @@ ngx_stream_upstream_get_random_peer(ngx_ ngx_stream_upstream_rr_peers_rlock(peers); - if (rp->tries > 20 || peers->single) { + if (rp->tries > 20 || peers->number < 2) { ngx_stream_upstream_rr_peers_unlock(peers); return ngx_stream_upstream_get_round_robin_peer(pc, rrp); } +#if (NGX_STREAM_UPSTREAM_ZONE) + if (peers->config && rrp->config != *peers->config) { + ngx_stream_upstream_rr_peers_unlock(peers); + return ngx_stream_upstream_get_round_robin_peer(pc, rrp); + } +#endif + pc->cached = 0; pc->connection = NULL; @@ -274,6 +293,7 @@ ngx_stream_upstream_get_random_peer(ngx_ } rrp->current = peer; + ngx_stream_upstream_rr_peer_ref(peers, peer); if (now - peer->checked > peer->fail_timeout) { peer->checked = now; @@ -314,11 +334,18 @@ ngx_stream_upstream_get_random2_peer(ngx ngx_stream_upstream_rr_peers_wlock(peers); - if (rp->tries > 20 || peers->single) { + if (rp->tries > 20 || peers->number < 2) { ngx_stream_upstream_rr_peers_unlock(peers); return ngx_stream_upstream_get_round_robin_peer(pc, rrp); } +#if (NGX_STREAM_UPSTREAM_ZONE) + if (peers->config && rrp->config != *peers->config) { + ngx_stream_upstream_rr_peers_unlock(peers); + return ngx_stream_upstream_get_round_robin_peer(pc, rrp); + } +#endif + pc->cached = 0; pc->connection = NULL; @@ -384,6 +411,7 @@ ngx_stream_upstream_get_random2_peer(ngx } rrp->current = peer; + ngx_stream_upstream_rr_peer_ref(peers, peer); if (now - peer->checked > peer->fail_timeout) { peer->checked = now; @@ -467,6 +495,7 @@ ngx_stream_upstream_random(ngx_conf_t *c uscf->peer.init_upstream = ngx_stream_upstream_init_random; uscf->flags = NGX_STREAM_UPSTREAM_CREATE + |NGX_STREAM_UPSTREAM_MODIFY |NGX_STREAM_UPSTREAM_WEIGHT |NGX_STREAM_UPSTREAM_MAX_CONNS |NGX_STREAM_UPSTREAM_MAX_FAILS diff --git a/src/stream/ngx_stream_upstream_round_robin.c b/src/stream/ngx_stream_upstream_round_robin.c --- a/src/stream/ngx_stream_upstream_round_robin.c +++ b/src/stream/ngx_stream_upstream_round_robin.c @@ -38,10 +38,15 @@ ngx_stream_upstream_init_round_robin(ngx ngx_stream_upstream_srv_conf_t *us) { ngx_url_t u; - ngx_uint_t i, j, n, w, t; + ngx_uint_t i, j, n, r, w, t; ngx_stream_upstream_server_t *server; ngx_stream_upstream_rr_peer_t *peer, **peerp; ngx_stream_upstream_rr_peers_t *peers, *backup; +#if (NGX_STREAM_UPSTREAM_ZONE) + ngx_uint_t resolve; + ngx_stream_core_srv_conf_t *cscf; + ngx_stream_upstream_rr_peer_t **rpeerp; +#endif us->peer.init = ngx_stream_upstream_init_round_robin_peer; @@ -49,23 +54,100 @@ ngx_stream_upstream_init_round_robin(ngx server = us->servers->elts; n = 0; + r = 0; w = 0; t = 0; +#if (NGX_STREAM_UPSTREAM_ZONE) + resolve = 0; +#endif + for (i = 0; i < us->servers->nelts; i++) { + +#if (NGX_STREAM_UPSTREAM_ZONE) + if (server[i].host.len) { + resolve = 1; + } +#endif + if (server[i].backup) { continue; } +#if (NGX_STREAM_UPSTREAM_ZONE) + if (server[i].host.len) { + r++; + + } else { + n += server[i].naddrs; + w += server[i].naddrs * server[i].weight; + + if (!server[i].down) { + t += server[i].naddrs; + } + } +#else n += server[i].naddrs; w += server[i].naddrs * server[i].weight; if (!server[i].down) { t += server[i].naddrs; } +#endif } - if (n == 0) { +#if (NGX_STREAM_UPSTREAM_ZONE) + if (us->shm_zone) { + + if (resolve && !(us->flags & NGX_STREAM_UPSTREAM_MODIFY)) { + ngx_log_error(NGX_LOG_EMERG, cf->log, 0, + "load balancing method does not support" + " resolving names at run time in" + " upstream \"%V\" in %s:%ui", + &us->host, us->file_name, us->line); + return NGX_ERROR; + } + + cscf = ngx_stream_conf_get_module_srv_conf(cf, + ngx_stream_core_module); + + us->resolver = cscf->resolver; + us->resolver_timeout = cscf->resolver_timeout; + + /* + * Without "resolver_timeout" in stream{}, the value is unset. + * Even if we set it in ngx_stream_core_merge_srv_conf(), it's + * still dependent on the module order and unreliable. + */ + ngx_conf_init_msec_value(us->resolver_timeout, 30000); + + if (resolve + && (us->resolver == NULL + || us->resolver->connections.nelts == 0)) + { + ngx_log_error(NGX_LOG_EMERG, cf->log, 0, + "no resolver defined to resolve names" + " at run time in upstream \"%V\" in %s:%ui", + &us->host, us->file_name, us->line); + return NGX_ERROR; + } + + } else if (resolve) { + + ngx_log_error(NGX_LOG_EMERG, cf->log, 0, + "resolving names at run time requires" + " upstream \"%V\" in %s:%ui" + " to be in shared memory", + &us->host, us->file_name, us->line); + return NGX_ERROR; + } +#endif + + if (n == 0 +#if (NGX_STREAM_UPSTREAM_ZONE) + && us->shm_zone == NULL +#endif + ) { ngx_log_error(NGX_LOG_EMERG, cf->log, 0, "no servers in upstream \"%V\" in %s:%ui", &us->host, us->file_name, us->line); @@ -77,7 +159,8 @@ ngx_stream_upstream_init_round_robin(ngx return NGX_ERROR; } - peer = ngx_pcalloc(cf->pool, sizeof(ngx_stream_upstream_rr_peer_t) * n); + peer = ngx_pcalloc(cf->pool, sizeof(ngx_stream_upstream_rr_peer_t) + * (n + r)); if (peer == NULL) { return NGX_ERROR; } @@ -92,11 +175,46 @@ ngx_stream_upstream_init_round_robin(ngx n = 0; peerp = &peers->peer; +#if (NGX_STREAM_UPSTREAM_ZONE) + rpeerp = &peers->resolve; +#endif + for (i = 0; i < us->servers->nelts; i++) { if (server[i].backup) { continue; } +#if (NGX_STREAM_UPSTREAM_ZONE) + if (server[i].host.len) { + + peer[n].host = ngx_pcalloc(cf->pool, + sizeof(ngx_stream_upstream_host_t)); + if (peer[n].host == NULL) { + return NGX_ERROR; + } + + peer[n].host->name = server[i].host; + + peer[n].sockaddr = server[i].addrs[0].sockaddr; + peer[n].socklen = server[i].addrs[0].socklen; + peer[n].name = server[i].addrs[0].name; + peer[n].weight = server[i].weight; + peer[n].effective_weight = server[i].weight; + peer[n].current_weight = 0; + peer[n].max_conns = server[i].max_conns; + peer[n].max_fails = server[i].max_fails; + peer[n].fail_timeout = server[i].fail_timeout; + peer[n].slow_start = server[i].slow_start; + peer[n].down = server[i].down; + peer[n].server = server[i].name; + *rpeerp = &peer[n]; + rpeerp = &peer[n].next; + n++; + + continue; + } +#endif + for (j = 0; j < server[i].naddrs; j++) { peer[n].sockaddr = server[i].addrs[j].sockaddr; peer[n].socklen = server[i].addrs[j].socklen; @@ -121,6 +239,7 @@ ngx_stream_upstream_init_round_robin(ngx /* backup servers */ n = 0; + r = 0; w = 0; t = 0; @@ -129,15 +248,37 @@ ngx_stream_upstream_init_round_robin(ngx continue; } +#if (NGX_STREAM_UPSTREAM_ZONE) + if (server[i].host.len) { + r++; + + } else { + n += server[i].naddrs; + w += server[i].naddrs * server[i].weight; + + if (!server[i].down) { + t += server[i].naddrs; + } + } +#else n += server[i].naddrs; w += server[i].naddrs * server[i].weight; if (!server[i].down) { t += server[i].naddrs; } +#endif } - if (n == 0) { + if (n == 0 +#if (NGX_STREAM_UPSTREAM_ZONE) + && us->shm_zone == NULL +#endif + ) { + return NGX_OK; + } + + if (n + r == 0 && !(us->flags & NGX_STREAM_UPSTREAM_BACKUP)) { return NGX_OK; } @@ -146,12 +287,16 @@ ngx_stream_upstream_init_round_robin(ngx return NGX_ERROR; } - peer = ngx_pcalloc(cf->pool, sizeof(ngx_stream_upstream_rr_peer_t) * n); + peer = ngx_pcalloc(cf->pool, sizeof(ngx_stream_upstream_rr_peer_t) + * (n + r)); if (peer == NULL) { return NGX_ERROR; } - peers->single = 0; + if (n > 0) { + peers->single = 0; + } + backup->single = 0; backup->number = n; backup->weighted = (w != n); @@ -162,11 +307,46 @@ ngx_stream_upstream_init_round_robin(ngx n = 0; peerp = &backup->peer; +#if (NGX_STREAM_UPSTREAM_ZONE) + rpeerp = &backup->resolve; +#endif + for (i = 0; i < us->servers->nelts; i++) { if (!server[i].backup) { continue; } +#if (NGX_STREAM_UPSTREAM_ZONE) + if (server[i].host.len) { + + peer[n].host = ngx_pcalloc(cf->pool, + sizeof(ngx_stream_upstream_host_t)); + if (peer[n].host == NULL) { + return NGX_ERROR; + } + + peer[n].host->name = server[i].host; + + peer[n].sockaddr = server[i].addrs[0].sockaddr; + peer[n].socklen = server[i].addrs[0].socklen; + peer[n].name = server[i].addrs[0].name; + peer[n].weight = server[i].weight; + peer[n].effective_weight = server[i].weight; + peer[n].current_weight = 0; + peer[n].max_conns = server[i].max_conns; + peer[n].max_fails = server[i].max_fails; + peer[n].fail_timeout = server[i].fail_timeout; + peer[n].slow_start = server[i].slow_start; + peer[n].down = server[i].down; + peer[n].server = server[i].name; + *rpeerp = &peer[n]; + rpeerp = &peer[n].next; + n++; + + continue; + } +#endif + for (j = 0; j < server[i].naddrs; j++) { peer[n].sockaddr = server[i].addrs[j].sockaddr; peer[n].socklen = server[i].addrs[j].socklen; @@ -280,7 +460,12 @@ ngx_stream_upstream_init_round_robin_pee rrp->peers = us->peer.data; rrp->current = NULL; - rrp->config = 0; + + ngx_stream_upstream_rr_peers_rlock(rrp->peers); + +#if (NGX_STREAM_UPSTREAM_ZONE) + rrp->config = rrp->peers->config ? *rrp->peers->config : 0; +#endif n = rrp->peers->number; @@ -288,6 +473,10 @@ ngx_stream_upstream_init_round_robin_pee n = rrp->peers->next->number; } + s->upstream->peer.tries = ngx_stream_upstream_tries(rrp->peers); + + ngx_stream_upstream_rr_peers_unlock(rrp->peers); + if (n <= 8 * sizeof(uintptr_t)) { rrp->tried = &rrp->data; rrp->data = 0; @@ -304,7 +493,7 @@ ngx_stream_upstream_init_round_robin_pee s->upstream->peer.get = ngx_stream_upstream_get_round_robin_peer; s->upstream->peer.free = ngx_stream_upstream_free_round_robin_peer; s->upstream->peer.notify = ngx_stream_upstream_notify_round_robin_peer; - s->upstream->peer.tries = ngx_stream_upstream_tries(rrp->peers); + #if (NGX_STREAM_SSL) s->upstream->peer.set_session = ngx_stream_upstream_set_round_robin_peer_session; @@ -455,6 +644,12 @@ ngx_stream_upstream_get_round_robin_peer peers = rrp->peers; ngx_stream_upstream_rr_peers_wlock(peers); +#if (NGX_STREAM_UPSTREAM_ZONE) + if (peers->config && rrp->config != *peers->config) { + goto busy; + } +#endif + if (peers->single) { peer = peers->peer; @@ -467,6 +662,7 @@ ngx_stream_upstream_get_round_robin_peer } rrp->current = peer; + ngx_stream_upstream_rr_peer_ref(peers, peer); } else { @@ -517,8 +713,18 @@ failed: } ngx_stream_upstream_rr_peers_wlock(peers); + +#if (NGX_STREAM_UPSTREAM_ZONE) + if (peers->config && rrp->config != *peers->config) { + goto busy; + } +#endif } +#if (NGX_STREAM_UPSTREAM_ZONE) +busy: +#endif + ngx_stream_upstream_rr_peers_unlock(peers); pc->name = peers->name; @@ -589,6 +795,7 @@ ngx_stream_upstream_get_peer(ngx_stream_ } rrp->current = best; + ngx_stream_upstream_rr_peer_ref(rrp->peers, best); n = p / (8 * sizeof(uintptr_t)); m = (uintptr_t) 1 << p % (8 * sizeof(uintptr_t)); @@ -623,9 +830,17 @@ ngx_stream_upstream_free_round_robin_pee ngx_stream_upstream_rr_peer_lock(rrp->peers, peer); if (rrp->peers->single) { + + if (peer->fails) { + peer->fails = 0; + } + peer->conns--; - ngx_stream_upstream_rr_peer_unlock(rrp->peers, peer); + if (ngx_stream_upstream_rr_peer_unref(rrp->peers, peer) == NGX_OK) { + ngx_stream_upstream_rr_peer_unlock(rrp->peers, peer); + } + ngx_stream_upstream_rr_peers_unlock(rrp->peers); pc->tries = 0; @@ -667,7 +882,10 @@ ngx_stream_upstream_free_round_robin_pee peer->conns--; - ngx_stream_upstream_rr_peer_unlock(rrp->peers, peer); + if (ngx_stream_upstream_rr_peer_unref(rrp->peers, peer) == NGX_OK) { + ngx_stream_upstream_rr_peer_unlock(rrp->peers, peer); + } + ngx_stream_upstream_rr_peers_unlock(rrp->peers); if (pc->tries) { diff --git a/src/stream/ngx_stream_upstream_round_robin.h b/src/stream/ngx_stream_upstream_round_robin.h --- a/src/stream/ngx_stream_upstream_round_robin.h +++ b/src/stream/ngx_stream_upstream_round_robin.h @@ -14,8 +14,23 @@ #include +typedef struct ngx_stream_upstream_rr_peers_s ngx_stream_upstream_rr_peers_t; typedef struct ngx_stream_upstream_rr_peer_s ngx_stream_upstream_rr_peer_t; + +#if (NGX_STREAM_UPSTREAM_ZONE) + +typedef struct { + ngx_event_t event; /* must be first */ + ngx_uint_t worker; + ngx_str_t name; + ngx_stream_upstream_rr_peers_t *peers; + ngx_stream_upstream_rr_peer_t *peer; +} ngx_stream_upstream_host_t; + +#endif + + struct ngx_stream_upstream_rr_peer_s { struct sockaddr *sockaddr; socklen_t socklen; @@ -44,7 +59,12 @@ struct ngx_stream_upstream_rr_peer_s { int ssl_session_len; #if (NGX_STREAM_UPSTREAM_ZONE) + unsigned zombie:1; + ngx_atomic_t lock; + ngx_uint_t id; + ngx_uint_t refs; + ngx_stream_upstream_host_t *host; #endif ngx_stream_upstream_rr_peer_t *next; @@ -54,8 +74,6 @@ struct ngx_stream_upstream_rr_peer_s { }; -typedef struct ngx_stream_upstream_rr_peers_s ngx_stream_upstream_rr_peers_t; - struct ngx_stream_upstream_rr_peers_s { ngx_uint_t number; @@ -76,6 +94,12 @@ struct ngx_stream_upstream_rr_peers_s { ngx_stream_upstream_rr_peers_t *next; ngx_stream_upstream_rr_peer_t *peer; + +#if (NGX_STREAM_UPSTREAM_ZONE) + ngx_uint_t *config; + ngx_stream_upstream_rr_peer_t *resolve; + ngx_uint_t zombies; +#endif }; @@ -112,6 +136,67 @@ struct ngx_stream_upstream_rr_peers_s { ngx_rwlock_unlock(&peer->lock); \ } + +#define ngx_stream_upstream_rr_peer_ref(peers, peer) \ + (peer)->refs++; + + +static ngx_inline void +ngx_stream_upstream_rr_peer_free_locked(ngx_stream_upstream_rr_peers_t *peers, + ngx_stream_upstream_rr_peer_t *peer) +{ + if (peer->refs) { + peer->zombie = 1; + peers->zombies++; + return; + } + + ngx_slab_free_locked(peers->shpool, peer->sockaddr); + ngx_slab_free_locked(peers->shpool, peer->name.data); + + if (peer->server.data && (peer->host == NULL || peer->host->peer == peer)) { + ngx_slab_free_locked(peers->shpool, peer->server.data); + } + +#if (NGX_STREAM_SSL) + if (peer->ssl_session) { + ngx_slab_free_locked(peers->shpool, peer->ssl_session); + } +#endif + + ngx_slab_free_locked(peers->shpool, peer); +} + + +static ngx_inline void +ngx_stream_upstream_rr_peer_free(ngx_stream_upstream_rr_peers_t *peers, + ngx_stream_upstream_rr_peer_t *peer) +{ + ngx_shmtx_lock(&peers->shpool->mutex); + ngx_stream_upstream_rr_peer_free_locked(peers, peer); + ngx_shmtx_unlock(&peers->shpool->mutex); +} + + +static ngx_inline ngx_int_t +ngx_stream_upstream_rr_peer_unref(ngx_stream_upstream_rr_peers_t *peers, + ngx_stream_upstream_rr_peer_t *peer) +{ + peer->refs--; + + if (peers->shpool == NULL) { + return NGX_OK; + } + + if (peer->refs == 0 && peer->zombie) { + ngx_stream_upstream_rr_peer_free(peers, peer); + peers->zombies--; + return NGX_DONE; + } + + return NGX_OK; +} + #else #define ngx_stream_upstream_rr_peers_rlock(peers) @@ -119,6 +204,8 @@ struct ngx_stream_upstream_rr_peers_s { #define ngx_stream_upstream_rr_peers_unlock(peers) #define ngx_stream_upstream_rr_peer_lock(peers, peer) #define ngx_stream_upstream_rr_peer_unlock(peers, peer) +#define ngx_stream_upstream_rr_peer_ref(peers, peer) +#define ngx_stream_upstream_rr_peer_unref(peers, peer) NGX_OK #endif diff --git a/src/stream/ngx_stream_upstream_zone_module.c b/src/stream/ngx_stream_upstream_zone_module.c --- a/src/stream/ngx_stream_upstream_zone_module.c +++ b/src/stream/ngx_stream_upstream_zone_module.c @@ -18,6 +18,10 @@ static ngx_stream_upstream_rr_peers_t *n ngx_slab_pool_t *shpool, ngx_stream_upstream_srv_conf_t *uscf); static ngx_stream_upstream_rr_peer_t *ngx_stream_upstream_zone_copy_peer( ngx_stream_upstream_rr_peers_t *peers, ngx_stream_upstream_rr_peer_t *src); +static void ngx_stream_upstream_zone_set_single( + ngx_stream_upstream_srv_conf_t *uscf); +static void ngx_stream_upstream_zone_remove_peer_locked( + ngx_stream_upstream_rr_peers_t *peers, ngx_stream_upstream_rr_peer_t *peer); static ngx_command_t ngx_stream_upstream_zone_commands[] = { @@ -33,6 +37,11 @@ static ngx_command_t ngx_stream_upstrea }; +static ngx_int_t ngx_stream_upstream_zone_init_worker(ngx_cycle_t *cycle); +static void ngx_stream_upstream_zone_resolve_timer(ngx_event_t *event); +static void ngx_stream_upstream_zone_resolve_handler(ngx_resolver_ctx_t *ctx); + + static ngx_stream_module_t ngx_stream_upstream_zone_module_ctx = { NULL, /* preconfiguration */ NULL, /* postconfiguration */ @@ -52,7 +61,7 @@ ngx_module_t ngx_stream_upstream_zone_m NGX_STREAM_MODULE, /* module type */ NULL, /* init master */ NULL, /* init module */ - NULL, /* init process */ + ngx_stream_upstream_zone_init_worker, /* init process */ NULL, /* init thread */ NULL, /* exit thread */ NULL, /* exit process */ @@ -185,9 +194,15 @@ ngx_stream_upstream_zone_copy_peers(ngx_ ngx_stream_upstream_srv_conf_t *uscf) { ngx_str_t *name; + ngx_uint_t *config; ngx_stream_upstream_rr_peer_t *peer, **peerp; ngx_stream_upstream_rr_peers_t *peers, *backup; + config = ngx_slab_calloc(shpool, sizeof(ngx_uint_t)); + if (config == NULL) { + return NULL; + } + peers = ngx_slab_alloc(shpool, sizeof(ngx_stream_upstream_rr_peers_t)); if (peers == NULL) { return NULL; @@ -211,6 +226,7 @@ ngx_stream_upstream_zone_copy_peers(ngx_ peers->name = name; peers->shpool = shpool; + peers->config = config; for (peerp = &peers->peer; *peerp; peerp = &peer->next) { /* pool is unlocked */ @@ -220,6 +236,17 @@ ngx_stream_upstream_zone_copy_peers(ngx_ } *peerp = peer; + peer->id = (*peers->config)++; + } + + for (peerp = &peers->resolve; *peerp; peerp = &peer->next) { + peer = ngx_stream_upstream_zone_copy_peer(peers, *peerp); + if (peer == NULL) { + return NULL; + } + + *peerp = peer; + peer->id = (*peers->config)++; } if (peers->next == NULL) { @@ -236,6 +263,7 @@ ngx_stream_upstream_zone_copy_peers(ngx_ backup->name = name; backup->shpool = shpool; + backup->config = config; for (peerp = &backup->peer; *peerp; peerp = &peer->next) { /* pool is unlocked */ @@ -245,6 +273,17 @@ ngx_stream_upstream_zone_copy_peers(ngx_ } *peerp = peer; + peer->id = (*backup->config)++; + } + + for (peerp = &backup->resolve; *peerp; peerp = &peer->next) { + peer = ngx_stream_upstream_zone_copy_peer(backup, *peerp); + if (peer == NULL) { + return NULL; + } + + *peerp = peer; + peer->id = (*backup->config)++; } peers->next = backup; @@ -276,6 +315,7 @@ ngx_stream_upstream_zone_copy_peer(ngx_s dst->sockaddr = NULL; dst->name.data = NULL; dst->server.data = NULL; + dst->host = NULL; } dst->sockaddr = ngx_slab_calloc_locked(pool, sizeof(ngx_sockaddr_t)); @@ -298,12 +338,37 @@ ngx_stream_upstream_zone_copy_peer(ngx_s } ngx_memcpy(dst->server.data, src->server.data, src->server.len); + + if (src->host) { + dst->host = ngx_slab_calloc_locked(pool, + sizeof(ngx_stream_upstream_host_t)); + if (dst->host == NULL) { + goto failed; + } + + dst->host->name.data = ngx_slab_alloc_locked(pool, + src->host->name.len); + if (dst->host->name.data == NULL) { + goto failed; + } + + dst->host->peers = peers; + dst->host->peer = dst; + + dst->host->name.len = src->host->name.len; + ngx_memcpy(dst->host->name.data, src->host->name.data, + src->host->name.len); + } } return dst; failed: + if (dst->host) { + ngx_slab_free_locked(pool, dst->host); + } + if (dst->server.data) { ngx_slab_free_locked(pool, dst->server.data); } @@ -320,3 +385,338 @@ failed: return NULL; } + + +static void +ngx_stream_upstream_zone_set_single(ngx_stream_upstream_srv_conf_t *uscf) +{ + ngx_stream_upstream_rr_peers_t *peers; + + peers = uscf->peer.data; + + if (peers->number == 1 + && (peers->next == NULL || peers->next->number == 0)) + { + peers->single = 1; + + } else { + peers->single = 0; + } +} + + +static void +ngx_stream_upstream_zone_remove_peer_locked( + ngx_stream_upstream_rr_peers_t *peers, ngx_stream_upstream_rr_peer_t *peer) +{ + peers->total_weight -= peer->weight; + peers->number--; + peers->tries -= (peer->down == 0); + (*peers->config)++; + peers->weighted = (peers->total_weight != peers->number); + + ngx_stream_upstream_rr_peer_free(peers, peer); +} + + +static ngx_int_t +ngx_stream_upstream_zone_init_worker(ngx_cycle_t *cycle) +{ + ngx_uint_t i; + ngx_event_t *event; + ngx_stream_upstream_rr_peer_t *peer; + ngx_stream_upstream_rr_peers_t *peers; + ngx_stream_upstream_srv_conf_t *uscf, **uscfp; + ngx_stream_upstream_main_conf_t *umcf; + + if (ngx_process != NGX_PROCESS_WORKER + && ngx_process != NGX_PROCESS_SINGLE) + { + return NGX_OK; + } + + umcf = ngx_stream_cycle_get_module_main_conf(cycle, + ngx_stream_upstream_module); + + if (umcf == NULL) { + return NGX_OK; + } + + uscfp = umcf->upstreams.elts; + + for (i = 0; i < umcf->upstreams.nelts; i++) { + + uscf = uscfp[i]; + + if (uscf->shm_zone == NULL) { + continue; + } + + peers = uscf->peer.data; + + do { + ngx_stream_upstream_rr_peers_wlock(peers); + + for (peer = peers->resolve; peer; peer = peer->next) { + + if (peer->host->worker != ngx_worker) { + continue; + } + + event = &peer->host->event; + ngx_memzero(event, sizeof(ngx_event_t)); + + event->data = uscf; + event->handler = ngx_stream_upstream_zone_resolve_timer; + event->log = cycle->log; + event->cancelable = 1; + + ngx_stream_upstream_rr_peer_ref(peers, peer); + ngx_add_timer(event, 1); + } + + ngx_stream_upstream_rr_peers_unlock(peers); + + peers = peers->next; + + } while (peers); + } + + return NGX_OK; +} + + +static void +ngx_stream_upstream_zone_resolve_timer(ngx_event_t *event) +{ + ngx_resolver_ctx_t *ctx; + ngx_stream_upstream_host_t *host; + ngx_stream_upstream_rr_peer_t *template; + ngx_stream_upstream_rr_peers_t *peers; + ngx_stream_upstream_srv_conf_t *uscf; + + host = (ngx_stream_upstream_host_t *) event; + uscf = event->data; + peers = host->peers; + template = host->peer; + + if (template->zombie) { + (void) ngx_stream_upstream_rr_peer_unref(peers, template); + + ngx_shmtx_lock(&peers->shpool->mutex); + + if (host->service.len) { + ngx_slab_free_locked(peers->shpool, host->service.data); + } + + ngx_slab_free_locked(peers->shpool, host->name.data); + ngx_slab_free_locked(peers->shpool, host); + ngx_shmtx_unlock(&peers->shpool->mutex); + + return; + } + + ctx = ngx_resolve_start(uscf->resolver, NULL); + if (ctx == NULL) { + goto retry; + } + + if (ctx == NGX_NO_RESOLVER) { + ngx_log_error(NGX_LOG_ERR, event->log, 0, + "no resolver defined to resolve %V", &host->name); + return; + } + + ctx->name = host->name; + ctx->handler = ngx_stream_upstream_zone_resolve_handler; + ctx->data = host; + ctx->timeout = uscf->resolver_timeout; + ctx->cancelable = 1; + + if (ngx_resolve_name(ctx) == NGX_OK) { + return; + } + +retry: + + ngx_add_timer(event, ngx_max(uscf->resolver_timeout, 1000)); +} + + +static void +ngx_stream_upstream_zone_resolve_handler(ngx_resolver_ctx_t *ctx) +{ + time_t now; + in_port_t port; + ngx_msec_t timer; + ngx_uint_t i, j; + ngx_event_t *event; + ngx_resolver_addr_t *addr; + ngx_stream_upstream_host_t *host; + ngx_stream_upstream_rr_peer_t *peer, *template, **peerp; + ngx_stream_upstream_rr_peers_t *peers; + ngx_stream_upstream_srv_conf_t *uscf; + + host = ctx->data; + event = &host->event; + uscf = event->data; + peers = host->peers; + template = host->peer; + + ngx_stream_upstream_rr_peers_wlock(peers); + + if (template->zombie) { + (void) ngx_stream_upstream_rr_peer_unref(peers, template); + + ngx_stream_upstream_rr_peers_unlock(peers); + + ngx_shmtx_lock(&peers->shpool->mutex); + ngx_slab_free_locked(peers->shpool, host->name.data); + ngx_slab_free_locked(peers->shpool, host); + ngx_shmtx_unlock(&peers->shpool->mutex); + + ngx_resolve_name_done(ctx); + + return; + } + + now = ngx_time(); + + if (ctx->state) { + ngx_log_error(NGX_LOG_ERR, event->log, 0, + "%V could not be resolved (%i: %s)", + &ctx->name, ctx->state, + ngx_resolver_strerror(ctx->state)); + + if (ctx->state != NGX_RESOLVE_NXDOMAIN) { + ngx_stream_upstream_rr_peers_unlock(peers); + + ngx_resolve_name_done(ctx); + + ngx_add_timer(event, ngx_max(uscf->resolver_timeout, 1000)); + return; + } + + /* NGX_RESOLVE_NXDOMAIN */ + + ctx->naddrs = 0; + } + +#if (NGX_DEBUG) + { + u_char text[NGX_SOCKADDR_STRLEN]; + size_t len; + + for (i = 0; i < ctx->naddrs; i++) { + len = ngx_sock_ntop(ctx->addrs[i].sockaddr, ctx->addrs[i].socklen, + text, NGX_SOCKADDR_STRLEN, 0); + + ngx_log_debug3(NGX_LOG_DEBUG_STREAM, event->log, 0, + "name %V was resolved to %*s", &host->name, len, text); + } + } +#endif + + for (peerp = &peers->peer; *peerp; /* void */ ) { + peer = *peerp; + + if (peer->host != host) { + goto next; + } + + for (j = 0; j < ctx->naddrs; j++) { + + addr = &ctx->addrs[j]; + + if (addr->name.len == 0 + && ngx_cmp_sockaddr(peer->sockaddr, peer->socklen, + addr->sockaddr, addr->socklen, 0) + == NGX_OK) + { + addr->name.len = 1; + goto next; + } + } + + *peerp = peer->next; + ngx_stream_upstream_zone_remove_peer_locked(peers, peer); + + ngx_stream_upstream_zone_set_single(uscf); + + continue; + + next: + + peerp = &peer->next; + } + + for (i = 0; i < ctx->naddrs; i++) { + + addr = &ctx->addrs[i]; + + if (addr->name.len == 1) { + addr->name.len = 0; + continue; + } + + ngx_shmtx_lock(&peers->shpool->mutex); + peer = ngx_stream_upstream_zone_copy_peer(peers, NULL); + ngx_shmtx_unlock(&peers->shpool->mutex); + if (peer == NULL) { + ngx_log_error(NGX_LOG_ERR, event->log, 0, + "cannot add new server to upstream \"%V\", " + "memory exhausted", peers->name); + break; + } + + ngx_memcpy(peer->sockaddr, addr->sockaddr, addr->socklen); + + port = ((struct sockaddr_in *) template->sockaddr)->sin_port; + + switch (peer->sockaddr->sa_family) { +#if (NGX_HAVE_INET6) + case AF_INET6: + ((struct sockaddr_in6 *) peer->sockaddr)->sin6_port = port; + break; +#endif + default: /* AF_INET */ + ((struct sockaddr_in *) peer->sockaddr)->sin_port = port; + } + + peer->socklen = addr->socklen; + + peer->name.len = ngx_sock_ntop(peer->sockaddr, peer->socklen, + peer->name.data, NGX_SOCKADDR_STRLEN, 1); + + peer->host = template->host; + peer->server = template->server; + + peer->weight = template->weight; + peer->effective_weight = peer->weight; + peer->max_conns = template->max_conns; + peer->max_fails = template->max_fails; + peer->fail_timeout = template->fail_timeout; + peer->slow_start = template->slow_start; + peer->down = template->down; + + *peerp = peer; + peerp = &peer->next; + + peers->number++; + peers->tries += (peer->down == 0); + peers->total_weight += peer->weight; + peers->weighted = (peers->total_weight != peers->number); + peer->id = (*peers->config)++; + + ngx_stream_upstream_zone_set_single(uscf); + } + + ngx_stream_upstream_rr_peers_unlock(peers); + + timer = (ngx_msec_t) 1000 * (ctx->valid > now ? ctx->valid - now + 1 : 1); + timer = ngx_min(timer, uscf->resolver_timeout); + + ngx_resolve_name_done(ctx); + + ngx_add_timer(event, timer); +} From a.bavshin at nginx.com Wed Feb 1 01:36:58 2023 From: a.bavshin at nginx.com (=?iso-8859-1?q?Aleksei_Bavshin?=) Date: Tue, 31 Jan 2023 17:36:58 -0800 Subject: [PATCH 3 of 6] Upstream: construct upstream peers from DNS SRV records In-Reply-To: References: Message-ID: <95ea48662e8a8b7ee73b.1675215418@fedora-wsl.> # HG changeset patch # User Dmitry Volyntsev # Date 1458229351 -10800 # Thu Mar 17 18:42:31 2016 +0300 # Node ID 95ea48662e8a8b7ee73b39c5791c06b5e17e3102 # Parent 09954272153fb0ad62a80799a854a22fd9ebcdc9 Upstream: construct upstream peers from DNS SRV records. diff --git a/src/http/modules/ngx_http_upstream_zone_module.c b/src/http/modules/ngx_http_upstream_zone_module.c --- a/src/http/modules/ngx_http_upstream_zone_module.c +++ b/src/http/modules/ngx_http_upstream_zone_module.c @@ -361,6 +361,18 @@ ngx_http_upstream_zone_copy_peer(ngx_htt dst->host->name.len = src->host->name.len; ngx_memcpy(dst->host->name.data, src->host->name.data, src->host->name.len); + + if (src->host->service.len) { + dst->host->service.data = ngx_slab_alloc_locked(pool, + src->host->service.len); + if (dst->host->service.data == NULL) { + goto failed; + } + + dst->host->service.len = src->host->service.len; + ngx_memcpy(dst->host->service.data, src->host->service.data, + src->host->service.len); + } } } @@ -369,6 +381,10 @@ ngx_http_upstream_zone_copy_peer(ngx_htt failed: if (dst->host) { + if (dst->host->name.data) { + ngx_slab_free_locked(pool, dst->host->name.data); + } + ngx_slab_free_locked(pool, dst->host); } @@ -533,6 +549,7 @@ ngx_http_upstream_zone_resolve_timer(ngx ctx->handler = ngx_http_upstream_zone_resolve_handler; ctx->data = host; ctx->timeout = uscf->resolver_timeout; + ctx->service = host->service; ctx->cancelable = 1; if (ngx_resolve_name(ctx) == NGX_OK) { @@ -545,15 +562,28 @@ retry: } +#define ngx_http_upstream_zone_addr_marked(addr) \ + ((uintptr_t) (addr)->sockaddr & 1) + +#define ngx_http_upstream_zone_mark_addr(addr) \ + (addr)->sockaddr = (struct sockaddr *) ((uintptr_t) (addr)->sockaddr | 1) + +#define ngx_http_upstream_zone_unmark_addr(addr) \ + (addr)->sockaddr = \ + (struct sockaddr *) ((uintptr_t) (addr)->sockaddr & ~((uintptr_t) 1)) + static void ngx_http_upstream_zone_resolve_handler(ngx_resolver_ctx_t *ctx) { time_t now; + u_short min_priority; in_port_t port; + ngx_str_t *server; ngx_msec_t timer; - ngx_uint_t i, j; + ngx_uint_t i, j, backup, addr_backup; ngx_event_t *event; ngx_resolver_addr_t *addr; + ngx_resolver_srv_name_t *srv; ngx_http_upstream_host_t *host; ngx_http_upstream_rr_peer_t *peer, *template, **peerp; ngx_http_upstream_rr_peers_t *peers; @@ -573,6 +603,11 @@ ngx_http_upstream_zone_resolve_handler(n ngx_http_upstream_rr_peers_unlock(peers); ngx_shmtx_lock(&peers->shpool->mutex); + + if (host->service.len) { + ngx_slab_free_locked(peers->shpool, host->service.data); + } + ngx_slab_free_locked(peers->shpool, host->name.data); ngx_slab_free_locked(peers->shpool, host); ngx_shmtx_unlock(&peers->shpool->mutex); @@ -584,11 +619,32 @@ ngx_http_upstream_zone_resolve_handler(n now = ngx_time(); + for (i = 0; i < ctx->nsrvs; i++) { + srv = &ctx->srvs[i]; + + if (srv->state) { + ngx_log_error(NGX_LOG_ERR, event->log, 0, + "%V could not be resolved (%i: %s) " + "while resolving service %V of %V", + &srv->name, srv->state, + ngx_resolver_strerror(srv->state), &ctx->service, + &ctx->name); + } + } + if (ctx->state) { - ngx_log_error(NGX_LOG_ERR, event->log, 0, - "%V could not be resolved (%i: %s)", - &ctx->name, ctx->state, - ngx_resolver_strerror(ctx->state)); + if (ctx->service.len) { + ngx_log_error(NGX_LOG_ERR, event->log, 0, + "service %V of %V could not be resolved (%i: %s)", + &ctx->service, &ctx->name, ctx->state, + ngx_resolver_strerror(ctx->state)); + + } else { + ngx_log_error(NGX_LOG_ERR, event->log, 0, + "%V could not be resolved (%i: %s)", + &ctx->name, ctx->state, + ngx_resolver_strerror(ctx->state)); + } if (ctx->state != NGX_RESOLVE_NXDOMAIN) { ngx_http_upstream_rr_peers_unlock(peers); @@ -604,6 +660,13 @@ ngx_http_upstream_zone_resolve_handler(n ctx->naddrs = 0; } + backup = 0; + min_priority = 65535; + + for (i = 0; i < ctx->naddrs; i++) { + min_priority = ngx_min(ctx->addrs[i].priority, min_priority); + } + #if (NGX_DEBUG) { u_char text[NGX_SOCKADDR_STRLEN]; @@ -611,14 +674,20 @@ ngx_http_upstream_zone_resolve_handler(n for (i = 0; i < ctx->naddrs; i++) { len = ngx_sock_ntop(ctx->addrs[i].sockaddr, ctx->addrs[i].socklen, - text, NGX_SOCKADDR_STRLEN, 0); + text, NGX_SOCKADDR_STRLEN, 1); - ngx_log_debug3(NGX_LOG_DEBUG_HTTP, event->log, 0, - "name %V was resolved to %*s", &host->name, len, text); + ngx_log_debug7(NGX_LOG_DEBUG_HTTP, event->log, 0, + "name %V was resolved to %*s " + "s:\"%V\" n:\"%V\" w:%d %s", + &host->name, len, text, &host->service, + &ctx->addrs[i].name, ctx->addrs[i].weight, + ctx->addrs[i].priority != min_priority ? "backup" : ""); } } #endif +again: + for (peerp = &peers->peer; *peerp; /* void */ ) { peer = *peerp; @@ -630,14 +699,39 @@ ngx_http_upstream_zone_resolve_handler(n addr = &ctx->addrs[j]; - if (addr->name.len == 0 - && ngx_cmp_sockaddr(peer->sockaddr, peer->socklen, - addr->sockaddr, addr->socklen, 0) - == NGX_OK) + addr_backup = (addr->priority != min_priority); + if (addr_backup != backup) { + continue; + } + + if (ngx_http_upstream_zone_addr_marked(addr)) { + continue; + } + + if (ngx_cmp_sockaddr(peer->sockaddr, peer->socklen, + addr->sockaddr, addr->socklen, + host->service.len != 0) + != NGX_OK) { - addr->name.len = 1; - goto next; + continue; } + + if (host->service.len) { + if (addr->name.len != peer->server.len + || ngx_strncmp(addr->name.data, peer->server.data, + addr->name.len)) + { + continue; + } + + if (template->weight == 1 && addr->weight != peer->weight) { + continue; + } + } + + ngx_http_upstream_zone_mark_addr(addr); + + goto next; } *peerp = peer->next; @@ -656,8 +750,13 @@ ngx_http_upstream_zone_resolve_handler(n addr = &ctx->addrs[i]; - if (addr->name.len == 1) { - addr->name.len = 0; + addr_backup = (addr->priority != min_priority); + if (addr_backup != backup) { + continue; + } + + if (ngx_http_upstream_zone_addr_marked(addr)) { + ngx_http_upstream_zone_unmark_addr(addr); continue; } @@ -669,21 +768,14 @@ ngx_http_upstream_zone_resolve_handler(n ngx_log_error(NGX_LOG_ERR, event->log, 0, "cannot add new server to upstream \"%V\", " "memory exhausted", peers->name); - break; + goto done; } ngx_memcpy(peer->sockaddr, addr->sockaddr, addr->socklen); - port = ((struct sockaddr_in *) template->sockaddr)->sin_port; - - switch (peer->sockaddr->sa_family) { -#if (NGX_HAVE_INET6) - case AF_INET6: - ((struct sockaddr_in6 *) peer->sockaddr)->sin6_port = port; - break; -#endif - default: /* AF_INET */ - ((struct sockaddr_in *) peer->sockaddr)->sin_port = port; + if (host->service.len == 0) { + port = ngx_inet_get_port(template->sockaddr); + ngx_inet_set_port(peer->sockaddr, port); } peer->socklen = addr->socklen; @@ -692,9 +784,30 @@ ngx_http_upstream_zone_resolve_handler(n peer->name.data, NGX_SOCKADDR_STRLEN, 1); peer->host = template->host; - peer->server = template->server; + + server = host->service.len ? &addr->name : &template->server; + + peer->server.data = ngx_slab_alloc(peers->shpool, server->len); + if (peer->server.data == NULL) { + ngx_http_upstream_rr_peer_free(peers, peer); - peer->weight = template->weight; + ngx_log_error(NGX_LOG_ERR, event->log, 0, + "cannot add new server to upstream \"%V\", " + "memory exhausted", peers->name); + goto done; + } + + peer->server.len = server->len; + ngx_memcpy(peer->server.data, server->data, server->len); + + if (host->service.len == 0) { + peer->weight = template->weight; + + } else { + peer->weight = (template->weight != 1 ? template->weight + : addr->weight); + } + peer->effective_weight = peer->weight; peer->max_conns = template->max_conns; peer->max_fails = template->max_fails; @@ -714,8 +827,25 @@ ngx_http_upstream_zone_resolve_handler(n ngx_http_upstream_zone_set_single(uscf); } + if (host->service.len && peers->next) { + ngx_http_upstream_rr_peers_unlock(peers); + + peers = peers->next; + backup = 1; + + ngx_http_upstream_rr_peers_wlock(peers); + + goto again; + } + +done: + ngx_http_upstream_rr_peers_unlock(peers); + while (++i < ctx->naddrs) { + ngx_http_upstream_zone_unmark_addr(&ctx->addrs[i]); + } + timer = (ngx_msec_t) 1000 * (ctx->valid > now ? ctx->valid - now + 1 : 1); timer = ngx_min(timer, uscf->resolver_timeout); diff --git a/src/http/ngx_http_upstream.c b/src/http/ngx_http_upstream.c --- a/src/http/ngx_http_upstream.c +++ b/src/http/ngx_http_upstream.c @@ -6273,6 +6273,19 @@ ngx_http_upstream_server(ngx_conf_t *cf, resolve = 1; continue; } + + if (ngx_strncmp(value[i].data, "service=", 8) == 0) { + + us->service.len = value[i].len - 8; + us->service.data = &value[i].data[8]; + + if (us->service.len == 0) { + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, "service is empty"); + return NGX_CONF_ERROR; + } + + continue; + } #endif goto invalid; @@ -6288,6 +6301,15 @@ ngx_http_upstream_server(ngx_conf_t *cf, /* resolve at run time */ u.no_resolve = 1; } + + if (us->service.len && !resolve) { + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, + "service upstream \"%V\" requires " + "\"resolve\" parameter", + &u.url); + return NGX_CONF_ERROR; + } + #endif if (ngx_parse_url(cf->pool, &u) != NGX_OK) { @@ -6303,6 +6325,22 @@ ngx_http_upstream_server(ngx_conf_t *cf, #if (NGX_HTTP_UPSTREAM_ZONE) + if (us->service.len && !u.no_port) { + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, + "service upstream \"%V\" may not have port", + &us->name); + + return NGX_CONF_ERROR; + } + + if (us->service.len && u.naddrs) { + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, + "service upstream \"%V\" requires domain name", + &us->name); + + return NGX_CONF_ERROR; + } + if (resolve && u.naddrs == 0) { ngx_addr_t *addr; diff --git a/src/http/ngx_http_upstream.h b/src/http/ngx_http_upstream.h --- a/src/http/ngx_http_upstream.h +++ b/src/http/ngx_http_upstream.h @@ -106,9 +106,10 @@ typedef struct { #if (NGX_HTTP_UPSTREAM_ZONE) ngx_str_t host; + ngx_str_t service; #endif - NGX_COMPAT_BEGIN(4) + NGX_COMPAT_BEGIN(2) NGX_COMPAT_END } ngx_http_upstream_server_t; diff --git a/src/http/ngx_http_upstream_round_robin.c b/src/http/ngx_http_upstream_round_robin.c --- a/src/http/ngx_http_upstream_round_robin.c +++ b/src/http/ngx_http_upstream_round_robin.c @@ -187,6 +187,7 @@ ngx_http_upstream_init_round_robin(ngx_c } peer[n].host->name = server[i].host; + peer[n].host->service = server[i].service; peer[n].sockaddr = server[i].addrs[0].sockaddr; peer[n].socklen = server[i].addrs[0].socklen; @@ -320,6 +321,7 @@ ngx_http_upstream_init_round_robin(ngx_c } peer[n].host->name = server[i].host; + peer[n].host->service = server[i].service; peer[n].sockaddr = server[i].addrs[0].sockaddr; peer[n].socklen = server[i].addrs[0].socklen; diff --git a/src/http/ngx_http_upstream_round_robin.h b/src/http/ngx_http_upstream_round_robin.h --- a/src/http/ngx_http_upstream_round_robin.h +++ b/src/http/ngx_http_upstream_round_robin.h @@ -24,6 +24,7 @@ typedef struct { ngx_event_t event; /* must be first */ ngx_uint_t worker; ngx_str_t name; + ngx_str_t service; ngx_http_upstream_rr_peers_t *peers; ngx_http_upstream_rr_peer_t *peer; } ngx_http_upstream_host_t; @@ -156,7 +157,7 @@ ngx_http_upstream_rr_peer_free_locked(ng ngx_slab_free_locked(peers->shpool, peer->sockaddr); ngx_slab_free_locked(peers->shpool, peer->name.data); - if (peer->server.data && (peer->host == NULL || peer->host->peer == peer)) { + if (peer->server.data) { ngx_slab_free_locked(peers->shpool, peer->server.data); } diff --git a/src/stream/ngx_stream_upstream.c b/src/stream/ngx_stream_upstream.c --- a/src/stream/ngx_stream_upstream.c +++ b/src/stream/ngx_stream_upstream.c @@ -527,6 +527,19 @@ ngx_stream_upstream_server(ngx_conf_t *c resolve = 1; continue; } + + if (ngx_strncmp(value[i].data, "service=", 8) == 0) { + + us->service.len = value[i].len - 8; + us->service.data = &value[i].data[8]; + + if (us->service.len == 0) { + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, "service is empty"); + return NGX_CONF_ERROR; + } + + continue; + } #endif goto invalid; @@ -541,6 +554,15 @@ ngx_stream_upstream_server(ngx_conf_t *c /* resolve at run time */ u.no_resolve = 1; } + + if (us->service.len && !resolve) { + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, + "service upstream \"%V\" requires " + "\"resolve\" parameter", + &u.url); + return NGX_CONF_ERROR; + } + #endif if (ngx_parse_url(cf->pool, &u) != NGX_OK) { @@ -552,7 +574,12 @@ ngx_stream_upstream_server(ngx_conf_t *c return NGX_CONF_ERROR; } - if (u.no_port) { + if (u.no_port +#if (NGX_STREAM_UPSTREAM_ZONE) + && us->service.len == 0 +#endif + ) + { ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, "no port in upstream \"%V\"", &u.url); return NGX_CONF_ERROR; @@ -562,6 +589,22 @@ ngx_stream_upstream_server(ngx_conf_t *c #if (NGX_STREAM_UPSTREAM_ZONE) + if (us->service.len && !u.no_port) { + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, + "service upstream \"%V\" may not have port", + &us->name); + + return NGX_CONF_ERROR; + } + + if (us->service.len && u.naddrs) { + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, + "service upstream \"%V\" requires domain name", + &us->name); + + return NGX_CONF_ERROR; + } + if (resolve && u.naddrs == 0) { ngx_addr_t *addr; diff --git a/src/stream/ngx_stream_upstream.h b/src/stream/ngx_stream_upstream.h --- a/src/stream/ngx_stream_upstream.h +++ b/src/stream/ngx_stream_upstream.h @@ -65,10 +65,8 @@ typedef struct { #if (NGX_STREAM_UPSTREAM_ZONE) ngx_str_t host; + ngx_str_t service; #endif - - NGX_COMPAT_BEGIN(2) - NGX_COMPAT_END } ngx_stream_upstream_server_t; diff --git a/src/stream/ngx_stream_upstream_round_robin.c b/src/stream/ngx_stream_upstream_round_robin.c --- a/src/stream/ngx_stream_upstream_round_robin.c +++ b/src/stream/ngx_stream_upstream_round_robin.c @@ -194,6 +194,7 @@ ngx_stream_upstream_init_round_robin(ngx } peer[n].host->name = server[i].host; + peer[n].host->service = server[i].service; peer[n].sockaddr = server[i].addrs[0].sockaddr; peer[n].socklen = server[i].addrs[0].socklen; @@ -326,6 +327,7 @@ ngx_stream_upstream_init_round_robin(ngx } peer[n].host->name = server[i].host; + peer[n].host->service = server[i].service; peer[n].sockaddr = server[i].addrs[0].sockaddr; peer[n].socklen = server[i].addrs[0].socklen; diff --git a/src/stream/ngx_stream_upstream_round_robin.h b/src/stream/ngx_stream_upstream_round_robin.h --- a/src/stream/ngx_stream_upstream_round_robin.h +++ b/src/stream/ngx_stream_upstream_round_robin.h @@ -24,6 +24,7 @@ typedef struct { ngx_event_t event; /* must be first */ ngx_uint_t worker; ngx_str_t name; + ngx_str_t service; ngx_stream_upstream_rr_peers_t *peers; ngx_stream_upstream_rr_peer_t *peer; } ngx_stream_upstream_host_t; @@ -154,7 +155,7 @@ ngx_stream_upstream_rr_peer_free_locked( ngx_slab_free_locked(peers->shpool, peer->sockaddr); ngx_slab_free_locked(peers->shpool, peer->name.data); - if (peer->server.data && (peer->host == NULL || peer->host->peer == peer)) { + if (peer->server.data) { ngx_slab_free_locked(peers->shpool, peer->server.data); } diff --git a/src/stream/ngx_stream_upstream_zone_module.c b/src/stream/ngx_stream_upstream_zone_module.c --- a/src/stream/ngx_stream_upstream_zone_module.c +++ b/src/stream/ngx_stream_upstream_zone_module.c @@ -358,6 +358,18 @@ ngx_stream_upstream_zone_copy_peer(ngx_s dst->host->name.len = src->host->name.len; ngx_memcpy(dst->host->name.data, src->host->name.data, src->host->name.len); + + if (src->host->service.len) { + dst->host->service.data = ngx_slab_alloc_locked(pool, + src->host->service.len); + if (dst->host->service.data == NULL) { + goto failed; + } + + dst->host->service.len = src->host->service.len; + ngx_memcpy(dst->host->service.data, src->host->service.data, + src->host->service.len); + } } } @@ -366,6 +378,10 @@ ngx_stream_upstream_zone_copy_peer(ngx_s failed: if (dst->host) { + if (dst->host->name.data) { + ngx_slab_free_locked(pool, dst->host->name.data); + } + ngx_slab_free_locked(pool, dst->host); } @@ -531,6 +547,7 @@ ngx_stream_upstream_zone_resolve_timer(n ctx->handler = ngx_stream_upstream_zone_resolve_handler; ctx->data = host; ctx->timeout = uscf->resolver_timeout; + ctx->service = host->service; ctx->cancelable = 1; if (ngx_resolve_name(ctx) == NGX_OK) { @@ -543,15 +560,28 @@ retry: } +#define ngx_stream_upstream_zone_addr_marked(addr) \ + ((uintptr_t) (addr)->sockaddr & 1) + +#define ngx_stream_upstream_zone_mark_addr(addr) \ + (addr)->sockaddr = (struct sockaddr *) ((uintptr_t) (addr)->sockaddr | 1) + +#define ngx_stream_upstream_zone_unmark_addr(addr) \ + (addr)->sockaddr = \ + (struct sockaddr *) ((uintptr_t) (addr)->sockaddr & ~((uintptr_t) 1)) + static void ngx_stream_upstream_zone_resolve_handler(ngx_resolver_ctx_t *ctx) { time_t now; + u_short min_priority; in_port_t port; + ngx_str_t *server; ngx_msec_t timer; - ngx_uint_t i, j; + ngx_uint_t i, j, backup, addr_backup; ngx_event_t *event; ngx_resolver_addr_t *addr; + ngx_resolver_srv_name_t *srv; ngx_stream_upstream_host_t *host; ngx_stream_upstream_rr_peer_t *peer, *template, **peerp; ngx_stream_upstream_rr_peers_t *peers; @@ -571,6 +601,11 @@ ngx_stream_upstream_zone_resolve_handler ngx_stream_upstream_rr_peers_unlock(peers); ngx_shmtx_lock(&peers->shpool->mutex); + + if (host->service.len) { + ngx_slab_free_locked(peers->shpool, host->service.data); + } + ngx_slab_free_locked(peers->shpool, host->name.data); ngx_slab_free_locked(peers->shpool, host); ngx_shmtx_unlock(&peers->shpool->mutex); @@ -582,11 +617,32 @@ ngx_stream_upstream_zone_resolve_handler now = ngx_time(); + for (i = 0; i < ctx->nsrvs; i++) { + srv = &ctx->srvs[i]; + + if (srv->state) { + ngx_log_error(NGX_LOG_ERR, event->log, 0, + "%V could not be resolved (%i: %s) " + "while resolving service %V of %V", + &srv->name, srv->state, + ngx_resolver_strerror(srv->state), &ctx->service, + &ctx->name); + } + } + if (ctx->state) { - ngx_log_error(NGX_LOG_ERR, event->log, 0, - "%V could not be resolved (%i: %s)", - &ctx->name, ctx->state, - ngx_resolver_strerror(ctx->state)); + if (ctx->service.len) { + ngx_log_error(NGX_LOG_ERR, event->log, 0, + "service %V of %V could not be resolved (%i: %s)", + &ctx->service, &ctx->name, ctx->state, + ngx_resolver_strerror(ctx->state)); + + } else { + ngx_log_error(NGX_LOG_ERR, event->log, 0, + "%V could not be resolved (%i: %s)", + &ctx->name, ctx->state, + ngx_resolver_strerror(ctx->state)); + } if (ctx->state != NGX_RESOLVE_NXDOMAIN) { ngx_stream_upstream_rr_peers_unlock(peers); @@ -602,6 +658,13 @@ ngx_stream_upstream_zone_resolve_handler ctx->naddrs = 0; } + backup = 0; + min_priority = 65535; + + for (i = 0; i < ctx->naddrs; i++) { + min_priority = ngx_min(ctx->addrs[i].priority, min_priority); + } + #if (NGX_DEBUG) { u_char text[NGX_SOCKADDR_STRLEN]; @@ -609,14 +672,20 @@ ngx_stream_upstream_zone_resolve_handler for (i = 0; i < ctx->naddrs; i++) { len = ngx_sock_ntop(ctx->addrs[i].sockaddr, ctx->addrs[i].socklen, - text, NGX_SOCKADDR_STRLEN, 0); + text, NGX_SOCKADDR_STRLEN, 1); - ngx_log_debug3(NGX_LOG_DEBUG_STREAM, event->log, 0, - "name %V was resolved to %*s", &host->name, len, text); + ngx_log_debug7(NGX_LOG_DEBUG_STREAM, event->log, 0, + "name %V was resolved to %*s " + "s:\"%V\" n:\"%V\" w:%d %s", + &host->name, len, text, &host->service, + &ctx->addrs[i].name, ctx->addrs[i].weight, + ctx->addrs[i].priority != min_priority ? "backup" : ""); } } #endif +again: + for (peerp = &peers->peer; *peerp; /* void */ ) { peer = *peerp; @@ -628,14 +697,39 @@ ngx_stream_upstream_zone_resolve_handler addr = &ctx->addrs[j]; - if (addr->name.len == 0 - && ngx_cmp_sockaddr(peer->sockaddr, peer->socklen, - addr->sockaddr, addr->socklen, 0) - == NGX_OK) + addr_backup = (addr->priority != min_priority); + if (addr_backup != backup) { + continue; + } + + if (ngx_stream_upstream_zone_addr_marked(addr)) { + continue; + } + + if (ngx_cmp_sockaddr(peer->sockaddr, peer->socklen, + addr->sockaddr, addr->socklen, + host->service.len != 0) + != NGX_OK) { - addr->name.len = 1; - goto next; + continue; } + + if (host->service.len) { + if (addr->name.len != peer->server.len + || ngx_strncmp(addr->name.data, peer->server.data, + addr->name.len)) + { + continue; + } + + if (template->weight == 1 && addr->weight != peer->weight) { + continue; + } + } + + ngx_stream_upstream_zone_mark_addr(addr); + + goto next; } *peerp = peer->next; @@ -654,33 +748,32 @@ ngx_stream_upstream_zone_resolve_handler addr = &ctx->addrs[i]; - if (addr->name.len == 1) { - addr->name.len = 0; + addr_backup = (addr->priority != min_priority); + if (addr_backup != backup) { + continue; + } + + if (ngx_stream_upstream_zone_addr_marked(addr)) { + ngx_stream_upstream_zone_unmark_addr(addr); continue; } ngx_shmtx_lock(&peers->shpool->mutex); peer = ngx_stream_upstream_zone_copy_peer(peers, NULL); ngx_shmtx_unlock(&peers->shpool->mutex); + if (peer == NULL) { ngx_log_error(NGX_LOG_ERR, event->log, 0, "cannot add new server to upstream \"%V\", " "memory exhausted", peers->name); - break; + goto done; } ngx_memcpy(peer->sockaddr, addr->sockaddr, addr->socklen); - port = ((struct sockaddr_in *) template->sockaddr)->sin_port; - - switch (peer->sockaddr->sa_family) { -#if (NGX_HAVE_INET6) - case AF_INET6: - ((struct sockaddr_in6 *) peer->sockaddr)->sin6_port = port; - break; -#endif - default: /* AF_INET */ - ((struct sockaddr_in *) peer->sockaddr)->sin_port = port; + if (host->service.len == 0) { + port = ngx_inet_get_port(template->sockaddr); + ngx_inet_set_port(peer->sockaddr, port); } peer->socklen = addr->socklen; @@ -689,9 +782,30 @@ ngx_stream_upstream_zone_resolve_handler peer->name.data, NGX_SOCKADDR_STRLEN, 1); peer->host = template->host; - peer->server = template->server; + + server = host->service.len ? &addr->name : &template->server; + + peer->server.data = ngx_slab_alloc(peers->shpool, server->len); + if (peer->server.data == NULL) { + ngx_stream_upstream_rr_peer_free(peers, peer); - peer->weight = template->weight; + ngx_log_error(NGX_LOG_ERR, event->log, 0, + "cannot add new server to upstream \"%V\", " + "memory exhausted", peers->name); + goto done; + } + + peer->server.len = server->len; + ngx_memcpy(peer->server.data, server->data, server->len); + + if (host->service.len == 0) { + peer->weight = template->weight; + + } else { + peer->weight = (template->weight != 1 ? template->weight + : addr->weight); + } + peer->effective_weight = peer->weight; peer->max_conns = template->max_conns; peer->max_fails = template->max_fails; @@ -711,8 +825,25 @@ ngx_stream_upstream_zone_resolve_handler ngx_stream_upstream_zone_set_single(uscf); } + if (host->service.len && peers->next) { + ngx_stream_upstream_rr_peers_unlock(peers); + + peers = peers->next; + backup = 1; + + ngx_stream_upstream_rr_peers_wlock(peers); + + goto again; + } + +done: + ngx_stream_upstream_rr_peers_unlock(peers); + while (++i < ctx->naddrs) { + ngx_stream_upstream_zone_unmark_addr(&ctx->addrs[i]); + } + timer = (ngx_msec_t) 1000 * (ctx->valid > now ? ctx->valid - now + 1 : 1); timer = ngx_min(timer, uscf->resolver_timeout); From a.bavshin at nginx.com Wed Feb 1 01:36:59 2023 From: a.bavshin at nginx.com (=?iso-8859-1?q?Aleksei_Bavshin?=) Date: Tue, 31 Jan 2023 17:36:59 -0800 Subject: [PATCH 4 of 6] Upstream: per-upstream resolver In-Reply-To: References: Message-ID: # HG changeset patch # User Vladimir Homutov # Date 1571405595 -10800 # Fri Oct 18 16:33:15 2019 +0300 # Node ID cfae397f1ea87a35c41febab6162fe5142aa767b # Parent 95ea48662e8a8b7ee73b39c5791c06b5e17e3102 Upstream: per-upstream resolver. The "resolver" and "resolver_timeout" directives can now be specified directly in the "upstream" block. diff --git a/src/http/ngx_http_upstream.c b/src/http/ngx_http_upstream.c --- a/src/http/ngx_http_upstream.c +++ b/src/http/ngx_http_upstream.c @@ -169,6 +169,10 @@ static ngx_int_t ngx_http_upstream_cooki static char *ngx_http_upstream(ngx_conf_t *cf, ngx_command_t *cmd, void *dummy); static char *ngx_http_upstream_server(ngx_conf_t *cf, ngx_command_t *cmd, void *conf); +#if (NGX_HTTP_UPSTREAM_ZONE) +static char *ngx_http_upstream_resolver(ngx_conf_t *cf, ngx_command_t *cmd, + void *conf); +#endif static ngx_int_t ngx_http_upstream_set_local(ngx_http_request_t *r, ngx_http_upstream_t *u, ngx_http_upstream_local_t *local); @@ -339,6 +343,24 @@ static ngx_command_t ngx_http_upstream_ 0, NULL }, +#if (NGX_HTTP_UPSTREAM_ZONE) + + { ngx_string("resolver"), + NGX_HTTP_UPS_CONF|NGX_CONF_1MORE, + ngx_http_upstream_resolver, + NGX_HTTP_SRV_CONF_OFFSET, + 0, + NULL }, + + { ngx_string("resolver_timeout"), + NGX_HTTP_UPS_CONF|NGX_CONF_TAKE1, + ngx_conf_set_msec_slot, + NGX_HTTP_SRV_CONF_OFFSET, + offsetof(ngx_http_upstream_srv_conf_t, resolver_timeout), + NULL }, + +#endif + ngx_null_command }; @@ -6401,6 +6423,32 @@ not_supported: } +#if (NGX_HTTP_UPSTREAM_ZONE) + +static char * +ngx_http_upstream_resolver(ngx_conf_t *cf, ngx_command_t *cmd, void *conf) +{ + ngx_http_upstream_srv_conf_t *uscf = conf; + + ngx_str_t *value; + + if (uscf->resolver) { + return "is duplicate"; + } + + value = cf->args->elts; + + uscf->resolver = ngx_resolver_create(cf, &value[1], cf->args->nelts - 1); + if (uscf->resolver == NULL) { + return NGX_CONF_ERROR; + } + + return NGX_CONF_OK; +} + +#endif + + ngx_http_upstream_srv_conf_t * ngx_http_upstream_add(ngx_conf_t *cf, ngx_url_t *u, ngx_uint_t flags) { @@ -6482,6 +6530,9 @@ ngx_http_upstream_add(ngx_conf_t *cf, ng uscf->line = cf->conf_file->line; uscf->port = u->port; uscf->no_port = u->no_port; +#if (NGX_HTTP_UPSTREAM_ZONE) + uscf->resolver_timeout = NGX_CONF_UNSET_MSEC; +#endif if (u->naddrs == 1 && (u->port || u->family == AF_UNIX)) { uscf->servers = ngx_array_create(cf->pool, 1, diff --git a/src/http/ngx_http_upstream_round_robin.c b/src/http/ngx_http_upstream_round_robin.c --- a/src/http/ngx_http_upstream_round_robin.c +++ b/src/http/ngx_http_upstream_round_robin.c @@ -104,15 +104,15 @@ ngx_http_upstream_init_round_robin(ngx_c clcf = ngx_http_conf_get_module_loc_conf(cf, ngx_http_core_module); - us->resolver = clcf->resolver; - us->resolver_timeout = clcf->resolver_timeout; + if (us->resolver == NULL) { + us->resolver = clcf->resolver; + } /* - * Without "resolver_timeout" in http{}, the value is unset. - * Even if we set it in ngx_http_core_merge_loc_conf(), it's - * still dependent on the module order and unreliable. + * Without "resolver_timeout" in http{} the merged value is unset. */ - ngx_conf_init_msec_value(us->resolver_timeout, 30000); + ngx_conf_merge_msec_value(us->resolver_timeout, + clcf->resolver_timeout, 30000); if (resolve && (us->resolver == NULL diff --git a/src/stream/ngx_stream_upstream.c b/src/stream/ngx_stream_upstream.c --- a/src/stream/ngx_stream_upstream.c +++ b/src/stream/ngx_stream_upstream.c @@ -22,6 +22,11 @@ static char *ngx_stream_upstream(ngx_con void *dummy); static char *ngx_stream_upstream_server(ngx_conf_t *cf, ngx_command_t *cmd, void *conf); +#if (NGX_STREAM_UPSTREAM_ZONE) +static char *ngx_stream_upstream_resolver(ngx_conf_t *cf, ngx_command_t *cmd, + void *conf); +#endif + static void *ngx_stream_upstream_create_main_conf(ngx_conf_t *cf); static char *ngx_stream_upstream_init_main_conf(ngx_conf_t *cf, void *conf); @@ -42,6 +47,24 @@ static ngx_command_t ngx_stream_upstrea 0, NULL }, +#if (NGX_STREAM_UPSTREAM_ZONE) + + { ngx_string("resolver"), + NGX_STREAM_UPS_CONF|NGX_CONF_1MORE, + ngx_stream_upstream_resolver, + NGX_STREAM_SRV_CONF_OFFSET, + 0, + NULL }, + + { ngx_string("resolver_timeout"), + NGX_STREAM_UPS_CONF|NGX_CONF_TAKE1, + ngx_conf_set_msec_slot, + NGX_STREAM_SRV_CONF_OFFSET, + offsetof(ngx_stream_upstream_srv_conf_t, resolver_timeout), + NULL }, + +#endif + ngx_null_command }; @@ -665,6 +688,32 @@ not_supported: } +#if (NGX_STREAM_UPSTREAM_ZONE) + +static char * +ngx_stream_upstream_resolver(ngx_conf_t *cf, ngx_command_t *cmd, void *conf) +{ + ngx_stream_upstream_srv_conf_t *uscf = conf; + + ngx_str_t *value; + + if (uscf->resolver) { + return "is duplicate"; + } + + value = cf->args->elts; + + uscf->resolver = ngx_resolver_create(cf, &value[1], cf->args->nelts - 1); + if (uscf->resolver == NULL) { + return NGX_CONF_ERROR; + } + + return NGX_CONF_OK; +} + +#endif + + ngx_stream_upstream_srv_conf_t * ngx_stream_upstream_add(ngx_conf_t *cf, ngx_url_t *u, ngx_uint_t flags) { @@ -743,6 +792,9 @@ ngx_stream_upstream_add(ngx_conf_t *cf, uscf->line = cf->conf_file->line; uscf->port = u->port; uscf->no_port = u->no_port; +#if (NGX_STREAM_UPSTREAM_ZONE) + uscf->resolver_timeout = NGX_CONF_UNSET_MSEC; +#endif if (u->naddrs == 1 && (u->port || u->family == AF_UNIX)) { uscf->servers = ngx_array_create(cf->pool, 1, diff --git a/src/stream/ngx_stream_upstream_round_robin.c b/src/stream/ngx_stream_upstream_round_robin.c --- a/src/stream/ngx_stream_upstream_round_robin.c +++ b/src/stream/ngx_stream_upstream_round_robin.c @@ -111,15 +111,15 @@ ngx_stream_upstream_init_round_robin(ngx cscf = ngx_stream_conf_get_module_srv_conf(cf, ngx_stream_core_module); - us->resolver = cscf->resolver; - us->resolver_timeout = cscf->resolver_timeout; + if (us->resolver == NULL) { + us->resolver = cscf->resolver; + } /* - * Without "resolver_timeout" in stream{}, the value is unset. - * Even if we set it in ngx_stream_core_merge_srv_conf(), it's - * still dependent on the module order and unreliable. + * Without "resolver_timeout" in stream{} the merged value is unset. */ - ngx_conf_init_msec_value(us->resolver_timeout, 30000); + ngx_conf_merge_msec_value(us->resolver_timeout, + cscf->resolver_timeout, 30000); if (resolve && (us->resolver == NULL From a.bavshin at nginx.com Wed Feb 1 01:37:00 2023 From: a.bavshin at nginx.com (=?iso-8859-1?q?Aleksei_Bavshin?=) Date: Tue, 31 Jan 2023 17:37:00 -0800 Subject: [PATCH 5 of 6] Upstream: allow any worker to resolve upstream servers In-Reply-To: References: Message-ID: # HG changeset patch # User Aleksei Bavshin # Date 1670883784 28800 # Mon Dec 12 14:23:04 2022 -0800 # Node ID f8eb6b94d8f46008eb5f2f1dbc747750d5755506 # Parent cfae397f1ea87a35c41febab6162fe5142aa767b Upstream: allow any worker to resolve upstream servers. This change addresses one of the limitations of the current re-resolve code, dependency on the worker 0. Now, each worker is able to pick a resolve task from a shared priority queue. The single worker implementation relies on the fact that each peer is assigned to a specific worker and no other process may access its data. Thus, it's safe to keep `peer->host.event` in the shared zone and modify as necessary. That assumption becomes invalid once we allow any free worker to update the peer. Now, all the workers have to know when the previous resolution result expires and maintain their own timers. A single shared event structure is no longer sufficient. The obvious solution is to make timer events local to a worker by moving them up to the nearest object in a local memory, upstream. From the upstream timer event handler we can walk the list of the peers and pick these that are expired and not already owned by another process. To reduce the time spent under a lock we can keep a priority queue of pending tasks, sorted by expiration time. Each worker is able to get an expired server from the head of the queue, perform the name resolution and put the peer back with a new expiration time. Per-upstream or per-zone rbtree was considered as a store for pending tasks, but there won't be any performance benefit until a certain large number of servers in the upstream. Per-zone queues also require more intricate locking. The benefits of the change are obvious: we're no longer tied to a lifetime of the first worker process and the distribution of the tasks is more even. There are certain disadvantages though: - SRV record may resolve into multiple A/AAAA records with different TTL kept in a worker-local cache of a resolver. The next query in the same worker will reuse all the cache entries that are still valid. With the task distribution implemented, any worker may schedule the next update of a peer and thus we lose the benefit of a local cache. - The change introduces an additional short lock on the list of peers and allows to acquire existing long locks from different processes. For example, it's possible that different workers will resolve large SRV records from the same upstream and attempt to update the list of peers at the same time. diff --git a/src/http/modules/ngx_http_upstream_zone_module.c b/src/http/modules/ngx_http_upstream_zone_module.c --- a/src/http/modules/ngx_http_upstream_zone_module.c +++ b/src/http/modules/ngx_http_upstream_zone_module.c @@ -10,6 +10,9 @@ #include +#define NGX_UPSTREAM_RESOLVE_NO_WORKER (ngx_uint_t) -1 + + static char *ngx_http_upstream_zone(ngx_conf_t *cf, ngx_command_t *cmd, void *conf); static ngx_int_t ngx_http_upstream_init_zone(ngx_shm_zone_t *shm_zone, @@ -40,6 +43,13 @@ static ngx_command_t ngx_http_upstream_ static ngx_int_t ngx_http_upstream_zone_init_worker(ngx_cycle_t *cycle); static void ngx_http_upstream_zone_resolve_timer(ngx_event_t *event); static void ngx_http_upstream_zone_resolve_handler(ngx_resolver_ctx_t *ctx); +static void ngx_http_upstream_zone_resolve_queue_insert(ngx_queue_t *queue, + ngx_http_upstream_host_t *host); +static void ngx_http_upstream_zone_start_resolve( + ngx_http_upstream_srv_conf_t *uscf, ngx_http_upstream_host_t *host); +static void ngx_http_upstream_zone_schedule_resolve( + ngx_http_upstream_srv_conf_t *uscf, ngx_http_upstream_host_t *host, + ngx_msec_t timer); static ngx_http_module_t ngx_http_upstream_zone_module_ctx = { @@ -231,6 +241,8 @@ ngx_http_upstream_zone_copy_peers(ngx_sl peers->shpool = shpool; peers->config = config; + ngx_queue_init(&peers->resolve_queue); + for (peerp = &peers->peer; *peerp; peerp = &peer->next) { /* pool is unlocked */ peer = ngx_http_upstream_zone_copy_peer(peers, *peerp); @@ -248,6 +260,9 @@ ngx_http_upstream_zone_copy_peers(ngx_sl return NULL; } + ngx_http_upstream_rr_peer_ref(peers, peer); + ngx_queue_insert_tail(&peers->resolve_queue, &peer->host->queue); + *peerp = peer; peer->id = (*peers->config)++; } @@ -268,6 +283,8 @@ ngx_http_upstream_zone_copy_peers(ngx_sl backup->shpool = shpool; backup->config = config; + ngx_queue_init(&backup->resolve_queue); + for (peerp = &backup->peer; *peerp; peerp = &peer->next) { /* pool is unlocked */ peer = ngx_http_upstream_zone_copy_peer(backup, *peerp); @@ -285,6 +302,9 @@ ngx_http_upstream_zone_copy_peers(ngx_sl return NULL; } + ngx_http_upstream_rr_peer_ref(backup, peer); + ngx_queue_insert_tail(&backup->resolve_queue, &peer->host->queue); + *peerp = peer; peer->id = (*backup->config)++; } @@ -357,6 +377,8 @@ ngx_http_upstream_zone_copy_peer(ngx_htt dst->host->peers = peers; dst->host->peer = dst; + dst->host->expires = ngx_current_msec; + dst->host->worker = NGX_UPSTREAM_RESOLVE_NO_WORKER; dst->host->name.len = src->host->name.len; ngx_memcpy(dst->host->name.data, src->host->name.data, @@ -438,13 +460,124 @@ ngx_http_upstream_zone_remove_peer_locke } +static void +ngx_http_upstream_zone_resolve_queue_insert(ngx_queue_t *queue, + ngx_http_upstream_host_t *host) +{ + ngx_queue_t *q; + ngx_http_upstream_host_t *item; + + q = ngx_queue_last(queue); + + while (q != ngx_queue_sentinel(queue)) { + item = ngx_queue_data(q, ngx_http_upstream_host_t, queue); + + if ((ngx_msec_int_t) (item->expires - host->expires) <= 0) { + break; + } + + q = ngx_queue_prev(q); + } + + ngx_queue_insert_after(q, &host->queue); +} + + +static void +ngx_http_upstream_zone_schedule_resolve(ngx_http_upstream_srv_conf_t *uscf, + ngx_http_upstream_host_t *host, + ngx_msec_t timer) +{ + ngx_msec_t now; + ngx_event_t *event; + ngx_http_upstream_host_t *head; + ngx_http_upstream_rr_peers_t *peers; + + now = ngx_current_msec; + event = &uscf->event; + peers = host->peers; + + ngx_http_upstream_rr_peers_wlock(peers); + + host->expires = now + timer; + host->worker = NGX_UPSTREAM_RESOLVE_NO_WORKER; + ngx_http_upstream_zone_resolve_queue_insert(&peers->resolve_queue, host); + + head = ngx_queue_data(ngx_queue_head(&peers->resolve_queue), + ngx_http_upstream_host_t, queue); + if ((ngx_msec_int_t) (head->expires - host->expires) < 0) { + timer = ngx_max((ngx_msec_int_t) (head->expires - now), 1); + } + + ngx_http_upstream_rr_peers_unlock(peers); + + if (!event->timer_set + || (ngx_msec_int_t) (now + timer - event->timer.key) < 0) + { + ngx_add_timer(event, timer); + } +} + + +static void +ngx_http_upstream_zone_resolve_timer(ngx_event_t *event) +{ + ngx_msec_t now, timer; + ngx_msec_int_t expires; + ngx_http_upstream_host_t *host; + ngx_http_upstream_srv_conf_t *uscf; + ngx_http_upstream_rr_peers_t *peers; + + uscf = event->data; + peers = uscf->peer.data; + now = ngx_current_msec; + + timer = (ngx_msec_t) 1000 * + (uscf->resolver->valid ? uscf->resolver->valid : 10); + + do { + for ( ;; ) { + ngx_http_upstream_rr_peers_wlock(peers); + + if (ngx_queue_empty(&peers->resolve_queue)) { + ngx_http_upstream_rr_peers_unlock(peers); + break; + } + + host = ngx_queue_data(ngx_queue_head(&peers->resolve_queue), + ngx_http_upstream_host_t, queue); + expires = host->expires - now; + + if (expires > 0) { + ngx_http_upstream_rr_peers_unlock(peers); + timer = ngx_min(timer, (ngx_msec_t) expires); + break; + } + + ngx_queue_remove(&host->queue); + host->worker = ngx_worker; + + ngx_http_upstream_rr_peers_unlock(peers); + ngx_http_upstream_zone_start_resolve(uscf, host); + } + + peers = peers->next; + + } while (peers); + + if (!event->timer_set + || ((ngx_msec_int_t) (now + timer - event->timer.key)) < 0) + { + ngx_add_timer(event, timer); + } +} + + static ngx_int_t ngx_http_upstream_zone_init_worker(ngx_cycle_t *cycle) { ngx_uint_t i; ngx_event_t *event; - ngx_http_upstream_rr_peer_t *peer; - ngx_http_upstream_rr_peers_t *peers; ngx_http_upstream_srv_conf_t *uscf, **uscfp; ngx_http_upstream_main_conf_t *umcf; @@ -470,34 +603,13 @@ ngx_http_upstream_zone_init_worker(ngx_c continue; } - peers = uscf->peer.data; - - do { - ngx_http_upstream_rr_peers_wlock(peers); - - for (peer = peers->resolve; peer; peer = peer->next) { - - if (peer->host->worker != ngx_worker) { - continue; - } - - event = &peer->host->event; - ngx_memzero(event, sizeof(ngx_event_t)); + event = &uscf->event; + event->data = uscf; + event->handler = ngx_http_upstream_zone_resolve_timer; + event->log = cycle->log; + event->cancelable = 1; - event->data = uscf; - event->handler = ngx_http_upstream_zone_resolve_timer; - event->log = cycle->log; - event->cancelable = 1; - - ngx_http_upstream_rr_peer_ref(peers, peer); - ngx_add_timer(event, 1); - } - - ngx_http_upstream_rr_peers_unlock(peers); - - peers = peers->next; - - } while (peers); + ngx_add_timer(event, 1); } return NGX_OK; @@ -505,16 +617,13 @@ ngx_http_upstream_zone_init_worker(ngx_c static void -ngx_http_upstream_zone_resolve_timer(ngx_event_t *event) +ngx_http_upstream_zone_start_resolve(ngx_http_upstream_srv_conf_t *uscf, + ngx_http_upstream_host_t *host) { ngx_resolver_ctx_t *ctx; - ngx_http_upstream_host_t *host; ngx_http_upstream_rr_peer_t *template; ngx_http_upstream_rr_peers_t *peers; - ngx_http_upstream_srv_conf_t *uscf; - host = (ngx_http_upstream_host_t *) event; - uscf = event->data; peers = host->peers; template = host->peer; @@ -540,11 +649,13 @@ ngx_http_upstream_zone_resolve_timer(ngx } if (ctx == NGX_NO_RESOLVER) { - ngx_log_error(NGX_LOG_ERR, event->log, 0, + ngx_log_error(NGX_LOG_ERR, uscf->event.log, 0, "no resolver defined to resolve %V", &host->name); return; } + host->upstream = uscf; + ctx->name = host->name; ctx->handler = ngx_http_upstream_zone_resolve_handler; ctx->data = host; @@ -558,7 +669,8 @@ ngx_http_upstream_zone_resolve_timer(ngx retry: - ngx_add_timer(event, ngx_max(uscf->resolver_timeout, 1000)); + ngx_http_upstream_zone_schedule_resolve(uscf, host, + ngx_max(uscf->resolver_timeout, 1000)); } @@ -590,8 +702,8 @@ ngx_http_upstream_zone_resolve_handler(n ngx_http_upstream_srv_conf_t *uscf; host = ctx->data; - event = &host->event; - uscf = event->data; + uscf = host->upstream; + event = &uscf->event; peers = host->peers; template = host->peer; @@ -651,7 +763,8 @@ ngx_http_upstream_zone_resolve_handler(n ngx_resolve_name_done(ctx); - ngx_add_timer(event, ngx_max(uscf->resolver_timeout, 1000)); + ngx_http_upstream_zone_schedule_resolve(uscf, host, + ngx_max(uscf->resolver_timeout, 1000)); return; } @@ -851,5 +964,5 @@ done: ngx_resolve_name_done(ctx); - ngx_add_timer(event, timer); + ngx_http_upstream_zone_schedule_resolve(uscf, host, timer); } diff --git a/src/http/ngx_http_upstream.h b/src/http/ngx_http_upstream.h --- a/src/http/ngx_http_upstream.h +++ b/src/http/ngx_http_upstream.h @@ -138,6 +138,7 @@ struct ngx_http_upstream_srv_conf_s { ngx_uint_t no_port; /* unsigned no_port:1 */ #if (NGX_HTTP_UPSTREAM_ZONE) + ngx_event_t event; ngx_shm_zone_t *shm_zone; ngx_resolver_t *resolver; ngx_msec_t resolver_timeout; diff --git a/src/http/ngx_http_upstream_round_robin.h b/src/http/ngx_http_upstream_round_robin.h --- a/src/http/ngx_http_upstream_round_robin.h +++ b/src/http/ngx_http_upstream_round_robin.h @@ -21,12 +21,14 @@ typedef struct ngx_http_upstream_rr_peer #if (NGX_HTTP_UPSTREAM_ZONE) typedef struct { - ngx_event_t event; /* must be first */ + ngx_queue_t queue; ngx_uint_t worker; + ngx_msec_t expires; ngx_str_t name; ngx_str_t service; ngx_http_upstream_rr_peers_t *peers; ngx_http_upstream_rr_peer_t *peer; + ngx_http_upstream_srv_conf_t *upstream; /* local */ } ngx_http_upstream_host_t; #endif @@ -101,6 +103,7 @@ struct ngx_http_upstream_rr_peers_s { #if (NGX_HTTP_UPSTREAM_ZONE) ngx_uint_t *config; ngx_http_upstream_rr_peer_t *resolve; + ngx_queue_t resolve_queue; ngx_uint_t zombies; #endif }; diff --git a/src/stream/ngx_stream_upstream.h b/src/stream/ngx_stream_upstream.h --- a/src/stream/ngx_stream_upstream.h +++ b/src/stream/ngx_stream_upstream.h @@ -85,6 +85,7 @@ struct ngx_stream_upstream_srv_conf_s { ngx_uint_t no_port; /* unsigned no_port:1 */ #if (NGX_STREAM_UPSTREAM_ZONE) + ngx_event_t event; ngx_shm_zone_t *shm_zone; ngx_resolver_t *resolver; ngx_msec_t resolver_timeout; diff --git a/src/stream/ngx_stream_upstream_round_robin.h b/src/stream/ngx_stream_upstream_round_robin.h --- a/src/stream/ngx_stream_upstream_round_robin.h +++ b/src/stream/ngx_stream_upstream_round_robin.h @@ -21,12 +21,14 @@ typedef struct ngx_stream_upstream_rr_pe #if (NGX_STREAM_UPSTREAM_ZONE) typedef struct { - ngx_event_t event; /* must be first */ + ngx_queue_t queue; ngx_uint_t worker; + ngx_msec_t expires; ngx_str_t name; ngx_str_t service; ngx_stream_upstream_rr_peers_t *peers; ngx_stream_upstream_rr_peer_t *peer; + ngx_stream_upstream_srv_conf_t *upstream; /* local */ } ngx_stream_upstream_host_t; #endif @@ -99,6 +101,7 @@ struct ngx_stream_upstream_rr_peers_s { #if (NGX_STREAM_UPSTREAM_ZONE) ngx_uint_t *config; ngx_stream_upstream_rr_peer_t *resolve; + ngx_queue_t resolve_queue; ngx_uint_t zombies; #endif }; diff --git a/src/stream/ngx_stream_upstream_zone_module.c b/src/stream/ngx_stream_upstream_zone_module.c --- a/src/stream/ngx_stream_upstream_zone_module.c +++ b/src/stream/ngx_stream_upstream_zone_module.c @@ -10,6 +10,9 @@ #include +#define NGX_UPSTREAM_RESOLVE_NO_WORKER (ngx_uint_t) -1 + + static char *ngx_stream_upstream_zone(ngx_conf_t *cf, ngx_command_t *cmd, void *conf); static ngx_int_t ngx_stream_upstream_init_zone(ngx_shm_zone_t *shm_zone, @@ -40,6 +43,13 @@ static ngx_command_t ngx_stream_upstrea static ngx_int_t ngx_stream_upstream_zone_init_worker(ngx_cycle_t *cycle); static void ngx_stream_upstream_zone_resolve_timer(ngx_event_t *event); static void ngx_stream_upstream_zone_resolve_handler(ngx_resolver_ctx_t *ctx); +static void ngx_stream_upstream_zone_resolve_queue_insert(ngx_queue_t *queue, + ngx_stream_upstream_host_t *host); +static void ngx_stream_upstream_zone_start_resolve( + ngx_stream_upstream_srv_conf_t *uscf, ngx_stream_upstream_host_t *host); +static void ngx_stream_upstream_zone_schedule_resolve( + ngx_stream_upstream_srv_conf_t *uscf, ngx_stream_upstream_host_t *host, + ngx_msec_t timer); static ngx_stream_module_t ngx_stream_upstream_zone_module_ctx = { @@ -228,6 +238,8 @@ ngx_stream_upstream_zone_copy_peers(ngx_ peers->shpool = shpool; peers->config = config; + ngx_queue_init(&peers->resolve_queue); + for (peerp = &peers->peer; *peerp; peerp = &peer->next) { /* pool is unlocked */ peer = ngx_stream_upstream_zone_copy_peer(peers, *peerp); @@ -245,6 +257,9 @@ ngx_stream_upstream_zone_copy_peers(ngx_ return NULL; } + ngx_stream_upstream_rr_peer_ref(peers, peer); + ngx_queue_insert_tail(&peers->resolve_queue, &peer->host->queue); + *peerp = peer; peer->id = (*peers->config)++; } @@ -265,6 +280,8 @@ ngx_stream_upstream_zone_copy_peers(ngx_ backup->shpool = shpool; backup->config = config; + ngx_queue_init(&backup->resolve_queue); + for (peerp = &backup->peer; *peerp; peerp = &peer->next) { /* pool is unlocked */ peer = ngx_stream_upstream_zone_copy_peer(backup, *peerp); @@ -282,6 +299,9 @@ ngx_stream_upstream_zone_copy_peers(ngx_ return NULL; } + ngx_stream_upstream_rr_peer_ref(backup, peer); + ngx_queue_insert_tail(&backup->resolve_queue, &peer->host->queue); + *peerp = peer; peer->id = (*backup->config)++; } @@ -354,6 +374,8 @@ ngx_stream_upstream_zone_copy_peer(ngx_s dst->host->peers = peers; dst->host->peer = dst; + dst->host->expires = ngx_current_msec; + dst->host->worker = NGX_UPSTREAM_RESOLVE_NO_WORKER; dst->host->name.len = src->host->name.len; ngx_memcpy(dst->host->name.data, src->host->name.data, @@ -435,13 +457,124 @@ ngx_stream_upstream_zone_remove_peer_loc } +static void +ngx_stream_upstream_zone_resolve_queue_insert(ngx_queue_t *queue, + ngx_stream_upstream_host_t *host) +{ + ngx_queue_t *q; + ngx_stream_upstream_host_t *item; + + q = ngx_queue_last(queue); + + while (q != ngx_queue_sentinel(queue)) { + item = ngx_queue_data(q, ngx_stream_upstream_host_t, queue); + + if ((ngx_msec_int_t) (item->expires - host->expires) <= 0) { + break; + } + + q = ngx_queue_prev(q); + } + + ngx_queue_insert_after(q, &host->queue); +} + + +static void +ngx_stream_upstream_zone_schedule_resolve(ngx_stream_upstream_srv_conf_t *uscf, + ngx_stream_upstream_host_t *host, + ngx_msec_t timer) +{ + ngx_msec_t now; + ngx_event_t *event; + ngx_stream_upstream_host_t *head; + ngx_stream_upstream_rr_peers_t *peers; + + now = ngx_current_msec; + event = &uscf->event; + peers = host->peers; + + ngx_stream_upstream_rr_peers_wlock(peers); + + host->expires = now + timer; + host->worker = NGX_UPSTREAM_RESOLVE_NO_WORKER; + ngx_stream_upstream_zone_resolve_queue_insert(&peers->resolve_queue, host); + + head = ngx_queue_data(ngx_queue_head(&peers->resolve_queue), + ngx_stream_upstream_host_t, queue); + if ((ngx_msec_int_t) (head->expires - host->expires) < 0) { + timer = ngx_max((ngx_msec_int_t) (head->expires - now), 1); + } + + ngx_stream_upstream_rr_peers_unlock(peers); + + if (!event->timer_set + || (ngx_msec_int_t) (now + timer - event->timer.key) < 0) + { + ngx_add_timer(event, timer); + } +} + + +static void +ngx_stream_upstream_zone_resolve_timer(ngx_event_t *event) +{ + ngx_msec_t now, timer; + ngx_msec_int_t expires; + ngx_stream_upstream_host_t *host; + ngx_stream_upstream_srv_conf_t *uscf; + ngx_stream_upstream_rr_peers_t *peers; + + uscf = event->data; + peers = uscf->peer.data; + now = ngx_current_msec; + + timer = (ngx_msec_t) 1000 * + (uscf->resolver->valid ? uscf->resolver->valid : 10); + + do { + for ( ;; ) { + ngx_stream_upstream_rr_peers_wlock(peers); + + if (ngx_queue_empty(&peers->resolve_queue)) { + ngx_stream_upstream_rr_peers_unlock(peers); + break; + } + + host = ngx_queue_data(ngx_queue_head(&peers->resolve_queue), + ngx_stream_upstream_host_t, queue); + expires = host->expires - now; + + if (expires > 0) { + ngx_stream_upstream_rr_peers_unlock(peers); + timer = ngx_min(timer, (ngx_msec_t) expires); + break; + } + + ngx_queue_remove(&host->queue); + host->worker = ngx_worker; + + ngx_stream_upstream_rr_peers_unlock(peers); + ngx_stream_upstream_zone_start_resolve(uscf, host); + } + + peers = peers->next; + + } while (peers); + + if (!event->timer_set + || (ngx_msec_int_t) (now + timer - event->timer.key) < 0) + { + ngx_add_timer(event, timer); + } +} + + static ngx_int_t ngx_stream_upstream_zone_init_worker(ngx_cycle_t *cycle) { ngx_uint_t i; ngx_event_t *event; - ngx_stream_upstream_rr_peer_t *peer; - ngx_stream_upstream_rr_peers_t *peers; ngx_stream_upstream_srv_conf_t *uscf, **uscfp; ngx_stream_upstream_main_conf_t *umcf; @@ -468,34 +601,13 @@ ngx_stream_upstream_zone_init_worker(ngx continue; } - peers = uscf->peer.data; - - do { - ngx_stream_upstream_rr_peers_wlock(peers); - - for (peer = peers->resolve; peer; peer = peer->next) { - - if (peer->host->worker != ngx_worker) { - continue; - } - - event = &peer->host->event; - ngx_memzero(event, sizeof(ngx_event_t)); + event = &uscf->event; + event->data = uscf; + event->handler = ngx_stream_upstream_zone_resolve_timer; + event->log = cycle->log; + event->cancelable = 1; - event->data = uscf; - event->handler = ngx_stream_upstream_zone_resolve_timer; - event->log = cycle->log; - event->cancelable = 1; - - ngx_stream_upstream_rr_peer_ref(peers, peer); - ngx_add_timer(event, 1); - } - - ngx_stream_upstream_rr_peers_unlock(peers); - - peers = peers->next; - - } while (peers); + ngx_add_timer(event, 1); } return NGX_OK; @@ -503,16 +615,13 @@ ngx_stream_upstream_zone_init_worker(ngx static void -ngx_stream_upstream_zone_resolve_timer(ngx_event_t *event) +ngx_stream_upstream_zone_start_resolve(ngx_stream_upstream_srv_conf_t *uscf, + ngx_stream_upstream_host_t *host) { ngx_resolver_ctx_t *ctx; - ngx_stream_upstream_host_t *host; ngx_stream_upstream_rr_peer_t *template; ngx_stream_upstream_rr_peers_t *peers; - ngx_stream_upstream_srv_conf_t *uscf; - host = (ngx_stream_upstream_host_t *) event; - uscf = event->data; peers = host->peers; template = host->peer; @@ -538,11 +647,13 @@ ngx_stream_upstream_zone_resolve_timer(n } if (ctx == NGX_NO_RESOLVER) { - ngx_log_error(NGX_LOG_ERR, event->log, 0, + ngx_log_error(NGX_LOG_ERR, uscf->event.log, 0, "no resolver defined to resolve %V", &host->name); return; } + host->upstream = uscf; + ctx->name = host->name; ctx->handler = ngx_stream_upstream_zone_resolve_handler; ctx->data = host; @@ -556,7 +667,8 @@ ngx_stream_upstream_zone_resolve_timer(n retry: - ngx_add_timer(event, ngx_max(uscf->resolver_timeout, 1000)); + ngx_stream_upstream_zone_schedule_resolve(uscf, host, + ngx_max(uscf->resolver_timeout, 1000)); } @@ -588,8 +700,8 @@ ngx_stream_upstream_zone_resolve_handler ngx_stream_upstream_srv_conf_t *uscf; host = ctx->data; - event = &host->event; - uscf = event->data; + uscf = host->upstream; + event = &uscf->event; peers = host->peers; template = host->peer; @@ -649,7 +761,8 @@ ngx_stream_upstream_zone_resolve_handler ngx_resolve_name_done(ctx); - ngx_add_timer(event, ngx_max(uscf->resolver_timeout, 1000)); + ngx_stream_upstream_zone_schedule_resolve(uscf, host, + ngx_max(uscf->resolver_timeout, 1000)); return; } @@ -849,5 +962,5 @@ done: ngx_resolve_name_done(ctx); - ngx_add_timer(event, timer); + ngx_stream_upstream_zone_schedule_resolve(uscf, host, timer); } From a.bavshin at nginx.com Wed Feb 1 01:37:01 2023 From: a.bavshin at nginx.com (=?iso-8859-1?q?Aleksei_Bavshin?=) Date: Tue, 31 Jan 2023 17:37:01 -0800 Subject: [PATCH 6 of 6] Upstream: reschedule in-progress resolve tasks at worker exit In-Reply-To: References: Message-ID: <34e439843ed3a2122cba.1675215421@fedora-wsl.> # HG changeset patch # User Aleksei Bavshin # Date 1671568350 28800 # Tue Dec 20 12:32:30 2022 -0800 # Node ID 34e439843ed3a2122cba54b1f40b77ea6e874078 # Parent f8eb6b94d8f46008eb5f2f1dbc747750d5755506 Upstream: reschedule in-progress resolve tasks at worker exit. Workers may exit at a different time, depending on the active connections and other factors. If the peer was being resolved in one of the workers at the moment of termination, it won't be returned to the resolve queue and remaining workers won't be able to continue updating it. To address that we can disown all interrupted resolve tasks and put them back to the queue at worker exit. The same logic runs on the process init to recover tasks affected by a worker crash. diff --git a/src/http/modules/ngx_http_upstream_zone_module.c b/src/http/modules/ngx_http_upstream_zone_module.c --- a/src/http/modules/ngx_http_upstream_zone_module.c +++ b/src/http/modules/ngx_http_upstream_zone_module.c @@ -40,7 +40,10 @@ static ngx_command_t ngx_http_upstream_ }; +static ngx_uint_t ngx_http_upstream_zone_recover_peers( + ngx_http_upstream_srv_conf_t *uscf); static ngx_int_t ngx_http_upstream_zone_init_worker(ngx_cycle_t *cycle); +static void ngx_http_upstream_zone_exit_worker(ngx_cycle_t *cycle); static void ngx_http_upstream_zone_resolve_timer(ngx_event_t *event); static void ngx_http_upstream_zone_resolve_handler(ngx_resolver_ctx_t *ctx); static void ngx_http_upstream_zone_resolve_queue_insert(ngx_queue_t *queue, @@ -77,7 +80,7 @@ ngx_module_t ngx_http_upstream_zone_mod ngx_http_upstream_zone_init_worker, /* init process */ NULL, /* init thread */ NULL, /* exit thread */ - NULL, /* exit process */ + ngx_http_upstream_zone_exit_worker, /* exit process */ NULL, /* exit master */ NGX_MODULE_V1_PADDING }; @@ -573,10 +576,51 @@ ngx_http_upstream_zone_resolve_timer(ngx } +static ngx_uint_t +ngx_http_upstream_zone_recover_peers(ngx_http_upstream_srv_conf_t *uscf) +{ + ngx_msec_t now; + ngx_uint_t n; + ngx_http_upstream_host_t *host; + ngx_http_upstream_rr_peer_t *peer; + ngx_http_upstream_rr_peers_t *peers; + + n = 0; + now = ngx_current_msec; + peers = uscf->peer.data; + + do { + ngx_http_upstream_rr_peers_wlock(peers); + + for (peer = peers->resolve; peer; peer = peer->next) { + + host = peer->host; + + if (host->worker != ngx_worker) { + continue; + } + + host->expires = now; + host->worker = NGX_UPSTREAM_RESOLVE_NO_WORKER; + ngx_queue_insert_head(&peers->resolve_queue, &host->queue); + + n++; + } + + ngx_http_upstream_rr_peers_unlock(peers); + + peers = peers->next; + + } while (peers); + + return n; +} + + static ngx_int_t ngx_http_upstream_zone_init_worker(ngx_cycle_t *cycle) { - ngx_uint_t i; + ngx_uint_t i, n; ngx_event_t *event; ngx_http_upstream_srv_conf_t *uscf, **uscfp; ngx_http_upstream_main_conf_t *umcf; @@ -594,6 +638,7 @@ ngx_http_upstream_zone_init_worker(ngx_c } uscfp = umcf->upstreams.elts; + n = 0; for (i = 0; i < umcf->upstreams.nelts; i++) { @@ -603,6 +648,8 @@ ngx_http_upstream_zone_init_worker(ngx_c continue; } + n += ngx_http_upstream_zone_recover_peers(uscf); + event = &uscf->event; event->data = uscf; event->handler = ngx_http_upstream_zone_resolve_timer; @@ -612,11 +659,56 @@ ngx_http_upstream_zone_init_worker(ngx_c ngx_add_timer(event, 1); } + if (n) { + ngx_log_debug1(NGX_LOG_DEBUG_HTTP, cycle->log, 0, + "upstream: recovered %ui resolvable peers", n); + } + return NGX_OK; } static void +ngx_http_upstream_zone_exit_worker(ngx_cycle_t *cycle) +{ + ngx_uint_t i, n; + ngx_http_upstream_srv_conf_t *uscf, **uscfp; + ngx_http_upstream_main_conf_t *umcf; + + if (ngx_process != NGX_PROCESS_WORKER + && ngx_process != NGX_PROCESS_SINGLE) + { + return; + } + + umcf = ngx_http_cycle_get_module_main_conf(cycle, ngx_http_upstream_module); + + if (umcf == NULL) { + return; + } + + uscfp = umcf->upstreams.elts; + n = 0; + + for (i = 0; i < umcf->upstreams.nelts; i++) { + + uscf = uscfp[i]; + + if (uscf->shm_zone == NULL) { + continue; + } + + n += ngx_http_upstream_zone_recover_peers(uscf); + } + + if (n) { + ngx_log_debug1(NGX_LOG_DEBUG_HTTP, cycle->log, 0, + "upstream: released %ui resolvable peers", n); + } +} + + +static void ngx_http_upstream_zone_start_resolve(ngx_http_upstream_srv_conf_t *uscf, ngx_http_upstream_host_t *host) { diff --git a/src/stream/ngx_stream_upstream_zone_module.c b/src/stream/ngx_stream_upstream_zone_module.c --- a/src/stream/ngx_stream_upstream_zone_module.c +++ b/src/stream/ngx_stream_upstream_zone_module.c @@ -40,7 +40,10 @@ static ngx_command_t ngx_stream_upstrea }; +static ngx_uint_t ngx_stream_upstream_zone_recover_peers( + ngx_stream_upstream_srv_conf_t *uscf); static ngx_int_t ngx_stream_upstream_zone_init_worker(ngx_cycle_t *cycle); +static void ngx_stream_upstream_zone_exit_worker(ngx_cycle_t *cycle); static void ngx_stream_upstream_zone_resolve_timer(ngx_event_t *event); static void ngx_stream_upstream_zone_resolve_handler(ngx_resolver_ctx_t *ctx); static void ngx_stream_upstream_zone_resolve_queue_insert(ngx_queue_t *queue, @@ -74,7 +77,7 @@ ngx_module_t ngx_stream_upstream_zone_m ngx_stream_upstream_zone_init_worker, /* init process */ NULL, /* init thread */ NULL, /* exit thread */ - NULL, /* exit process */ + ngx_stream_upstream_zone_exit_worker, /* exit process */ NULL, /* exit master */ NGX_MODULE_V1_PADDING }; @@ -570,10 +573,51 @@ ngx_stream_upstream_zone_resolve_timer(n } +static ngx_uint_t +ngx_stream_upstream_zone_recover_peers(ngx_stream_upstream_srv_conf_t *uscf) +{ + ngx_msec_t now; + ngx_uint_t n; + ngx_stream_upstream_host_t *host; + ngx_stream_upstream_rr_peer_t *peer; + ngx_stream_upstream_rr_peers_t *peers; + + n = 0; + now = ngx_current_msec; + peers = uscf->peer.data; + + do { + ngx_stream_upstream_rr_peers_wlock(peers); + + for (peer = peers->resolve; peer; peer = peer->next) { + + host = peer->host; + + if (host->worker != ngx_worker) { + continue; + } + + host->expires = now; + host->worker = NGX_UPSTREAM_RESOLVE_NO_WORKER; + ngx_queue_insert_head(&peers->resolve_queue, &host->queue); + + n++; + } + + ngx_stream_upstream_rr_peers_unlock(peers); + + peers = peers->next; + + } while (peers); + + return n; +} + + static ngx_int_t ngx_stream_upstream_zone_init_worker(ngx_cycle_t *cycle) { - ngx_uint_t i; + ngx_uint_t i, n; ngx_event_t *event; ngx_stream_upstream_srv_conf_t *uscf, **uscfp; ngx_stream_upstream_main_conf_t *umcf; @@ -592,6 +636,7 @@ ngx_stream_upstream_zone_init_worker(ngx } uscfp = umcf->upstreams.elts; + n = 0; for (i = 0; i < umcf->upstreams.nelts; i++) { @@ -601,6 +646,8 @@ ngx_stream_upstream_zone_init_worker(ngx continue; } + n += ngx_stream_upstream_zone_recover_peers(uscf); + event = &uscf->event; event->data = uscf; event->handler = ngx_stream_upstream_zone_resolve_timer; @@ -610,11 +657,57 @@ ngx_stream_upstream_zone_init_worker(ngx ngx_add_timer(event, 1); } + if (n) { + ngx_log_debug1(NGX_LOG_DEBUG_STREAM, cycle->log, 0, + "upstream: recovered %ui resolvable peers", n); + } + return NGX_OK; } static void +ngx_stream_upstream_zone_exit_worker(ngx_cycle_t *cycle) +{ + ngx_uint_t i, n; + ngx_stream_upstream_srv_conf_t *uscf, **uscfp; + ngx_stream_upstream_main_conf_t *umcf; + + if (ngx_process != NGX_PROCESS_WORKER + && ngx_process != NGX_PROCESS_SINGLE) + { + return; + } + + umcf = ngx_stream_cycle_get_module_main_conf(cycle, + ngx_stream_upstream_module); + + if (umcf == NULL) { + return; + } + + uscfp = umcf->upstreams.elts; + n = 0; + + for (i = 0; i < umcf->upstreams.nelts; i++) { + + uscf = uscfp[i]; + + if (uscf->shm_zone == NULL) { + continue; + } + + n += ngx_stream_upstream_zone_recover_peers(uscf); + } + + if (n) { + ngx_log_debug1(NGX_LOG_DEBUG_STREAM, cycle->log, 0, + "upstream: released %ui resolvable peers", n); + } +} + + +static void ngx_stream_upstream_zone_start_resolve(ngx_stream_upstream_srv_conf_t *uscf, ngx_stream_upstream_host_t *host) { From arut at nginx.com Wed Feb 1 14:01:07 2023 From: arut at nginx.com (=?iso-8859-1?q?Roman_Arutyunyan?=) Date: Wed, 01 Feb 2023 18:01:07 +0400 Subject: [PATCH 0 of 3] Directives for enabling http2 and http3 In-Reply-To: References: Message-ID: Hi, Updates in this series: - Detect HTTP-level redirects to servers where the negotiated protocol is not available. Now 421 is returned. - Retained old http3 configuration for compatibility. - Reimplemented the http2 patch. Now http2 preface is read in ngx_http_wait_request_handler(). -- Roman Arutyunyan From arut at nginx.com Wed Feb 1 14:01:08 2023 From: arut at nginx.com (=?iso-8859-1?q?Roman_Arutyunyan?=) Date: Wed, 01 Feb 2023 18:01:08 +0400 Subject: [PATCH 1 of 3] HTTP/3: "quic" parameter of "listen" directive In-Reply-To: References: Message-ID: <2fcc1c60be1c89aad546.1675260068@arut-laptop> # HG changeset patch # User Roman Arutyunyan # Date 1675254688 -14400 # Wed Feb 01 16:31:28 2023 +0400 # Branch quic # Node ID 2fcc1c60be1c89aad5464bcc06f1189d1adc885a # Parent def8e398d7c50131f8dac844814fff729da5c86c HTTP/3: "quic" parameter of "listen" directive. Now "listen" directve has a new "quic" parameter which enables QUIC protocol for the address. Further, to enable HTTP/3, a new directive "http3" is introduced. The hq-interop protocol is enabled by "http3_hq" as before. Now application protocol is chosen by ALPN. Previously used "http3" parameter of "listen" is deprecated. diff --git a/README b/README --- a/README +++ b/README @@ -93,13 +93,13 @@ 2. Installing 3. Configuration - The HTTP "listen" directive got a new option "http3" which enables - HTTP/3 over QUIC on the specified port. + The HTTP "listen" directive got a new option "quic" which enables + QUIC as client transport protocol instead of TCP. The Stream "listen" directive got a new option "quic" which enables QUIC as client transport protocol instead of TCP or plain UDP. - Along with "http3" or "quic", it's also possible to specify "reuseport" + Along with "quic", it's also possible to specify "reuseport" option [8] to make it work properly with multiple workers. To enable address validation: @@ -133,12 +133,13 @@ 3. Configuration A number of directives were added that configure HTTP/3: + http3 + http3_hq http3_stream_buffer_size http3_max_concurrent_pushes http3_max_concurrent_streams http3_push http3_push_preload - http3_hq (requires NGX_HTTP_V3_HQ macro) In http, an additional variable is available: $http3. The value of $http3 is "h3" for HTTP/3 connections, @@ -160,13 +161,15 @@ Example configuration: server { # for better compatibility it's recommended # to use the same port for quic and https - listen 8443 http3 reuseport; + listen 8443 quic reuseport; listen 8443 ssl; ssl_certificate certs/example.com.crt; ssl_certificate_key certs/example.com.key; ssl_protocols TLSv1.3; + http3 on; + location / { # required for browsers to direct them into quic port add_header Alt-Svc 'h3=":8443"; ma=86400'; diff --git a/src/http/modules/ngx_http_ssl_module.c b/src/http/modules/ngx_http_ssl_module.c --- a/src/http/modules/ngx_http_ssl_module.c +++ b/src/http/modules/ngx_http_ssl_module.c @@ -427,7 +427,7 @@ ngx_http_ssl_alpn_select(ngx_ssl_conn_t #if (NGX_HTTP_V2 || NGX_HTTP_V3) ngx_http_connection_t *hc; #endif -#if (NGX_HTTP_V3 && NGX_HTTP_V3_HQ) +#if (NGX_HTTP_V3) ngx_http_v3_srv_conf_t *h3scf; #endif #if (NGX_HTTP_V2 || NGX_HTTP_V3 || NGX_DEBUG) @@ -455,19 +455,26 @@ ngx_http_ssl_alpn_select(ngx_ssl_conn_t } else #endif #if (NGX_HTTP_V3) - if (hc->addr_conf->http3) { + if (hc->addr_conf->quic) { -#if (NGX_HTTP_V3_HQ) h3scf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_v3_module); - if (h3scf->hq) { + if (h3scf->enable && h3scf->enable_hq) { + srv = (unsigned char *) NGX_HTTP_V3_ALPN_PROTO + NGX_HTTP_V3_HQ_ALPN_PROTO; + srvlen = sizeof(NGX_HTTP_V3_ALPN_PROTO NGX_HTTP_V3_HQ_ALPN_PROTO) + - 1; + + } else if (h3scf->enable_hq) { srv = (unsigned char *) NGX_HTTP_V3_HQ_ALPN_PROTO; srvlen = sizeof(NGX_HTTP_V3_HQ_ALPN_PROTO) - 1; - } else -#endif - { + + } else if (h3scf->enable || hc->addr_conf->http3) { srv = (unsigned char *) NGX_HTTP_V3_ALPN_PROTO; srvlen = sizeof(NGX_HTTP_V3_ALPN_PROTO) - 1; + + } else { + return SSL_TLSEXT_ERR_ALERT_FATAL; } } else @@ -1313,12 +1320,12 @@ ngx_http_ssl_init(ngx_conf_t *cf) addr = port[p].addrs.elts; for (a = 0; a < port[p].addrs.nelts; a++) { - if (!addr[a].opt.ssl && !addr[a].opt.http3) { + if (!addr[a].opt.ssl && !addr[a].opt.quic) { continue; } - if (addr[a].opt.http3) { - name = "http3"; + if (addr[a].opt.quic) { + name = "quic"; } else { name = "ssl"; @@ -1329,7 +1336,7 @@ ngx_http_ssl_init(ngx_conf_t *cf) if (sscf->certificates) { - if (addr[a].opt.http3 && !(sscf->protocols & NGX_SSL_TLSv1_3)) { + if (addr[a].opt.quic && !(sscf->protocols & NGX_SSL_TLSv1_3)) { ngx_log_error(NGX_LOG_EMERG, cf->log, 0, "\"ssl_protocols\" must enable TLSv1.3 for " "the \"listen ... %s\" directive in %s:%ui", diff --git a/src/http/ngx_http.c b/src/http/ngx_http.c --- a/src/http/ngx_http.c +++ b/src/http/ngx_http.c @@ -1242,6 +1242,7 @@ ngx_http_add_addresses(ngx_conf_t *cf, n #endif #if (NGX_HTTP_V3) ngx_uint_t http3; + ngx_uint_t quic; #endif /* @@ -1280,6 +1281,7 @@ ngx_http_add_addresses(ngx_conf_t *cf, n #endif #if (NGX_HTTP_V3) http3 = lsopt->http3 || addr[i].opt.http3; + quic = lsopt->quic || addr[i].opt.quic; #endif if (lsopt->set) { @@ -1319,6 +1321,7 @@ ngx_http_add_addresses(ngx_conf_t *cf, n #endif #if (NGX_HTTP_V3) addr[i].opt.http3 = http3; + addr[i].opt.quic = quic; #endif return NGX_OK; @@ -1823,7 +1826,7 @@ ngx_http_add_listening(ngx_conf_t *cf, n #if (NGX_HTTP_V3) - ls->quic = addr->opt.http3; + ls->quic = addr->opt.quic; if (ls->quic) { ngx_rbtree_init(&ls->rbtree, &ls->sentinel, @@ -1866,6 +1869,7 @@ ngx_http_add_addrs(ngx_conf_t *cf, ngx_h #endif #if (NGX_HTTP_V3) addrs[i].conf.http3 = addr[i].opt.http3; + addrs[i].conf.quic = addr[i].opt.quic; #endif addrs[i].conf.proxy_protocol = addr[i].opt.proxy_protocol; @@ -1934,6 +1938,7 @@ ngx_http_add_addrs6(ngx_conf_t *cf, ngx_ #endif #if (NGX_HTTP_V3) addrs6[i].conf.http3 = addr[i].opt.http3; + addrs6[i].conf.quic = addr[i].opt.quic; #endif addrs6[i].conf.proxy_protocol = addr[i].opt.proxy_protocol; diff --git a/src/http/ngx_http_core_module.c b/src/http/ngx_http_core_module.c --- a/src/http/ngx_http_core_module.c +++ b/src/http/ngx_http_core_module.c @@ -4191,6 +4191,11 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx if (ngx_strcmp(value[n].data, "http3") == 0) { #if (NGX_HTTP_V3) + ngx_conf_log_error(NGX_LOG_WARN, cf, 0, + "the \"listen ... http3\" directive " + "is deprecated, use " + "the \"http3\" directive instead"); + lsopt.quic = 1; lsopt.http3 = 1; lsopt.type = SOCK_DGRAM; continue; @@ -4202,6 +4207,19 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx #endif } + if (ngx_strcmp(value[n].data, "quic") == 0) { +#if (NGX_HTTP_V3) + lsopt.quic = 1; + lsopt.type = SOCK_DGRAM; + continue; +#else + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, + "the \"quic\" parameter requires " + "ngx_http_v3_module"); + return NGX_CONF_ERROR; +#endif + } + if (ngx_strncmp(value[n].data, "so_keepalive=", 13) == 0) { if (ngx_strcmp(&value[n].data[13], "on") == 0) { @@ -4304,8 +4322,8 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx } #if (NGX_HTTP_SSL && NGX_HTTP_V3) - if (lsopt.ssl && lsopt.http3) { - return "\"ssl\" parameter is incompatible with \"http3\""; + if (lsopt.ssl && lsopt.quic) { + return "\"ssl\" parameter is incompatible with \"quic\""; } #endif diff --git a/src/http/ngx_http_core_module.h b/src/http/ngx_http_core_module.h --- a/src/http/ngx_http_core_module.h +++ b/src/http/ngx_http_core_module.h @@ -76,6 +76,7 @@ typedef struct { unsigned ssl:1; unsigned http2:1; unsigned http3:1; + unsigned quic:1; #if (NGX_HAVE_INET6) unsigned ipv6only:1; #endif @@ -240,6 +241,7 @@ struct ngx_http_addr_conf_s { unsigned ssl:1; unsigned http2:1; unsigned http3:1; + unsigned quic:1; unsigned proxy_protocol:1; }; diff --git a/src/http/ngx_http_request.c b/src/http/ngx_http_request.c --- a/src/http/ngx_http_request.c +++ b/src/http/ngx_http_request.c @@ -325,7 +325,7 @@ ngx_http_init_connection(ngx_connection_ #endif #if (NGX_HTTP_V3) - if (hc->addr_conf->http3) { + if (hc->addr_conf->quic) { ngx_http_v3_init_stream(c); return; } diff --git a/src/http/v3/ngx_http_v3.c b/src/http/v3/ngx_http_v3.c --- a/src/http/v3/ngx_http_v3.c +++ b/src/http/v3/ngx_http_v3.c @@ -17,12 +17,9 @@ static void ngx_http_v3_cleanup_session( ngx_int_t ngx_http_v3_init_session(ngx_connection_t *c) { - ngx_pool_cleanup_t *cln; - ngx_http_connection_t *hc; - ngx_http_v3_session_t *h3c; -#if (NGX_HTTP_V3_HQ) - ngx_http_v3_srv_conf_t *h3scf; -#endif + ngx_pool_cleanup_t *cln; + ngx_http_connection_t *hc; + ngx_http_v3_session_t *h3c; hc = c->data; @@ -36,13 +33,6 @@ ngx_http_v3_init_session(ngx_connection_ h3c->max_push_id = (uint64_t) -1; h3c->goaway_push_id = (uint64_t) -1; -#if (NGX_HTTP_V3_HQ) - h3scf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_v3_module); - if (h3scf->hq) { - h3c->hq = 1; - } -#endif - ngx_queue_init(&h3c->blocked); ngx_queue_init(&h3c->pushing); diff --git a/src/http/v3/ngx_http_v3.h b/src/http/v3/ngx_http_v3.h --- a/src/http/v3/ngx_http_v3.h +++ b/src/http/v3/ngx_http_v3.h @@ -21,6 +21,7 @@ #define NGX_HTTP_V3_ALPN_PROTO "\x02h3" #define NGX_HTTP_V3_HQ_ALPN_PROTO "\x0Ahq-interop" +#define NGX_HTTP_V3_HQ_PROTO "hq-interop" #define NGX_HTTP_V3_VARLEN_INT_LEN 4 #define NGX_HTTP_V3_PREFIX_INT_LEN 11 @@ -101,13 +102,12 @@ typedef struct { + ngx_flag_t enable; + ngx_flag_t enable_hq; size_t max_table_capacity; ngx_uint_t max_blocked_streams; ngx_uint_t max_concurrent_pushes; ngx_uint_t max_concurrent_streams; -#if (NGX_HTTP_V3_HQ) - ngx_flag_t hq; -#endif ngx_quic_conf_t quic; } ngx_http_v3_srv_conf_t; @@ -147,9 +147,7 @@ struct ngx_http_v3_session_s { off_t payload_bytes; unsigned goaway:1; -#if (NGX_HTTP_V3_HQ) unsigned hq:1; -#endif ngx_connection_t *known_streams[NGX_HTTP_V3_MAX_KNOWN_STREAM]; }; diff --git a/src/http/v3/ngx_http_v3_module.c b/src/http/v3/ngx_http_v3_module.c --- a/src/http/v3/ngx_http_v3_module.c +++ b/src/http/v3/ngx_http_v3_module.c @@ -32,6 +32,20 @@ static ngx_conf_post_t ngx_http_quic_mt static ngx_command_t ngx_http_v3_commands[] = { + { ngx_string("http3"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_CONF_FLAG, + ngx_conf_set_flag_slot, + NGX_HTTP_SRV_CONF_OFFSET, + offsetof(ngx_http_v3_srv_conf_t, enable), + NULL }, + + { ngx_string("http3_hq"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_CONF_FLAG, + ngx_conf_set_flag_slot, + NGX_HTTP_SRV_CONF_OFFSET, + offsetof(ngx_http_v3_srv_conf_t, enable_hq), + NULL }, + { ngx_string("http3_max_concurrent_pushes"), NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_CONF_TAKE1, ngx_conf_set_num_slot, @@ -46,15 +60,6 @@ static ngx_command_t ngx_http_v3_comman offsetof(ngx_http_v3_srv_conf_t, max_concurrent_streams), NULL }, -#if (NGX_HTTP_V3_HQ) - { ngx_string("http3_hq"), - NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_CONF_FLAG, - ngx_conf_set_flag_slot, - NGX_HTTP_SRV_CONF_OFFSET, - offsetof(ngx_http_v3_srv_conf_t, hq), - NULL }, -#endif - { ngx_string("http3_push"), NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1, ngx_http_v3_push, @@ -160,14 +165,12 @@ static ngx_int_t ngx_http_v3_variable(ngx_http_request_t *r, ngx_http_variable_value_t *v, uintptr_t data) { - if (r->connection->quic) { -#if (NGX_HTTP_V3_HQ) + ngx_http_v3_session_t *h3c; - ngx_http_v3_srv_conf_t *h3scf; + if (r->connection->quic) { + h3c = ngx_http_v3_get_session(r->connection); - h3scf = ngx_http_get_module_srv_conf(r, ngx_http_v3_module); - - if (h3scf->hq) { + if (h3c->hq) { v->len = sizeof("hq") - 1; v->valid = 1; v->no_cacheable = 0; @@ -177,8 +180,6 @@ ngx_http_v3_variable(ngx_http_request_t return NGX_OK; } -#endif - v->len = sizeof("h3") - 1; v->valid = 1; v->no_cacheable = 0; @@ -232,12 +233,12 @@ ngx_http_v3_create_srv_conf(ngx_conf_t * * h3scf->quic.timeout = 0; * h3scf->max_blocked_streams = 0; */ + + h3scf->enable = NGX_CONF_UNSET; + h3scf->enable_hq = NGX_CONF_UNSET; h3scf->max_table_capacity = NGX_HTTP_V3_MAX_TABLE_CAPACITY; h3scf->max_concurrent_pushes = NGX_CONF_UNSET_UINT; h3scf->max_concurrent_streams = NGX_CONF_UNSET_UINT; -#if (NGX_HTTP_V3_HQ) - h3scf->hq = NGX_CONF_UNSET; -#endif h3scf->quic.mtu = NGX_CONF_UNSET_SIZE; h3scf->quic.stream_buffer_size = NGX_CONF_UNSET_SIZE; @@ -264,6 +265,10 @@ ngx_http_v3_merge_srv_conf(ngx_conf_t *c ngx_http_ssl_srv_conf_t *sscf; + ngx_conf_merge_value(conf->enable, prev->enable, 0); + + ngx_conf_merge_value(conf->enable_hq, prev->enable_hq, 0); + ngx_conf_merge_uint_value(conf->max_concurrent_pushes, prev->max_concurrent_pushes, 10); @@ -272,11 +277,6 @@ ngx_http_v3_merge_srv_conf(ngx_conf_t *c conf->max_blocked_streams = conf->max_concurrent_streams; -#if (NGX_HTTP_V3_HQ) - ngx_conf_merge_value(conf->hq, prev->hq, 0); -#endif - - ngx_conf_merge_size_value(conf->quic.mtu, prev->quic.mtu, NGX_QUIC_MAX_UDP_PAYLOAD_SIZE); diff --git a/src/http/v3/ngx_http_v3_request.c b/src/http/v3/ngx_http_v3_request.c --- a/src/http/v3/ngx_http_v3_request.c +++ b/src/http/v3/ngx_http_v3_request.c @@ -110,7 +110,10 @@ ngx_http_v3_init_stream(ngx_connection_t ngx_int_t ngx_http_v3_init(ngx_connection_t *c) { + unsigned int len; + const unsigned char *data; ngx_http_v3_session_t *h3c; + ngx_http_v3_srv_conf_t *h3scf; ngx_http_core_loc_conf_t *clcf; ngx_log_debug0(NGX_LOG_DEBUG_HTTP, c->log, 0, "http3 init"); @@ -119,11 +122,23 @@ ngx_http_v3_init(ngx_connection_t *c) clcf = ngx_http_v3_get_module_loc_conf(c, ngx_http_core_module); ngx_add_timer(&h3c->keepalive, clcf->keepalive_timeout); -#if (NGX_HTTP_V3_HQ) - if (h3c->hq) { - return NGX_OK; + h3scf = ngx_http_v3_get_module_srv_conf(c, ngx_http_v3_module); + + if (h3scf->enable_hq) { + if (!h3scf->enable) { + h3c->hq = 1; + return NGX_OK; + } + + SSL_get0_alpn_selected(c->ssl->connection, &data, &len); + + if (len == sizeof(NGX_HTTP_V3_HQ_PROTO) - 1 + && ngx_strncmp(data, NGX_HTTP_V3_HQ_PROTO, len) == 0) + { + h3c->hq = 1; + return NGX_OK; + } } -#endif return ngx_http_v3_send_settings(c); } @@ -147,10 +162,7 @@ ngx_http_v3_shutdown(ngx_connection_t *c if (!h3c->goaway) { h3c->goaway = 1; -#if (NGX_HTTP_V3_HQ) - if (!h3c->hq) -#endif - { + if (!h3c->hq) { (void) ngx_http_v3_send_goaway(c, h3c->next_request_id); } @@ -205,10 +217,7 @@ ngx_http_v3_init_request_stream(ngx_conn { h3c->goaway = 1; -#if (NGX_HTTP_V3_HQ) - if (!h3c->hq) -#endif - { + if (!h3c->hq) { if (ngx_http_v3_send_goaway(c, h3c->next_request_id) != NGX_OK) { ngx_http_close_connection(c); return; @@ -236,10 +245,7 @@ ngx_http_v3_init_request_stream(ngx_conn rev = c->read; -#if (NGX_HTTP_V3_HQ) - if (!h3c->hq) -#endif - { + if (!h3c->hq) { rev->handler = ngx_http_v3_wait_request_handler; c->write->handler = ngx_http_empty_handler; } @@ -398,14 +404,14 @@ ngx_http_v3_wait_request_handler(ngx_eve void ngx_http_v3_reset_stream(ngx_connection_t *c) { + ngx_http_v3_session_t *h3c; ngx_http_v3_srv_conf_t *h3scf; h3scf = ngx_http_v3_get_module_srv_conf(c, ngx_http_v3_module); - if (h3scf->max_table_capacity > 0 && !c->read->eof -#if (NGX_HTTP_V3_HQ) - && !h3scf->hq -#endif + h3c = ngx_http_v3_get_session(c); + + if (h3scf->max_table_capacity > 0 && !c->read->eof && !h3c->hq && (c->quic->id & NGX_QUIC_STREAM_UNIDIRECTIONAL) == 0) { (void) ngx_http_v3_send_cancel_stream(c, c->quic->id); @@ -993,9 +999,11 @@ failed: static ngx_int_t ngx_http_v3_process_request_header(ngx_http_request_t *r) { - ssize_t n; - ngx_buf_t *b; - ngx_connection_t *c; + ssize_t n; + ngx_buf_t *b; + ngx_connection_t *c; + ngx_http_v3_session_t *h3c; + ngx_http_v3_srv_conf_t *h3scf; c = r->connection; @@ -1003,6 +1011,19 @@ ngx_http_v3_process_request_header(ngx_h return NGX_ERROR; } + h3c = ngx_http_v3_get_session(c); + h3scf = ngx_http_get_module_srv_conf(r, ngx_http_v3_module); + + if (!r->http_connection->addr_conf->http3) { + if ((h3c->hq && !h3scf->enable_hq) || (!h3c->hq && !h3scf->enable)) { + ngx_log_error(NGX_LOG_INFO, c->log, 0, + "client attempted to request the server name " + "for which the negotitated protocol is unavailable"); + ngx_http_finalize_request(r, NGX_HTTP_MISDIRECTED_REQUEST); + return NGX_ERROR; + } + } + if (ngx_http_v3_construct_cookie_header(r) != NGX_OK) { return NGX_ERROR; } diff --git a/src/http/v3/ngx_http_v3_uni.c b/src/http/v3/ngx_http_v3_uni.c --- a/src/http/v3/ngx_http_v3_uni.c +++ b/src/http/v3/ngx_http_v3_uni.c @@ -37,12 +37,9 @@ void ngx_http_v3_init_uni_stream(ngx_connection_t *c) { uint64_t n; -#if (NGX_HTTP_V3_HQ) ngx_http_v3_session_t *h3c; -#endif ngx_http_v3_uni_stream_t *us; -#if (NGX_HTTP_V3_HQ) h3c = ngx_http_v3_get_session(c); if (h3c->hq) { ngx_http_v3_finalize_connection(c, @@ -52,7 +49,6 @@ ngx_http_v3_init_uni_stream(ngx_connecti ngx_http_v3_close_uni_stream(c); return; } -#endif ngx_log_debug0(NGX_LOG_DEBUG_HTTP, c->log, 0, "http3 init uni stream"); From arut at nginx.com Wed Feb 1 14:01:09 2023 From: arut at nginx.com (=?iso-8859-1?q?Roman_Arutyunyan?=) Date: Wed, 01 Feb 2023 18:01:09 +0400 Subject: [PATCH 2 of 3] HTTP/2: "http2" directive In-Reply-To: References: Message-ID: <139b348815b3f1975317.1675260069@arut-laptop> # HG changeset patch # User Roman Arutyunyan # Date 1675255790 -14400 # Wed Feb 01 16:49:50 2023 +0400 # Branch quic # Node ID 139b348815b3f19753176988f06b590a3e0af4c0 # Parent 2fcc1c60be1c89aad5464bcc06f1189d1adc885a HTTP/2: "http2" directive. The directive enables HTTP/2 in the current server. The previous way to enable HTTP/2 via "listen ... http2" is now deprecated. diff --git a/src/http/modules/ngx_http_ssl_module.c b/src/http/modules/ngx_http_ssl_module.c --- a/src/http/modules/ngx_http_ssl_module.c +++ b/src/http/modules/ngx_http_ssl_module.c @@ -427,6 +427,9 @@ ngx_http_ssl_alpn_select(ngx_ssl_conn_t #if (NGX_HTTP_V2 || NGX_HTTP_V3) ngx_http_connection_t *hc; #endif +#if (NGX_HTTP_V2) + ngx_http_v2_srv_conf_t *h2scf; +#endif #if (NGX_HTTP_V3) ngx_http_v3_srv_conf_t *h3scf; #endif @@ -448,12 +451,9 @@ ngx_http_ssl_alpn_select(ngx_ssl_conn_t hc = c->data; #endif -#if (NGX_HTTP_V2) - if (hc->addr_conf->http2) { - srv = (unsigned char *) NGX_HTTP_V2_ALPN_PROTO NGX_HTTP_ALPN_PROTOS; - srvlen = sizeof(NGX_HTTP_V2_ALPN_PROTO NGX_HTTP_ALPN_PROTOS) - 1; - } else -#endif + srv = (unsigned char *) NGX_HTTP_ALPN_PROTOS; + srvlen = sizeof(NGX_HTTP_ALPN_PROTOS) - 1; + #if (NGX_HTTP_V3) if (hc->addr_conf->quic) { @@ -479,10 +479,16 @@ ngx_http_ssl_alpn_select(ngx_ssl_conn_t } else #endif +#if (NGX_HTTP_V2) { - srv = (unsigned char *) NGX_HTTP_ALPN_PROTOS; - srvlen = sizeof(NGX_HTTP_ALPN_PROTOS) - 1; + h2scf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_v2_module); + + if (h2scf->enable || hc->addr_conf->http2) { + srv = (unsigned char *) NGX_HTTP_V2_ALPN_PROTO NGX_HTTP_ALPN_PROTOS; + srvlen = sizeof(NGX_HTTP_V2_ALPN_PROTO NGX_HTTP_ALPN_PROTOS) - 1; + } } +#endif if (SSL_select_next_proto((unsigned char **) out, outlen, srv, srvlen, in, inlen) diff --git a/src/http/ngx_http_core_module.c b/src/http/ngx_http_core_module.c --- a/src/http/ngx_http_core_module.c +++ b/src/http/ngx_http_core_module.c @@ -4179,6 +4179,11 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx if (ngx_strcmp(value[n].data, "http2") == 0) { #if (NGX_HTTP_V2) + ngx_conf_log_error(NGX_LOG_WARN, cf, 0, + "the \"listen ... http2\" directive " + "is deprecated, use " + "the \"http2\" directive instead"); + lsopt.http2 = 1; continue; #else diff --git a/src/http/ngx_http_request.c b/src/http/ngx_http_request.c --- a/src/http/ngx_http_request.c +++ b/src/http/ngx_http_request.c @@ -318,12 +318,6 @@ ngx_http_init_connection(ngx_connection_ rev->handler = ngx_http_wait_request_handler; c->write->handler = ngx_http_empty_handler; -#if (NGX_HTTP_V2) - if (hc->addr_conf->http2) { - rev->handler = ngx_http_v2_init; - } -#endif - #if (NGX_HTTP_V3) if (hc->addr_conf->quic) { ngx_http_v3_init_stream(c); @@ -383,6 +377,9 @@ ngx_http_wait_request_handler(ngx_event_ ngx_buf_t *b; ngx_connection_t *c; ngx_http_connection_t *hc; +#if (NGX_HTTP_V2) + ngx_http_v2_srv_conf_t *h2scf; +#endif ngx_http_core_srv_conf_t *cscf; c = rev->data; @@ -429,6 +426,8 @@ ngx_http_wait_request_handler(ngx_event_ b->end = b->last + size; } + size = b->end - b->last; + n = c->recv(c, b->last, size); if (n == NGX_AGAIN) { @@ -443,12 +442,16 @@ ngx_http_wait_request_handler(ngx_event_ return; } - /* - * We are trying to not hold c->buffer's memory for an idle connection. - */ - - if (ngx_pfree(c->pool, b->start) == NGX_OK) { - b->start = NULL; + if (b->pos == b->last) { + + /* + * We are trying to not hold c->buffer's memory for an + * idle connection. + */ + + if (ngx_pfree(c->pool, b->start) == NGX_OK) { + b->start = NULL; + } } return; @@ -489,6 +492,30 @@ ngx_http_wait_request_handler(ngx_event_ } } +#if (NGX_HTTP_V2) + + h2scf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_v2_module); + + if (!c->ssl && (h2scf->enable || hc->addr_conf->http2)) { + + size = ngx_min(sizeof(NGX_HTTP_V2_PREFACE) - 1, + (size_t) (b->last - b->pos)); + + if (ngx_memcmp(b->pos, NGX_HTTP_V2_PREFACE, size) == 0) { + + if (size == sizeof(NGX_HTTP_V2_PREFACE) - 1) { + ngx_http_v2_init(rev); + return; + } + + c->log->action = "waiting for request"; + ngx_post_event(rev, &ngx_posted_events); + return; + } + } + +#endif + c->log->action = "reading client request line"; ngx_reusable_connection(c, 0); @@ -808,13 +835,16 @@ ngx_http_ssl_handshake_handler(ngx_conne #if (NGX_HTTP_V2 \ && defined TLSEXT_TYPE_application_layer_protocol_negotiation) { - unsigned int len; - const unsigned char *data; - ngx_http_connection_t *hc; + unsigned int len; + const unsigned char *data; + ngx_http_connection_t *hc; + ngx_http_v2_srv_conf_t *h2scf; hc = c->data; - if (hc->addr_conf->http2) { + h2scf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_v2_module); + + if (h2scf->enable || hc->addr_conf->http2) { SSL_get0_alpn_selected(c->ssl->connection, &data, &len); diff --git a/src/http/v2/ngx_http_v2.c b/src/http/v2/ngx_http_v2.c --- a/src/http/v2/ngx_http_v2.c +++ b/src/http/v2/ngx_http_v2.c @@ -232,6 +232,7 @@ static ngx_http_v2_parse_header_t ngx_h void ngx_http_v2_init(ngx_event_t *rev) { + u_char *p, *end; ngx_connection_t *c; ngx_pool_cleanup_t *cln; ngx_http_connection_t *hc; @@ -335,6 +336,29 @@ ngx_http_v2_init(ngx_event_t *rev) c->idle = 1; ngx_reusable_connection(c, 0); + if (c->buffer) { + p = c->buffer->pos; + end = c->buffer->last; + + do { + p = h2c->state.handler(h2c, p, end); + + if (p == NULL) { + return; + } + + } while (p != end); + + h2c->total_bytes += p - c->buffer->pos; + c->buffer->pos = p; + + if (h2c->total_bytes / 8 > h2c->payload_bytes + 1048576) { + ngx_log_error(NGX_LOG_INFO, c->log, 0, "http2 flood detected"); + ngx_http_v2_finalize_connection(h2c, NGX_HTTP_V2_NO_ERROR); + return; + } + } + ngx_http_v2_read_handler(rev); } @@ -871,7 +895,7 @@ static u_char * ngx_http_v2_state_preface(ngx_http_v2_connection_t *h2c, u_char *pos, u_char *end) { - static const u_char preface[] = "PRI * HTTP/2.0\r\n"; + static const u_char preface[] = NGX_HTTP_V2_PREFACE_START; if ((size_t) (end - pos) < sizeof(preface) - 1) { return ngx_http_v2_state_save(h2c, pos, end, ngx_http_v2_state_preface); @@ -892,7 +916,7 @@ static u_char * ngx_http_v2_state_preface_end(ngx_http_v2_connection_t *h2c, u_char *pos, u_char *end) { - static const u_char preface[] = "\r\nSM\r\n\r\n"; + static const u_char preface[] = NGX_HTTP_V2_PREFACE_END; if ((size_t) (end - pos) < sizeof(preface) - 1) { return ngx_http_v2_state_save(h2c, pos, end, @@ -3946,10 +3970,22 @@ static void ngx_http_v2_run_request(ngx_http_request_t *r) { ngx_connection_t *fc; + ngx_http_v3_srv_conf_t *h2scf; ngx_http_v2_connection_t *h2c; fc = r->connection; + h2scf = ngx_http_get_module_srv_conf(r, ngx_http_v2_module); + + if (!h2scf->enable && !r->http_connection->addr_conf->http2) { + ngx_log_error(NGX_LOG_INFO, fc->log, 0, + "client attempted to request the server name " + "for which the negotitated protocol is unavailable"); + + ngx_http_finalize_request(r, NGX_HTTP_MISDIRECTED_REQUEST); + goto failed; + } + if (ngx_http_v2_construct_request_line(r) != NGX_OK) { goto failed; } diff --git a/src/http/v2/ngx_http_v2.h b/src/http/v2/ngx_http_v2.h --- a/src/http/v2/ngx_http_v2.h +++ b/src/http/v2/ngx_http_v2.h @@ -64,6 +64,16 @@ typedef u_char *(*ngx_http_v2_handler_pt typedef struct { + ngx_flag_t enable; + size_t pool_size; + ngx_uint_t concurrent_streams; + ngx_uint_t concurrent_pushes; + size_t preread_size; + ngx_uint_t streams_index_mask; +} ngx_http_v2_srv_conf_t; + + +typedef struct { ngx_str_t name; ngx_str_t value; } ngx_http_v2_header_t; @@ -408,9 +418,17 @@ ngx_int_t ngx_http_v2_table_size(ngx_htt #define NGX_HTTP_V2_USER_AGENT_INDEX 58 #define NGX_HTTP_V2_VARY_INDEX 59 +#define NGX_HTTP_V2_PREFACE_START "PRI * HTTP/2.0\r\n" +#define NGX_HTTP_V2_PREFACE_END "\r\nSM\r\n\r\n" +#define NGX_HTTP_V2_PREFACE NGX_HTTP_V2_PREFACE_START \ + NGX_HTTP_V2_PREFACE_END + u_char *ngx_http_v2_string_encode(u_char *dst, u_char *src, size_t len, u_char *tmp, ngx_uint_t lower); +extern ngx_module_t ngx_http_v2_module; + + #endif /* _NGX_HTTP_V2_H_INCLUDED_ */ diff --git a/src/http/v2/ngx_http_v2_module.c b/src/http/v2/ngx_http_v2_module.c --- a/src/http/v2/ngx_http_v2_module.c +++ b/src/http/v2/ngx_http_v2_module.c @@ -75,6 +75,13 @@ static ngx_conf_post_t ngx_http_v2_chun static ngx_command_t ngx_http_v2_commands[] = { + { ngx_string("http2"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_CONF_FLAG, + ngx_conf_set_flag_slot, + NGX_HTTP_SRV_CONF_OFFSET, + offsetof(ngx_http_v2_srv_conf_t, enable), + NULL }, + { ngx_string("http2_recv_buffer_size"), NGX_HTTP_MAIN_CONF|NGX_CONF_TAKE1, ngx_conf_set_size_slot, @@ -314,6 +321,8 @@ ngx_http_v2_create_srv_conf(ngx_conf_t * return NULL; } + h2scf->enable = NGX_CONF_UNSET; + h2scf->pool_size = NGX_CONF_UNSET_SIZE; h2scf->concurrent_streams = NGX_CONF_UNSET_UINT; @@ -333,6 +342,8 @@ ngx_http_v2_merge_srv_conf(ngx_conf_t *c ngx_http_v2_srv_conf_t *prev = parent; ngx_http_v2_srv_conf_t *conf = child; + ngx_conf_merge_value(conf->enable, prev->enable, 0); + ngx_conf_merge_size_value(conf->pool_size, prev->pool_size, 4096); ngx_conf_merge_uint_value(conf->concurrent_streams, diff --git a/src/http/v2/ngx_http_v2_module.h b/src/http/v2/ngx_http_v2_module.h --- a/src/http/v2/ngx_http_v2_module.h +++ b/src/http/v2/ngx_http_v2_module.h @@ -21,15 +21,6 @@ typedef struct { typedef struct { - size_t pool_size; - ngx_uint_t concurrent_streams; - ngx_uint_t concurrent_pushes; - size_t preread_size; - ngx_uint_t streams_index_mask; -} ngx_http_v2_srv_conf_t; - - -typedef struct { size_t chunk_size; ngx_flag_t push_preload; @@ -39,7 +30,4 @@ typedef struct { } ngx_http_v2_loc_conf_t; -extern ngx_module_t ngx_http_v2_module; - - #endif /* _NGX_HTTP_V2_MODULE_H_INCLUDED_ */ From arut at nginx.com Wed Feb 1 14:01:10 2023 From: arut at nginx.com (=?iso-8859-1?q?Roman_Arutyunyan?=) Date: Wed, 01 Feb 2023 18:01:10 +0400 Subject: [PATCH 3 of 3] HTTP/3: trigger more compatibility errors for "listen quic" In-Reply-To: References: Message-ID: # HG changeset patch # User Roman Arutyunyan # Date 1674732333 -14400 # Thu Jan 26 15:25:33 2023 +0400 # Branch quic # Node ID c38eacf014df31766c72fe120ca21a6f64354a8b # Parent 139b348815b3f19753176988f06b590a3e0af4c0 HTTP/3: trigger more compatibility errors for "listen quic". Now "ssl", "proxy_protocol" and "http2" are not allowed with "quic" in "listen" directive. Previously, only "ssl" was not allowed. diff --git a/src/http/ngx_http_core_module.c b/src/http/ngx_http_core_module.c --- a/src/http/ngx_http_core_module.c +++ b/src/http/ngx_http_core_module.c @@ -4326,10 +4326,26 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx return NGX_CONF_ERROR; } -#if (NGX_HTTP_SSL && NGX_HTTP_V3) - if (lsopt.ssl && lsopt.quic) { - return "\"ssl\" parameter is incompatible with \"quic\""; - } +#if (NGX_HTTP_V3) + + if (lsopt.quic) { +#if (NGX_HTTP_SSL) + if (lsopt.ssl) { + return "\"ssl\" parameter is incompatible with \"quic\""; + } +#endif + +#if (NGX_HTTP_V2) + if (lsopt.http2) { + return "\"http2\" parameter is incompatible with \"quic\""; + } +#endif + + if (lsopt.proxy_protocol) { + return "\"proxy_protocol\" parameter is incompatible with \"quic\""; + } + } + #endif for (n = 0; n < u.naddrs; n++) { From l.crilly at f5.com Wed Feb 1 14:18:45 2023 From: l.crilly at f5.com (Liam Crilly) Date: Wed, 1 Feb 2023 14:18:45 +0000 Subject: [PATCH 1 of 3] HTTP/3: "quic" parameter of "listen" directive In-Reply-To: <2fcc1c60be1c89aad546.1675260068@arut-laptop> References: <2fcc1c60be1c89aad546.1675260068@arut-laptop> Message-ID: >     In http, an additional variable is available: $http3. >     The value of $http3 is "h3" for HTTP/3 connections, Will the $server_protocol variable also be extended so that it evaluates to "HTTP/3.0" when appropriate? This is valuable for logging, to avoid complex log_format strings, or the use of map{}s to derive a single variable. Note that the same facility is afforded to HTTP/2 with $http2 and $server_protocol evaluating to "HTTP/2.0". Cheers, Liam. _____ From: nginx-devel on behalf of Roman Arutyunyan Sent: 01 February 2023 14:01 To: nginx-devel at nginx.org Subject: [PATCH 1 of 3] HTTP/3: "quic" parameter of "listen" directive   EXTERNAL MAIL: nginx-devel-bounces at nginx.org # HG changeset patch # User Roman Arutyunyan # Date 1675254688 -14400 #      Wed Feb 01 16:31:28 2023 +0400 # Branch quic # Node ID 2fcc1c60be1c89aad5464bcc06f1189d1adc885a # Parent  def8e398d7c50131f8dac844814fff729da5c86c HTTP/3: "quic" parameter of "listen" directive. Now "listen" directve has a new "quic" parameter which enables QUIC protocol for the address.  Further, to enable HTTP/3, a new directive "http3" is introduced.  The hq-interop protocol is enabled by "http3_hq" as before. Now application protocol is chosen by ALPN. Previously used "http3" parameter of "listen" is deprecated. diff --git a/README b/README --- a/README +++ b/README @@ -93,13 +93,13 @@ 2. Installing    3. Configuration   -    The HTTP "listen" directive got a new option "http3" which enables -    HTTP/3 over QUIC on the specified port. +    The HTTP "listen" directive got a new option "quic" which enables +    QUIC as client transport protocol instead of TCP.        The Stream "listen" directive got a new option "quic" which enables      QUIC as client transport protocol instead of TCP or plain UDP.   -    Along with "http3" or "quic", it's also possible to specify "reuseport" +    Along with "quic", it's also possible to specify "reuseport"      option [8] to make it work properly with multiple workers.        To enable address validation: @@ -133,12 +133,13 @@ 3. Configuration        A number of directives were added that configure HTTP/3:   +        http3 +        http3_hq          http3_stream_buffer_size          http3_max_concurrent_pushes          http3_max_concurrent_streams          http3_push          http3_push_preload -        http3_hq (requires NGX_HTTP_V3_HQ macro)        In http, an additional variable is available: $http3.      The value of $http3 is "h3" for HTTP/3 connections, @@ -160,13 +161,15 @@ Example configuration:          server {              # for better compatibility it's recommended              # to use the same port for quic and https -            listen 8443 http3 reuseport; +            listen 8443 quic reuseport;              listen 8443 ssl;                ssl_certificate     certs/example.com.crt;              ssl_certificate_key certs/example.com.key;              ssl_protocols       TLSv1.3;   +            http3 on; +              location / {                  # required for browsers to direct them into quic port                  add_header Alt-Svc 'h3=":8443"; ma=86400'; diff --git a/src/http/modules/ngx_http_ssl_module.c b/src/http/modules/ngx_http_ssl_module.c --- a/src/http/modules/ngx_http_ssl_module.c +++ b/src/http/modules/ngx_http_ssl_module.c @@ -427,7 +427,7 @@ ngx_http_ssl_alpn_select(ngx_ssl_conn_t  #if (NGX_HTTP_V2 || NGX_HTTP_V3)      ngx_http_connection_t   *hc;  #endif -#if (NGX_HTTP_V3 && NGX_HTTP_V3_HQ) +#if (NGX_HTTP_V3)      ngx_http_v3_srv_conf_t  *h3scf;  #endif  #if (NGX_HTTP_V2 || NGX_HTTP_V3 || NGX_DEBUG) @@ -455,19 +455,26 @@ ngx_http_ssl_alpn_select(ngx_ssl_conn_t      } else  #endif  #if (NGX_HTTP_V3) -    if (hc->addr_conf->http3) { +    if (hc->addr_conf->quic) {   -#if (NGX_HTTP_V3_HQ)          h3scf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_v3_module);   -        if (h3scf->hq) { +        if (h3scf->enable && h3scf->enable_hq) { +            srv = (unsigned char *) NGX_HTTP_V3_ALPN_PROTO +                                    NGX_HTTP_V3_HQ_ALPN_PROTO; +            srvlen = sizeof(NGX_HTTP_V3_ALPN_PROTO NGX_HTTP_V3_HQ_ALPN_PROTO) +                     - 1; + +        } else if (h3scf->enable_hq) {              srv = (unsigned char *) NGX_HTTP_V3_HQ_ALPN_PROTO;              srvlen = sizeof(NGX_HTTP_V3_HQ_ALPN_PROTO) - 1; -        } else -#endif -        { + +        } else if (h3scf->enable || hc->addr_conf->http3) {              srv = (unsigned char *) NGX_HTTP_V3_ALPN_PROTO;              srvlen = sizeof(NGX_HTTP_V3_ALPN_PROTO) - 1; + +        } else { +            return SSL_TLSEXT_ERR_ALERT_FATAL;          }        } else @@ -1313,12 +1320,12 @@ ngx_http_ssl_init(ngx_conf_t *cf)          addr = port[p].addrs.elts;          for (a = 0; a < port[p].addrs.nelts; a++) {   -            if (!addr[a].opt.ssl && !addr[a].opt.http3) { +            if (!addr[a].opt.ssl && !addr[a].opt.quic) {                  continue;              }   -            if (addr[a].opt.http3) { -                name = "http3"; +            if (addr[a].opt.quic) { +                name = "quic";                } else {                  name = "ssl"; @@ -1329,7 +1336,7 @@ ngx_http_ssl_init(ngx_conf_t *cf)                if (sscf->certificates) {   -                if (addr[a].opt.http3 && !(sscf->protocols & NGX_SSL_TLSv1_3)) { +                if (addr[a].opt.quic && !(sscf->protocols & NGX_SSL_TLSv1_3)) {                      ngx_log_error(NGX_LOG_EMERG, cf->log, 0,                                    "\"ssl_protocols\" must enable TLSv1.3 for "                                    "the \"listen ... %s\" directive in %s:%ui", diff --git a/src/http/ngx_http.c b/src/http/ngx_http.c --- a/src/http/ngx_http.c +++ b/src/http/ngx_http.c @@ -1242,6 +1242,7 @@ ngx_http_add_addresses(ngx_conf_t *cf, n  #endif  #if (NGX_HTTP_V3)      ngx_uint_t             http3; +    ngx_uint_t             quic;  #endif        /* @@ -1280,6 +1281,7 @@ ngx_http_add_addresses(ngx_conf_t *cf, n  #endif  #if (NGX_HTTP_V3)          http3 = lsopt->http3 || addr[i].opt.http3; +        quic = lsopt->quic || addr[i].opt.quic;  #endif            if (lsopt->set) { @@ -1319,6 +1321,7 @@ ngx_http_add_addresses(ngx_conf_t *cf, n  #endif  #if (NGX_HTTP_V3)          addr[i].opt.http3 = http3; +        addr[i].opt.quic = quic;  #endif            return NGX_OK; @@ -1823,7 +1826,7 @@ ngx_http_add_listening(ngx_conf_t *cf, n    #if (NGX_HTTP_V3)   -    ls->quic = addr->opt.http3; +    ls->quic = addr->opt.quic;        if (ls->quic) {          ngx_rbtree_init(&ls->rbtree, &ls->sentinel, @@ -1866,6 +1869,7 @@ ngx_http_add_addrs(ngx_conf_t *cf, ngx_h  #endif  #if (NGX_HTTP_V3)          addrs[i].conf.http3 = addr[i].opt.http3; +        addrs[i].conf.quic = addr[i].opt.quic;  #endif          addrs[i].conf.proxy_protocol = addr[i].opt.proxy_protocol;   @@ -1934,6 +1938,7 @@ ngx_http_add_addrs6(ngx_conf_t *cf, ngx_  #endif  #if (NGX_HTTP_V3)          addrs6[i].conf.http3 = addr[i].opt.http3; +        addrs6[i].conf.quic = addr[i].opt.quic;  #endif          addrs6[i].conf.proxy_protocol = addr[i].opt.proxy_protocol;   diff --git a/src/http/ngx_http_core_module.c b/src/http/ngx_http_core_module.c --- a/src/http/ngx_http_core_module.c +++ b/src/http/ngx_http_core_module.c @@ -4191,6 +4191,11 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx            if (ngx_strcmp(value[n].data, "http3") == 0) {  #if (NGX_HTTP_V3) +            ngx_conf_log_error(NGX_LOG_WARN, cf, 0, +                               "the \"listen ... http3\" directive " +                               "is deprecated, use " +                               "the \"http3\" directive instead"); +            lsopt.quic = 1;              lsopt.http3 = 1;              lsopt.type = SOCK_DGRAM;              continue; @@ -4202,6 +4207,19 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx  #endif          }   +        if (ngx_strcmp(value[n].data, "quic") == 0) { +#if (NGX_HTTP_V3) +            lsopt.quic = 1; +            lsopt.type = SOCK_DGRAM; +            continue; +#else +            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, +                               "the \"quic\" parameter requires " +                               "ngx_http_v3_module"); +            return NGX_CONF_ERROR; +#endif +        } +          if (ngx_strncmp(value[n].data, "so_keepalive=", 13) == 0) {                if (ngx_strcmp(&value[n].data[13], "on") == 0) { @@ -4304,8 +4322,8 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx      }    #if (NGX_HTTP_SSL && NGX_HTTP_V3) -    if (lsopt.ssl && lsopt.http3) { -        return "\"ssl\" parameter is incompatible with \"http3\""; +    if (lsopt.ssl && lsopt.quic) { +        return "\"ssl\" parameter is incompatible with \"quic\"";      }  #endif   diff --git a/src/http/ngx_http_core_module.h b/src/http/ngx_http_core_module.h --- a/src/http/ngx_http_core_module.h +++ b/src/http/ngx_http_core_module.h @@ -76,6 +76,7 @@ typedef struct {      unsigned                   ssl:1;      unsigned                   http2:1;      unsigned                   http3:1; +    unsigned                   quic:1;  #if (NGX_HAVE_INET6)      unsigned                   ipv6only:1;  #endif @@ -240,6 +241,7 @@ struct ngx_http_addr_conf_s {      unsigned                   ssl:1;      unsigned                   http2:1;      unsigned                   http3:1; +    unsigned                   quic:1;      unsigned                   proxy_protocol:1;  };   diff --git a/src/http/ngx_http_request.c b/src/http/ngx_http_request.c --- a/src/http/ngx_http_request.c +++ b/src/http/ngx_http_request.c @@ -325,7 +325,7 @@ ngx_http_init_connection(ngx_connection_  #endif    #if (NGX_HTTP_V3) -    if (hc->addr_conf->http3) { +    if (hc->addr_conf->quic) {          ngx_http_v3_init_stream(c);          return;      } diff --git a/src/http/v3/ngx_http_v3.c b/src/http/v3/ngx_http_v3.c --- a/src/http/v3/ngx_http_v3.c +++ b/src/http/v3/ngx_http_v3.c @@ -17,12 +17,9 @@ static void ngx_http_v3_cleanup_session(  ngx_int_t  ngx_http_v3_init_session(ngx_connection_t *c)  { -    ngx_pool_cleanup_t      *cln; -    ngx_http_connection_t   *hc; -    ngx_http_v3_session_t   *h3c; -#if (NGX_HTTP_V3_HQ) -    ngx_http_v3_srv_conf_t  *h3scf; -#endif +    ngx_pool_cleanup_t     *cln; +    ngx_http_connection_t  *hc; +    ngx_http_v3_session_t  *h3c;        hc = c->data;   @@ -36,13 +33,6 @@ ngx_http_v3_init_session(ngx_connection_      h3c->max_push_id = (uint64_t) -1;      h3c->goaway_push_id = (uint64_t) -1;   -#if (NGX_HTTP_V3_HQ) -    h3scf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_v3_module); -    if (h3scf->hq) { -        h3c->hq = 1; -    } -#endif -      ngx_queue_init(&h3c->blocked);      ngx_queue_init(&h3c->pushing);   diff --git a/src/http/v3/ngx_http_v3.h b/src/http/v3/ngx_http_v3.h --- a/src/http/v3/ngx_http_v3.h +++ b/src/http/v3/ngx_http_v3.h @@ -21,6 +21,7 @@    #define NGX_HTTP_V3_ALPN_PROTO                     "\x02h3"  #define NGX_HTTP_V3_HQ_ALPN_PROTO                  "\x0Ahq-interop" +#define NGX_HTTP_V3_HQ_PROTO                       "hq-interop"    #define NGX_HTTP_V3_VARLEN_INT_LEN                 4  #define NGX_HTTP_V3_PREFIX_INT_LEN                 11 @@ -101,13 +102,12 @@      typedef struct { +    ngx_flag_t                    enable; +    ngx_flag_t                    enable_hq;      size_t                        max_table_capacity;      ngx_uint_t                    max_blocked_streams;      ngx_uint_t                    max_concurrent_pushes;      ngx_uint_t                    max_concurrent_streams; -#if (NGX_HTTP_V3_HQ) -    ngx_flag_t                    hq; -#endif      ngx_quic_conf_t               quic;  } ngx_http_v3_srv_conf_t;   @@ -147,9 +147,7 @@ struct ngx_http_v3_session_s {      off_t                         payload_bytes;        unsigned                      goaway:1; -#if (NGX_HTTP_V3_HQ)      unsigned                      hq:1; -#endif        ngx_connection_t             *known_streams[NGX_HTTP_V3_MAX_KNOWN_STREAM];  }; diff --git a/src/http/v3/ngx_http_v3_module.c b/src/http/v3/ngx_http_v3_module.c --- a/src/http/v3/ngx_http_v3_module.c +++ b/src/http/v3/ngx_http_v3_module.c @@ -32,6 +32,20 @@ static ngx_conf_post_t  ngx_http_quic_mt    static ngx_command_t  ngx_http_v3_commands[] = {   +    { ngx_string("http3"), +      NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_CONF_FLAG, +      ngx_conf_set_flag_slot, +      NGX_HTTP_SRV_CONF_OFFSET, +      offsetof(ngx_http_v3_srv_conf_t, enable), +      NULL }, + +    { ngx_string("http3_hq"), +      NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_CONF_FLAG, +      ngx_conf_set_flag_slot, +      NGX_HTTP_SRV_CONF_OFFSET, +      offsetof(ngx_http_v3_srv_conf_t, enable_hq), +      NULL }, +      { ngx_string("http3_max_concurrent_pushes"),        NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_CONF_TAKE1,        ngx_conf_set_num_slot, @@ -46,15 +60,6 @@ static ngx_command_t  ngx_http_v3_comman        offsetof(ngx_http_v3_srv_conf_t, max_concurrent_streams),        NULL },   -#if (NGX_HTTP_V3_HQ) -    { ngx_string("http3_hq"), -      NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_CONF_FLAG, -      ngx_conf_set_flag_slot, -      NGX_HTTP_SRV_CONF_OFFSET, -      offsetof(ngx_http_v3_srv_conf_t, hq), -      NULL }, -#endif -      { ngx_string("http3_push"),        NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1,        ngx_http_v3_push, @@ -160,14 +165,12 @@ static ngx_int_t  ngx_http_v3_variable(ngx_http_request_t *r,      ngx_http_variable_value_t *v, uintptr_t data)  { -    if (r->connection->quic) { -#if (NGX_HTTP_V3_HQ) +    ngx_http_v3_session_t  *h3c;   -        ngx_http_v3_srv_conf_t  *h3scf; +    if (r->connection->quic) { +        h3c = ngx_http_v3_get_session(r->connection);   -        h3scf = ngx_http_get_module_srv_conf(r, ngx_http_v3_module); - -        if (h3scf->hq) { +        if (h3c->hq) {              v->len = sizeof("hq") - 1;              v->valid = 1;              v->no_cacheable = 0; @@ -177,8 +180,6 @@ ngx_http_v3_variable(ngx_http_request_t              return NGX_OK;          }   -#endif -          v->len = sizeof("h3") - 1;          v->valid = 1;          v->no_cacheable = 0; @@ -232,12 +233,12 @@ ngx_http_v3_create_srv_conf(ngx_conf_t *       *     h3scf->quic.timeout = 0;       *     h3scf->max_blocked_streams = 0;       */ + +    h3scf->enable = NGX_CONF_UNSET; +    h3scf->enable_hq = NGX_CONF_UNSET;      h3scf->max_table_capacity = NGX_HTTP_V3_MAX_TABLE_CAPACITY;      h3scf->max_concurrent_pushes = NGX_CONF_UNSET_UINT;      h3scf->max_concurrent_streams = NGX_CONF_UNSET_UINT; -#if (NGX_HTTP_V3_HQ) -    h3scf->hq = NGX_CONF_UNSET; -#endif        h3scf->quic.mtu = NGX_CONF_UNSET_SIZE;      h3scf->quic.stream_buffer_size = NGX_CONF_UNSET_SIZE; @@ -264,6 +265,10 @@ ngx_http_v3_merge_srv_conf(ngx_conf_t *c        ngx_http_ssl_srv_conf_t  *sscf;   +    ngx_conf_merge_value(conf->enable, prev->enable, 0); + +    ngx_conf_merge_value(conf->enable_hq, prev->enable_hq, 0); +      ngx_conf_merge_uint_value(conf->max_concurrent_pushes,                                prev->max_concurrent_pushes, 10);   @@ -272,11 +277,6 @@ ngx_http_v3_merge_srv_conf(ngx_conf_t *c        conf->max_blocked_streams = conf->max_concurrent_streams;   -#if (NGX_HTTP_V3_HQ) -    ngx_conf_merge_value(conf->hq, prev->hq, 0); -#endif - -      ngx_conf_merge_size_value(conf->quic.mtu, prev->quic.mtu,                                NGX_QUIC_MAX_UDP_PAYLOAD_SIZE);   diff --git a/src/http/v3/ngx_http_v3_request.c b/src/http/v3/ngx_http_v3_request.c --- a/src/http/v3/ngx_http_v3_request.c +++ b/src/http/v3/ngx_http_v3_request.c @@ -110,7 +110,10 @@ ngx_http_v3_init_stream(ngx_connection_t  ngx_int_t  ngx_http_v3_init(ngx_connection_t *c)  { +    unsigned int               len; +    const unsigned char       *data;      ngx_http_v3_session_t     *h3c; +    ngx_http_v3_srv_conf_t    *h3scf;      ngx_http_core_loc_conf_t  *clcf;        ngx_log_debug0(NGX_LOG_DEBUG_HTTP, c->log, 0, "http3 init"); @@ -119,11 +122,23 @@ ngx_http_v3_init(ngx_connection_t *c)      clcf = ngx_http_v3_get_module_loc_conf(c, ngx_http_core_module);      ngx_add_timer(&h3c->keepalive, clcf->keepalive_timeout);   -#if (NGX_HTTP_V3_HQ) -    if (h3c->hq) { -        return NGX_OK; +    h3scf = ngx_http_v3_get_module_srv_conf(c, ngx_http_v3_module); + +    if (h3scf->enable_hq) { +        if (!h3scf->enable) { +            h3c->hq = 1; +            return NGX_OK; +        } + +        SSL_get0_alpn_selected(c->ssl->connection, &data, &len); + +        if (len == sizeof(NGX_HTTP_V3_HQ_PROTO) - 1 +            && ngx_strncmp(data, NGX_HTTP_V3_HQ_PROTO, len) == 0) +        { +            h3c->hq = 1; +            return NGX_OK; +        }      } -#endif        return ngx_http_v3_send_settings(c);  } @@ -147,10 +162,7 @@ ngx_http_v3_shutdown(ngx_connection_t *c      if (!h3c->goaway) {          h3c->goaway = 1;   -#if (NGX_HTTP_V3_HQ) -        if (!h3c->hq) -#endif -        { +        if (!h3c->hq) {              (void) ngx_http_v3_send_goaway(c, h3c->next_request_id);          }   @@ -205,10 +217,7 @@ ngx_http_v3_init_request_stream(ngx_conn      {          h3c->goaway = 1;   -#if (NGX_HTTP_V3_HQ) -        if (!h3c->hq) -#endif -        { +        if (!h3c->hq) {              if (ngx_http_v3_send_goaway(c, h3c->next_request_id) != NGX_OK) {                  ngx_http_close_connection(c);                  return; @@ -236,10 +245,7 @@ ngx_http_v3_init_request_stream(ngx_conn        rev = c->read;   -#if (NGX_HTTP_V3_HQ) -    if (!h3c->hq) -#endif -    { +    if (!h3c->hq) {          rev->handler = ngx_http_v3_wait_request_handler;          c->write->handler = ngx_http_empty_handler;      } @@ -398,14 +404,14 @@ ngx_http_v3_wait_request_handler(ngx_eve  void  ngx_http_v3_reset_stream(ngx_connection_t *c)  { +    ngx_http_v3_session_t   *h3c;      ngx_http_v3_srv_conf_t  *h3scf;        h3scf = ngx_http_v3_get_module_srv_conf(c, ngx_http_v3_module);   -    if (h3scf->max_table_capacity > 0 && !c->read->eof -#if (NGX_HTTP_V3_HQ) -        && !h3scf->hq -#endif +    h3c = ngx_http_v3_get_session(c); + +    if (h3scf->max_table_capacity > 0 && !c->read->eof && !h3c->hq          && (c->quic->id & NGX_QUIC_STREAM_UNIDIRECTIONAL) == 0)      {          (void) ngx_http_v3_send_cancel_stream(c, c->quic->id); @@ -993,9 +999,11 @@ failed:  static ngx_int_t  ngx_http_v3_process_request_header(ngx_http_request_t *r)  { -    ssize_t            n; -    ngx_buf_t         *b; -    ngx_connection_t  *c; +    ssize_t                  n; +    ngx_buf_t               *b; +    ngx_connection_t        *c; +    ngx_http_v3_session_t   *h3c; +    ngx_http_v3_srv_conf_t  *h3scf;        c = r->connection;   @@ -1003,6 +1011,19 @@ ngx_http_v3_process_request_header(ngx_h          return NGX_ERROR;      }   +    h3c = ngx_http_v3_get_session(c); +    h3scf = ngx_http_get_module_srv_conf(r, ngx_http_v3_module); + +    if (!r->http_connection->addr_conf->http3) { +        if ((h3c->hq && !h3scf->enable_hq) || (!h3c->hq && !h3scf->enable)) { +            ngx_log_error(NGX_LOG_INFO, c->log, 0, +                          "client attempted to request the server name " +                          "for which the negotitated protocol is unavailable"); +            ngx_http_finalize_request(r, NGX_HTTP_MISDIRECTED_REQUEST); +            return NGX_ERROR; +        } +    } +      if (ngx_http_v3_construct_cookie_header(r) != NGX_OK) {          return NGX_ERROR;      } diff --git a/src/http/v3/ngx_http_v3_uni.c b/src/http/v3/ngx_http_v3_uni.c --- a/src/http/v3/ngx_http_v3_uni.c +++ b/src/http/v3/ngx_http_v3_uni.c @@ -37,12 +37,9 @@ void  ngx_http_v3_init_uni_stream(ngx_connection_t *c)  {      uint64_t                   n; -#if (NGX_HTTP_V3_HQ)      ngx_http_v3_session_t     *h3c; -#endif      ngx_http_v3_uni_stream_t  *us;   -#if (NGX_HTTP_V3_HQ)      h3c = ngx_http_v3_get_session(c);      if (h3c->hq) {          ngx_http_v3_finalize_connection(c, @@ -52,7 +49,6 @@ ngx_http_v3_init_uni_stream(ngx_connecti          ngx_http_v3_close_uni_stream(c);          return;      } -#endif        ngx_log_debug0(NGX_LOG_DEBUG_HTTP, c->log, 0, "http3 init uni stream");   _______________________________________________ nginx-devel mailing list nginx-devel at nginx.org https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailman.nginx.org%2Fmailman%2Flistinfo%2Fnginx-devel&data=05%7C01%7Cl.crilly%40f5.com%7C318773b48c5c4c53549108db045cd5c2%7Cdd3dfd2f6a3b40d19be0bf8327d81c50%7C0%7C0%7C638108568989117258%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=9SW2Zzrz7qhk%2FNSH2RYLCQMuSLKIsL9tPBLzPISDXDw%3D&reserved=0 From arut at nginx.com Wed Feb 1 14:25:28 2023 From: arut at nginx.com (Roman Arutyunyan) Date: Wed, 1 Feb 2023 18:25:28 +0400 Subject: [PATCH 1 of 3] HTTP/3: "quic" parameter of "listen" directive In-Reply-To: References: <2fcc1c60be1c89aad546.1675260068@arut-laptop> Message-ID: <20230201142528.wr4l3fa43wab4ekb@N00W24XTQX> Hi Liam, On Wed, Feb 01, 2023 at 02:18:45PM +0000, Liam Crilly via nginx-devel wrote: > >     In http, an additional variable is available: $http3. > >     The value of $http3 is "h3" for HTTP/3 connections, > > Will the $server_protocol variable also be extended so that it evaluates > to "HTTP/3.0" when appropriate? > > This is valuable for logging, to avoid complex log_format strings, or the > use of map{}s to derive a single variable. Note that the same facility is > afforded to HTTP/2 with $http2 and $server_protocol evaluating to "HTTP/2.0". Yes. In fact, $server_protocol has already been supported for http3. Its value is exactly "HTTP/3.0". > > Cheers, > Liam. > > _____ > From: nginx-devel on behalf of Roman Arutyunyan > Sent: 01 February 2023 14:01 > To: nginx-devel at nginx.org > Subject: [PATCH 1 of 3] HTTP/3: "quic" parameter of "listen" directive >   > EXTERNAL MAIL: nginx-devel-bounces at nginx.org > > # HG changeset patch > # User Roman Arutyunyan > # Date 1675254688 -14400 > #      Wed Feb 01 16:31:28 2023 +0400 > # Branch quic > # Node ID 2fcc1c60be1c89aad5464bcc06f1189d1adc885a > # Parent  def8e398d7c50131f8dac844814fff729da5c86c > HTTP/3: "quic" parameter of "listen" directive. > > Now "listen" directve has a new "quic" parameter which enables QUIC protocol > for the address.  Further, to enable HTTP/3, a new directive "http3" is > introduced.  The hq-interop protocol is enabled by "http3_hq" as before. > Now application protocol is chosen by ALPN. > > Previously used "http3" parameter of "listen" is deprecated. > > diff --git a/README b/README > --- a/README > +++ b/README > @@ -93,13 +93,13 @@ 2. Installing >   >  3. Configuration >   > -    The HTTP "listen" directive got a new option "http3" which enables > -    HTTP/3 over QUIC on the specified port. > +    The HTTP "listen" directive got a new option "quic" which enables > +    QUIC as client transport protocol instead of TCP. >   >      The Stream "listen" directive got a new option "quic" which enables >      QUIC as client transport protocol instead of TCP or plain UDP. >   > -    Along with "http3" or "quic", it's also possible to specify "reuseport" > +    Along with "quic", it's also possible to specify "reuseport" >      option [8] to make it work properly with multiple workers. >   >      To enable address validation: > @@ -133,12 +133,13 @@ 3. Configuration >   >      A number of directives were added that configure HTTP/3: >   > +        http3 > +        http3_hq >          http3_stream_buffer_size >          http3_max_concurrent_pushes >          http3_max_concurrent_streams >          http3_push >          http3_push_preload > -        http3_hq (requires NGX_HTTP_V3_HQ macro) >   >      In http, an additional variable is available: $http3. >      The value of $http3 is "h3" for HTTP/3 connections, > @@ -160,13 +161,15 @@ Example configuration: >          server { >              # for better compatibility it's recommended >              # to use the same port for quic and https > -            listen 8443 http3 reuseport; > +            listen 8443 quic reuseport; >              listen 8443 ssl; >   >              ssl_certificate     certs/example.com.crt; >              ssl_certificate_key certs/example.com.key; >              ssl_protocols       TLSv1.3; >   > +            http3 on; > + >              location / { >                  # required for browsers to direct them into quic port >                  add_header Alt-Svc 'h3=":8443"; ma=86400'; > diff --git a/src/http/modules/ngx_http_ssl_module.c b/src/http/modules/ngx_http_ssl_module.c > --- a/src/http/modules/ngx_http_ssl_module.c > +++ b/src/http/modules/ngx_http_ssl_module.c > @@ -427,7 +427,7 @@ ngx_http_ssl_alpn_select(ngx_ssl_conn_t >  #if (NGX_HTTP_V2 || NGX_HTTP_V3) >      ngx_http_connection_t   *hc; >  #endif > -#if (NGX_HTTP_V3 && NGX_HTTP_V3_HQ) > +#if (NGX_HTTP_V3) >      ngx_http_v3_srv_conf_t  *h3scf; >  #endif >  #if (NGX_HTTP_V2 || NGX_HTTP_V3 || NGX_DEBUG) > @@ -455,19 +455,26 @@ ngx_http_ssl_alpn_select(ngx_ssl_conn_t >      } else >  #endif >  #if (NGX_HTTP_V3) > -    if (hc->addr_conf->http3) { > +    if (hc->addr_conf->quic) { >   > -#if (NGX_HTTP_V3_HQ) >          h3scf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_v3_module); >   > -        if (h3scf->hq) { > +        if (h3scf->enable && h3scf->enable_hq) { > +            srv = (unsigned char *) NGX_HTTP_V3_ALPN_PROTO > +                                    NGX_HTTP_V3_HQ_ALPN_PROTO; > +            srvlen = sizeof(NGX_HTTP_V3_ALPN_PROTO NGX_HTTP_V3_HQ_ALPN_PROTO) > +                     - 1; > + > +        } else if (h3scf->enable_hq) { >              srv = (unsigned char *) NGX_HTTP_V3_HQ_ALPN_PROTO; >              srvlen = sizeof(NGX_HTTP_V3_HQ_ALPN_PROTO) - 1; > -        } else > -#endif > -        { > + > +        } else if (h3scf->enable || hc->addr_conf->http3) { >              srv = (unsigned char *) NGX_HTTP_V3_ALPN_PROTO; >              srvlen = sizeof(NGX_HTTP_V3_ALPN_PROTO) - 1; > + > +        } else { > +            return SSL_TLSEXT_ERR_ALERT_FATAL; >          } >   >      } else > @@ -1313,12 +1320,12 @@ ngx_http_ssl_init(ngx_conf_t *cf) >          addr = port[p].addrs.elts; >          for (a = 0; a < port[p].addrs.nelts; a++) { >   > -            if (!addr[a].opt.ssl && !addr[a].opt.http3) { > +            if (!addr[a].opt.ssl && !addr[a].opt.quic) { >                  continue; >              } >   > -            if (addr[a].opt.http3) { > -                name = "http3"; > +            if (addr[a].opt.quic) { > +                name = "quic"; >   >              } else { >                  name = "ssl"; > @@ -1329,7 +1336,7 @@ ngx_http_ssl_init(ngx_conf_t *cf) >   >              if (sscf->certificates) { >   > -                if (addr[a].opt.http3 && !(sscf->protocols & NGX_SSL_TLSv1_3)) { > +                if (addr[a].opt.quic && !(sscf->protocols & NGX_SSL_TLSv1_3)) { >                      ngx_log_error(NGX_LOG_EMERG, cf->log, 0, >                                    "\"ssl_protocols\" must enable TLSv1.3 for " >                                    "the \"listen ... %s\" directive in %s:%ui", > diff --git a/src/http/ngx_http.c b/src/http/ngx_http.c > --- a/src/http/ngx_http.c > +++ b/src/http/ngx_http.c > @@ -1242,6 +1242,7 @@ ngx_http_add_addresses(ngx_conf_t *cf, n >  #endif >  #if (NGX_HTTP_V3) >      ngx_uint_t             http3; > +    ngx_uint_t             quic; >  #endif >   >      /* > @@ -1280,6 +1281,7 @@ ngx_http_add_addresses(ngx_conf_t *cf, n >  #endif >  #if (NGX_HTTP_V3) >          http3 = lsopt->http3 || addr[i].opt.http3; > +        quic = lsopt->quic || addr[i].opt.quic; >  #endif >   >          if (lsopt->set) { > @@ -1319,6 +1321,7 @@ ngx_http_add_addresses(ngx_conf_t *cf, n >  #endif >  #if (NGX_HTTP_V3) >          addr[i].opt.http3 = http3; > +        addr[i].opt.quic = quic; >  #endif >   >          return NGX_OK; > @@ -1823,7 +1826,7 @@ ngx_http_add_listening(ngx_conf_t *cf, n >   >  #if (NGX_HTTP_V3) >   > -    ls->quic = addr->opt.http3; > +    ls->quic = addr->opt.quic; >   >      if (ls->quic) { >          ngx_rbtree_init(&ls->rbtree, &ls->sentinel, > @@ -1866,6 +1869,7 @@ ngx_http_add_addrs(ngx_conf_t *cf, ngx_h >  #endif >  #if (NGX_HTTP_V3) >          addrs[i].conf.http3 = addr[i].opt.http3; > +        addrs[i].conf.quic = addr[i].opt.quic; >  #endif >          addrs[i].conf.proxy_protocol = addr[i].opt.proxy_protocol; >   > @@ -1934,6 +1938,7 @@ ngx_http_add_addrs6(ngx_conf_t *cf, ngx_ >  #endif >  #if (NGX_HTTP_V3) >          addrs6[i].conf.http3 = addr[i].opt.http3; > +        addrs6[i].conf.quic = addr[i].opt.quic; >  #endif >          addrs6[i].conf.proxy_protocol = addr[i].opt.proxy_protocol; >   > diff --git a/src/http/ngx_http_core_module.c b/src/http/ngx_http_core_module.c > --- a/src/http/ngx_http_core_module.c > +++ b/src/http/ngx_http_core_module.c > @@ -4191,6 +4191,11 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx >   >          if (ngx_strcmp(value[n].data, "http3") == 0) { >  #if (NGX_HTTP_V3) > +            ngx_conf_log_error(NGX_LOG_WARN, cf, 0, > +                               "the \"listen ... http3\" directive " > +                               "is deprecated, use " > +                               "the \"http3\" directive instead"); > +            lsopt.quic = 1; >              lsopt.http3 = 1; >              lsopt.type = SOCK_DGRAM; >              continue; > @@ -4202,6 +4207,19 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx >  #endif >          } >   > +        if (ngx_strcmp(value[n].data, "quic") == 0) { > +#if (NGX_HTTP_V3) > +            lsopt.quic = 1; > +            lsopt.type = SOCK_DGRAM; > +            continue; > +#else > +            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, > +                               "the \"quic\" parameter requires " > +                               "ngx_http_v3_module"); > +            return NGX_CONF_ERROR; > +#endif > +        } > + >          if (ngx_strncmp(value[n].data, "so_keepalive=", 13) == 0) { >   >              if (ngx_strcmp(&value[n].data[13], "on") == 0) { > @@ -4304,8 +4322,8 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx >      } >   >  #if (NGX_HTTP_SSL && NGX_HTTP_V3) > -    if (lsopt.ssl && lsopt.http3) { > -        return "\"ssl\" parameter is incompatible with \"http3\""; > +    if (lsopt.ssl && lsopt.quic) { > +        return "\"ssl\" parameter is incompatible with \"quic\""; >      } >  #endif >   > diff --git a/src/http/ngx_http_core_module.h b/src/http/ngx_http_core_module.h > --- a/src/http/ngx_http_core_module.h > +++ b/src/http/ngx_http_core_module.h > @@ -76,6 +76,7 @@ typedef struct { >      unsigned                   ssl:1; >      unsigned                   http2:1; >      unsigned                   http3:1; > +    unsigned                   quic:1; >  #if (NGX_HAVE_INET6) >      unsigned                   ipv6only:1; >  #endif > @@ -240,6 +241,7 @@ struct ngx_http_addr_conf_s { >      unsigned                   ssl:1; >      unsigned                   http2:1; >      unsigned                   http3:1; > +    unsigned                   quic:1; >      unsigned                   proxy_protocol:1; >  }; >   > diff --git a/src/http/ngx_http_request.c b/src/http/ngx_http_request.c > --- a/src/http/ngx_http_request.c > +++ b/src/http/ngx_http_request.c > @@ -325,7 +325,7 @@ ngx_http_init_connection(ngx_connection_ >  #endif >   >  #if (NGX_HTTP_V3) > -    if (hc->addr_conf->http3) { > +    if (hc->addr_conf->quic) { >          ngx_http_v3_init_stream(c); >          return; >      } > diff --git a/src/http/v3/ngx_http_v3.c b/src/http/v3/ngx_http_v3.c > --- a/src/http/v3/ngx_http_v3.c > +++ b/src/http/v3/ngx_http_v3.c > @@ -17,12 +17,9 @@ static void ngx_http_v3_cleanup_session( >  ngx_int_t >  ngx_http_v3_init_session(ngx_connection_t *c) >  { > -    ngx_pool_cleanup_t      *cln; > -    ngx_http_connection_t   *hc; > -    ngx_http_v3_session_t   *h3c; > -#if (NGX_HTTP_V3_HQ) > -    ngx_http_v3_srv_conf_t  *h3scf; > -#endif > +    ngx_pool_cleanup_t     *cln; > +    ngx_http_connection_t  *hc; > +    ngx_http_v3_session_t  *h3c; >   >      hc = c->data; >   > @@ -36,13 +33,6 @@ ngx_http_v3_init_session(ngx_connection_ >      h3c->max_push_id = (uint64_t) -1; >      h3c->goaway_push_id = (uint64_t) -1; >   > -#if (NGX_HTTP_V3_HQ) > -    h3scf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_v3_module); > -    if (h3scf->hq) { > -        h3c->hq = 1; > -    } > -#endif > - >      ngx_queue_init(&h3c->blocked); >      ngx_queue_init(&h3c->pushing); >   > diff --git a/src/http/v3/ngx_http_v3.h b/src/http/v3/ngx_http_v3.h > --- a/src/http/v3/ngx_http_v3.h > +++ b/src/http/v3/ngx_http_v3.h > @@ -21,6 +21,7 @@ >   >  #define NGX_HTTP_V3_ALPN_PROTO                     "\x02h3" >  #define NGX_HTTP_V3_HQ_ALPN_PROTO                  "\x0Ahq-interop" > +#define NGX_HTTP_V3_HQ_PROTO                       "hq-interop" >   >  #define NGX_HTTP_V3_VARLEN_INT_LEN                 4 >  #define NGX_HTTP_V3_PREFIX_INT_LEN                 11 > @@ -101,13 +102,12 @@ >   >   >  typedef struct { > +    ngx_flag_t                    enable; > +    ngx_flag_t                    enable_hq; >      size_t                        max_table_capacity; >      ngx_uint_t                    max_blocked_streams; >      ngx_uint_t                    max_concurrent_pushes; >      ngx_uint_t                    max_concurrent_streams; > -#if (NGX_HTTP_V3_HQ) > -    ngx_flag_t                    hq; > -#endif >      ngx_quic_conf_t               quic; >  } ngx_http_v3_srv_conf_t; >   > @@ -147,9 +147,7 @@ struct ngx_http_v3_session_s { >      off_t                         payload_bytes; >   >      unsigned                      goaway:1; > -#if (NGX_HTTP_V3_HQ) >      unsigned                      hq:1; > -#endif >   >      ngx_connection_t             *known_streams[NGX_HTTP_V3_MAX_KNOWN_STREAM]; >  }; > diff --git a/src/http/v3/ngx_http_v3_module.c b/src/http/v3/ngx_http_v3_module.c > --- a/src/http/v3/ngx_http_v3_module.c > +++ b/src/http/v3/ngx_http_v3_module.c > @@ -32,6 +32,20 @@ static ngx_conf_post_t  ngx_http_quic_mt >   >  static ngx_command_t  ngx_http_v3_commands[] = { >   > +    { ngx_string("http3"), > +      NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_CONF_FLAG, > +      ngx_conf_set_flag_slot, > +      NGX_HTTP_SRV_CONF_OFFSET, > +      offsetof(ngx_http_v3_srv_conf_t, enable), > +      NULL }, > + > +    { ngx_string("http3_hq"), > +      NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_CONF_FLAG, > +      ngx_conf_set_flag_slot, > +      NGX_HTTP_SRV_CONF_OFFSET, > +      offsetof(ngx_http_v3_srv_conf_t, enable_hq), > +      NULL }, > + >      { ngx_string("http3_max_concurrent_pushes"), >        NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_CONF_TAKE1, >        ngx_conf_set_num_slot, > @@ -46,15 +60,6 @@ static ngx_command_t  ngx_http_v3_comman >        offsetof(ngx_http_v3_srv_conf_t, max_concurrent_streams), >        NULL }, >   > -#if (NGX_HTTP_V3_HQ) > -    { ngx_string("http3_hq"), > -      NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_CONF_FLAG, > -      ngx_conf_set_flag_slot, > -      NGX_HTTP_SRV_CONF_OFFSET, > -      offsetof(ngx_http_v3_srv_conf_t, hq), > -      NULL }, > -#endif > - >      { ngx_string("http3_push"), >        NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1, >        ngx_http_v3_push, > @@ -160,14 +165,12 @@ static ngx_int_t >  ngx_http_v3_variable(ngx_http_request_t *r, >      ngx_http_variable_value_t *v, uintptr_t data) >  { > -    if (r->connection->quic) { > -#if (NGX_HTTP_V3_HQ) > +    ngx_http_v3_session_t  *h3c; >   > -        ngx_http_v3_srv_conf_t  *h3scf; > +    if (r->connection->quic) { > +        h3c = ngx_http_v3_get_session(r->connection); >   > -        h3scf = ngx_http_get_module_srv_conf(r, ngx_http_v3_module); > - > -        if (h3scf->hq) { > +        if (h3c->hq) { >              v->len = sizeof("hq") - 1; >              v->valid = 1; >              v->no_cacheable = 0; > @@ -177,8 +180,6 @@ ngx_http_v3_variable(ngx_http_request_t >              return NGX_OK; >          } >   > -#endif > - >          v->len = sizeof("h3") - 1; >          v->valid = 1; >          v->no_cacheable = 0; > @@ -232,12 +233,12 @@ ngx_http_v3_create_srv_conf(ngx_conf_t * >       *     h3scf->quic.timeout = 0; >       *     h3scf->max_blocked_streams = 0; >       */ > + > +    h3scf->enable = NGX_CONF_UNSET; > +    h3scf->enable_hq = NGX_CONF_UNSET; >      h3scf->max_table_capacity = NGX_HTTP_V3_MAX_TABLE_CAPACITY; >      h3scf->max_concurrent_pushes = NGX_CONF_UNSET_UINT; >      h3scf->max_concurrent_streams = NGX_CONF_UNSET_UINT; > -#if (NGX_HTTP_V3_HQ) > -    h3scf->hq = NGX_CONF_UNSET; > -#endif >   >      h3scf->quic.mtu = NGX_CONF_UNSET_SIZE; >      h3scf->quic.stream_buffer_size = NGX_CONF_UNSET_SIZE; > @@ -264,6 +265,10 @@ ngx_http_v3_merge_srv_conf(ngx_conf_t *c >   >      ngx_http_ssl_srv_conf_t  *sscf; >   > +    ngx_conf_merge_value(conf->enable, prev->enable, 0); > + > +    ngx_conf_merge_value(conf->enable_hq, prev->enable_hq, 0); > + >      ngx_conf_merge_uint_value(conf->max_concurrent_pushes, >                                prev->max_concurrent_pushes, 10); >   > @@ -272,11 +277,6 @@ ngx_http_v3_merge_srv_conf(ngx_conf_t *c >   >      conf->max_blocked_streams = conf->max_concurrent_streams; >   > -#if (NGX_HTTP_V3_HQ) > -    ngx_conf_merge_value(conf->hq, prev->hq, 0); > -#endif > - > - >      ngx_conf_merge_size_value(conf->quic.mtu, prev->quic.mtu, >                                NGX_QUIC_MAX_UDP_PAYLOAD_SIZE); >   > diff --git a/src/http/v3/ngx_http_v3_request.c b/src/http/v3/ngx_http_v3_request.c > --- a/src/http/v3/ngx_http_v3_request.c > +++ b/src/http/v3/ngx_http_v3_request.c > @@ -110,7 +110,10 @@ ngx_http_v3_init_stream(ngx_connection_t >  ngx_int_t >  ngx_http_v3_init(ngx_connection_t *c) >  { > +    unsigned int               len; > +    const unsigned char       *data; >      ngx_http_v3_session_t     *h3c; > +    ngx_http_v3_srv_conf_t    *h3scf; >      ngx_http_core_loc_conf_t  *clcf; >   >      ngx_log_debug0(NGX_LOG_DEBUG_HTTP, c->log, 0, "http3 init"); > @@ -119,11 +122,23 @@ ngx_http_v3_init(ngx_connection_t *c) >      clcf = ngx_http_v3_get_module_loc_conf(c, ngx_http_core_module); >      ngx_add_timer(&h3c->keepalive, clcf->keepalive_timeout); >   > -#if (NGX_HTTP_V3_HQ) > -    if (h3c->hq) { > -        return NGX_OK; > +    h3scf = ngx_http_v3_get_module_srv_conf(c, ngx_http_v3_module); > + > +    if (h3scf->enable_hq) { > +        if (!h3scf->enable) { > +            h3c->hq = 1; > +            return NGX_OK; > +        } > + > +        SSL_get0_alpn_selected(c->ssl->connection, &data, &len); > + > +        if (len == sizeof(NGX_HTTP_V3_HQ_PROTO) - 1 > +            && ngx_strncmp(data, NGX_HTTP_V3_HQ_PROTO, len) == 0) > +        { > +            h3c->hq = 1; > +            return NGX_OK; > +        } >      } > -#endif >   >      return ngx_http_v3_send_settings(c); >  } > @@ -147,10 +162,7 @@ ngx_http_v3_shutdown(ngx_connection_t *c >      if (!h3c->goaway) { >          h3c->goaway = 1; >   > -#if (NGX_HTTP_V3_HQ) > -        if (!h3c->hq) > -#endif > -        { > +        if (!h3c->hq) { >              (void) ngx_http_v3_send_goaway(c, h3c->next_request_id); >          } >   > @@ -205,10 +217,7 @@ ngx_http_v3_init_request_stream(ngx_conn >      { >          h3c->goaway = 1; >   > -#if (NGX_HTTP_V3_HQ) > -        if (!h3c->hq) > -#endif > -        { > +        if (!h3c->hq) { >              if (ngx_http_v3_send_goaway(c, h3c->next_request_id) != NGX_OK) { >                  ngx_http_close_connection(c); >                  return; > @@ -236,10 +245,7 @@ ngx_http_v3_init_request_stream(ngx_conn >   >      rev = c->read; >   > -#if (NGX_HTTP_V3_HQ) > -    if (!h3c->hq) > -#endif > -    { > +    if (!h3c->hq) { >          rev->handler = ngx_http_v3_wait_request_handler; >          c->write->handler = ngx_http_empty_handler; >      } > @@ -398,14 +404,14 @@ ngx_http_v3_wait_request_handler(ngx_eve >  void >  ngx_http_v3_reset_stream(ngx_connection_t *c) >  { > +    ngx_http_v3_session_t   *h3c; >      ngx_http_v3_srv_conf_t  *h3scf; >   >      h3scf = ngx_http_v3_get_module_srv_conf(c, ngx_http_v3_module); >   > -    if (h3scf->max_table_capacity > 0 && !c->read->eof > -#if (NGX_HTTP_V3_HQ) > -        && !h3scf->hq > -#endif > +    h3c = ngx_http_v3_get_session(c); > + > +    if (h3scf->max_table_capacity > 0 && !c->read->eof && !h3c->hq >          && (c->quic->id & NGX_QUIC_STREAM_UNIDIRECTIONAL) == 0) >      { >          (void) ngx_http_v3_send_cancel_stream(c, c->quic->id); > @@ -993,9 +999,11 @@ failed: >  static ngx_int_t >  ngx_http_v3_process_request_header(ngx_http_request_t *r) >  { > -    ssize_t            n; > -    ngx_buf_t         *b; > -    ngx_connection_t  *c; > +    ssize_t                  n; > +    ngx_buf_t               *b; > +    ngx_connection_t        *c; > +    ngx_http_v3_session_t   *h3c; > +    ngx_http_v3_srv_conf_t  *h3scf; >   >      c = r->connection; >   > @@ -1003,6 +1011,19 @@ ngx_http_v3_process_request_header(ngx_h >          return NGX_ERROR; >      } >   > +    h3c = ngx_http_v3_get_session(c); > +    h3scf = ngx_http_get_module_srv_conf(r, ngx_http_v3_module); > + > +    if (!r->http_connection->addr_conf->http3) { > +        if ((h3c->hq && !h3scf->enable_hq) || (!h3c->hq && !h3scf->enable)) { > +            ngx_log_error(NGX_LOG_INFO, c->log, 0, > +                          "client attempted to request the server name " > +                          "for which the negotitated protocol is unavailable"); > +            ngx_http_finalize_request(r, NGX_HTTP_MISDIRECTED_REQUEST); > +            return NGX_ERROR; > +        } > +    } > + >      if (ngx_http_v3_construct_cookie_header(r) != NGX_OK) { >          return NGX_ERROR; >      } > diff --git a/src/http/v3/ngx_http_v3_uni.c b/src/http/v3/ngx_http_v3_uni.c > --- a/src/http/v3/ngx_http_v3_uni.c > +++ b/src/http/v3/ngx_http_v3_uni.c > @@ -37,12 +37,9 @@ void >  ngx_http_v3_init_uni_stream(ngx_connection_t *c) >  { >      uint64_t                   n; > -#if (NGX_HTTP_V3_HQ) >      ngx_http_v3_session_t     *h3c; > -#endif >      ngx_http_v3_uni_stream_t  *us; >   > -#if (NGX_HTTP_V3_HQ) >      h3c = ngx_http_v3_get_session(c); >      if (h3c->hq) { >          ngx_http_v3_finalize_connection(c, > @@ -52,7 +49,6 @@ ngx_http_v3_init_uni_stream(ngx_connecti >          ngx_http_v3_close_uni_stream(c); >          return; >      } > -#endif >   >      ngx_log_debug0(NGX_LOG_DEBUG_HTTP, c->log, 0, "http3 init uni stream"); >   > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailman.nginx.org%2Fmailman%2Flistinfo%2Fnginx-devel&data=05%7C01%7Cl.crilly%40f5.com%7C318773b48c5c4c53549108db045cd5c2%7Cdd3dfd2f6a3b40d19be0bf8327d81c50%7C0%7C0%7C638108568989117258%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=9SW2Zzrz7qhk%2FNSH2RYLCQMuSLKIsL9tPBLzPISDXDw%3D&reserved=0 > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > https://mailman.nginx.org/mailman/listinfo/nginx-devel -- Roman Arutyunyan From xeioex at nginx.com Thu Feb 2 03:07:56 2023 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Thu, 02 Feb 2023 03:07:56 +0000 Subject: [njs] Types: added ts types for the xml API. Message-ID: details: https://hg.nginx.org/njs/rev/502a63c67947 branches: changeset: 2035:502a63c67947 user: Dmitry Volyntsev date: Wed Feb 01 19:04:39 2023 -0800 description: Types: added ts types for the xml API. diffstat: test/ts/package.json | 2 +- test/ts/test.ts | 20 +++++ ts/index.d.ts | 1 + ts/njs_modules/xml.d.ts | 162 ++++++++++++++++++++++++++++++++++++++++++++++++ ts/package.json | 2 +- 5 files changed, 185 insertions(+), 2 deletions(-) diffs (234 lines): diff -r 948b167202b2 -r 502a63c67947 test/ts/package.json --- a/test/ts/package.json Mon Jan 30 18:19:16 2023 -0800 +++ b/test/ts/package.json Wed Feb 01 19:04:39 2023 -0800 @@ -10,6 +10,6 @@ "license": "BSD-2-Clause", "devDependencies": { "njs-types": "file:../../ts", - "typescript": "~4.0.3" + "typescript": "~4.9.5" } } diff -r 948b167202b2 -r 502a63c67947 test/ts/test.ts --- a/test/ts/test.ts Mon Jan 30 18:19:16 2023 -0800 +++ b/test/ts/test.ts Wed Feb 01 19:04:39 2023 -0800 @@ -1,6 +1,7 @@ import fs from 'fs'; import qs from 'querystring'; import cr from 'crypto'; +import xml from 'xml'; async function http_module(r: NginxHTTPRequest) { var bs: NjsByteString; @@ -147,6 +148,25 @@ function qs_module(str: NjsByteString) { s = qs.stringify(o); } +function xml_module(str: NjsByteString) { + let doc; + let node; + let children, selectedChildren; + + doc = xml.parse(str); + node = doc.$root; + + node.$ns; + children = node.$tags; + selectedChildren = node.$tags$xxx; + + node?.xxx?.yyy?.$attr$zzz; + + let buf:Buffer = xml.exclusiveC14n(node); + buf = xml.exclusiveC14n(doc, node.xxx, false); + buf = xml.exclusiveC14n(node, null, true, "aa bb"); +} + function crypto_module(str: NjsByteString) { var h; var b:Buffer; diff -r 948b167202b2 -r 502a63c67947 ts/index.d.ts --- a/ts/index.d.ts Mon Jan 30 18:19:16 2023 -0800 +++ b/ts/index.d.ts Wed Feb 01 19:04:39 2023 -0800 @@ -2,4 +2,5 @@ /// /// /// +/// /// diff -r 948b167202b2 -r 502a63c67947 ts/njs_modules/xml.d.ts --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/ts/njs_modules/xml.d.ts Wed Feb 01 19:04:39 2023 -0800 @@ -0,0 +1,162 @@ +/// + +declare module "xml" { + + type XMLTagName = + | `_${string}` + | `a${string}` + | `b${string}` + | `c${string}` + | `d${string}` + | `e${string}` + | `f${string}` + | `g${string}` + | `h${string}` + | `i${string}` + | `j${string}` + | `k${string}` + | `l${string}` + | `m${string}` + | `n${string}` + | `o${string}` + | `p${string}` + | `q${string}` + | `r${string}` + | `s${string}` + | `t${string}` + | `u${string}` + | `v${string}` + | `w${string}` + | `x${string}` + | `y${string}` + | `z${string}` + | `A${string}` + | `B${string}` + | `C${string}` + | `D${string}` + | `E${string}` + | `F${string}` + | `G${string}` + | `H${string}` + | `I${string}` + | `J${string}` + | `K${string}` + | `L${string}` + | `M${string}` + | `N${string}` + | `O${string}` + | `P${string}` + | `Q${string}` + | `R${string}` + | `S${string}` + | `T${string}` + | `U${string}` + | `V${string}` + | `W${string}` + | `X${string}` + | `Y${string}` + | `Z${string}`; + + export interface XMLDoc { + /** + * The doc's root node. + */ + readonly $root: XMLNode; + + /** + * The doc's root by its name or undefined. + */ + readonly [rootTagName: XMLTagName]: XMLNode | undefined; + } + + export interface XMLNode { + /** + * node.$attr$xxx - the node's attribute value of "xxx". + */ + readonly [key: `$attr$${string}`]: string | undefined; + + /** + * node.$attrs - an XMLAttr wrapper object for all the attributes + * of the node. + */ + readonly $attrs: XMLAttr; + + /** + * node.$tag$xxx - the node's first child tag named "xxx". + */ + readonly [key: `$tag$${string}`]: XMLNode | undefined; + + /** + * node.$tags$xxx - all children tags named "xxx" of the node. + */ + readonly [key: `$tags$${string}`]: XMLNode[] | undefined; + + /** + * node.$name - the name of the node. + */ + readonly $name: string; + + /** + * node.$ns - the namespace of the node. + */ + readonly $ns: string; + + /** + * node.$parent - the parent node of the current node. + */ + readonly $parent: string; + + /** + * node.$text - the content of the node. + */ + readonly $text: string; + + /** + * node.$tags - all the node's children tags. + */ + readonly $tags: XMLNode[] | undefined; + + /** + * node.xxx is the same as node.$tag$xxx. + */ + readonly [key: XMLTagName]: XMLNode | undefined; + } + + export interface XMLAttr { + /** + * attr.xxx is the attribute value of "xxx". + */ + readonly [key: string]: string | undefined; + } + + interface Xml { + /** + * Parses src buffer for an XML document and returns a wrapper object. + * + * @param src a string or a buffer with an XML document. + * @return A XMLDoc wrapper object representing the parsed XML document. + */ + parse(src: NjsStringLike): XMLDoc; + + /** + * Canonicalizes root_node and its children according to + * https://www.w3.org/TR/xml-exc-c14n/. + * + * @param root - XMLDoc or XMLNode. + * @param excluding_node - allows to omit from the output a part of the + * document corresponding to the excluding_node and its children. + * @param withComments - a boolean (false by default). When withComments + * is true canonicalization corresponds to + * http://www.w3.org/2001/10/xml-exc-c14n#WithComments. + * @param prefix_list - an optional string with a space separated namespace + * prefixes for namespaces that should also be included into the output. + * @return Buffer object containing canonicalized output. + */ + exclusiveC14n(root: XMLDoc | XMLNode, excluding_node?: XMLNode | null | undefined, + withComments?: boolean, prefix_list?: string): Buffer; + } + + const xml: Xml; + + export default xml; +} diff -r 948b167202b2 -r 502a63c67947 ts/package.json --- a/ts/package.json Mon Jan 30 18:19:16 2023 -0800 +++ b/ts/package.json Wed Feb 01 19:04:39 2023 -0800 @@ -26,6 +26,6 @@ }, "homepage": "https://nginx.org/en/docs/njs/", "devDependencies": { - "typescript": "^4.0.3" + "typescript": "^4.9.5" } } From xeioex at nginx.com Thu Feb 2 03:07:58 2023 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Thu, 02 Feb 2023 03:07:58 +0000 Subject: [njs] Properly fixed memory leak after 948b167202b2. Message-ID: details: https://hg.nginx.org/njs/rev/0f7aedb08056 branches: changeset: 2036:0f7aedb08056 user: Dmitry Volyntsev date: Wed Feb 01 19:06:22 2023 -0800 description: Properly fixed memory leak after 948b167202b2. This prevents double-free introduced in 948b167202b2. diffstat: external/njs_xml_module.c | 43 +++++++++++++++++++++---------------------- 1 files changed, 21 insertions(+), 22 deletions(-) diffs (122 lines): diff -r 502a63c67947 -r 0f7aedb08056 external/njs_xml_module.c --- a/external/njs_xml_module.c Wed Feb 01 19:04:39 2023 -0800 +++ b/external/njs_xml_module.c Wed Feb 01 19:06:22 2023 -0800 @@ -990,14 +990,6 @@ njs_xml_ext_canonicalization(njs_vm_t *v } } - njs_chb_init(&chain, njs_vm_memory_pool(vm)); - - buf = xmlOutputBufferCreateIO(njs_xml_buf_write_cb, NULL, &chain, NULL); - if (njs_slow_path(buf == NULL)) { - njs_vm_error(vm, "xmlOutputBufferCreateIO() failed"); - return NJS_ERROR; - } - comments = njs_value_bool(njs_arg(args, nargs, 3)); excluding = njs_arg(args, nargs, 2); @@ -1006,19 +998,19 @@ njs_xml_ext_canonicalization(njs_vm_t *v node = njs_vm_external(vm, njs_xml_node_proto_id, excluding); if (njs_slow_path(node == NULL)) { njs_vm_error(vm, "\"excluding\" argument is not a XMLNode object"); - goto error; + return NJS_ERROR; } nset = njs_xml_nset_create(vm, current->doc, current, XML_NSET_TREE_NO_COMMENTS); if (njs_slow_path(nset == NULL)) { - goto error; + return NJS_ERROR; } children = njs_xml_nset_create(vm, node->doc, node, XML_NSET_TREE_INVERT); if (njs_slow_path(children == NULL)) { - goto error; + return NJS_ERROR; } nset = njs_xml_nset_add(nset, children); @@ -1028,7 +1020,7 @@ njs_xml_ext_canonicalization(njs_vm_t *v comments ? XML_NSET_TREE : XML_NSET_TREE_NO_COMMENTS); if (njs_slow_path(nset == NULL)) { - goto error; + return NJS_ERROR; } } @@ -1038,46 +1030,53 @@ njs_xml_ext_canonicalization(njs_vm_t *v if (!njs_value_is_null_or_undefined(prefixes)) { if (!njs_value_is_string(prefixes)) { njs_vm_error(vm, "\"prefixes\" argument is not a string"); - goto error; + return NJS_ERROR; } ret = njs_vm_value_string(vm, &string, prefixes); if (njs_slow_path(ret != NJS_OK)) { - goto error; + return NJS_ERROR; } prefix_list = njs_xml_parse_ns_list(vm, &string); if (njs_slow_path(prefix_list == NULL)) { - goto error; + return NJS_ERROR; } } + njs_chb_init(&chain, njs_vm_memory_pool(vm)); + + buf = xmlOutputBufferCreateIO(njs_xml_buf_write_cb, NULL, &chain, NULL); + if (njs_slow_path(buf == NULL)) { + njs_vm_error(vm, "xmlOutputBufferCreateIO() failed"); + return NJS_ERROR; + } + ret = xmlC14NExecute(current->doc, njs_xml_c14n_visibility_cb, nset, exclusive ? XML_C14N_EXCLUSIVE_1_0 : XML_C14N_1_0, prefix_list, comments, buf); - (void) xmlOutputBufferClose(buf); - if (njs_slow_path(ret < 0)) { njs_vm_error(vm, "xmlC14NExecute() failed"); + ret = NJS_ERROR; goto error; } size = njs_chb_size(&chain); if (njs_slow_path(size < 0)) { njs_vm_memory_error(vm); + ret = NJS_ERROR; goto error; } ret = njs_chb_join(&chain, &data); if (njs_slow_path(ret != NJS_OK)) { + ret = NJS_ERROR; goto error; } - njs_chb_destroy(&chain); - - return njs_vm_value_buffer_set(vm, njs_vm_retval(vm), data.start, - data.length); + ret = njs_vm_value_buffer_set(vm, njs_vm_retval(vm), data.start, + data.length); error: @@ -1085,7 +1084,7 @@ error: njs_chb_destroy(&chain); - return NJS_ERROR; + return ret; } From pluknet at nginx.com Thu Feb 2 09:05:19 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Thu, 2 Feb 2023 13:05:19 +0400 Subject: [PATCH] Gzip static: ranges support (ticket #2349) In-Reply-To: References: <262ABC58-7AB2-4B53-90D4-648464DE4F44@nginx.com> Message-ID: <33B9DA06-D8F7-4013-B696-5DB776D5DDAE@nginx.com> > On 29 Jan 2023, at 00:03, Maxim Dounin wrote: > > Hello! > > On Wed, Jan 25, 2023 at 01:50:56PM +0400, Sergey Kandaurov wrote: > >>> On 24 Jan 2023, at 06:19, Maxim Dounin wrote: >>> >>>>> [..] >>> >>> diff --git a/src/http/modules/ngx_http_flv_module.c b/src/http/modules/ngx_http_flv_module.c >>> --- a/src/http/modules/ngx_http_flv_module.c >>> +++ b/src/http/modules/ngx_http_flv_module.c >>> @@ -235,6 +235,7 @@ ngx_http_flv_handler(ngx_http_request_t >>> b->in_file = b->file_last ? 1: 0; >> >> btw, there are old style issues originated in the static module >> (it was fixed in gzip_static when it was copied from there) > > This is actually an (old) style, which was previously used in many > places. Now it remains mostly in multi-line forms of the > conditional operator, e.g.: > > r->loc_conf = (node->exact) ? node->exact->loc_conf: > node->inclusive->loc_conf; > > Single-line form is indeed mostly unused now, and probably it's > time to remove remaining occurrences. Added a patch for this. > > [...] > >>> @@ -238,10 +238,6 @@ ngx_http_static_handler(ngx_http_request >>> return NGX_HTTP_INTERNAL_SERVER_ERROR; >>> } >>> >>> - if (r != r->main && of.size == 0) { >>> - return ngx_http_send_header(r); >>> - } >>> - >>> r->allow_ranges = 1; >>> >>> /* we need to allocate all before the header would be sent */ >>> @@ -268,6 +264,7 @@ ngx_http_static_handler(ngx_http_request >>> b->in_file = b->file_last ? 1: 0; >>> b->last_buf = (r == r->main) ? 1: 0; >>> b->last_in_chain = 1; >>> + b->sync = (r == r->main || b->file_last) ? 0 : 1; >>> >>> b->file->fd = of.fd; >>> b->file->name = path; >>> >> >> In addition to the static module change, any reason not to include >> ngx_http_cache_send()? It has a similar behaviour to skip cached >> empty body if served in subrequests. Yet another similar place is >> subrequests processed with rewrite such as "return 204;". > > Indeed, thanks for catching, missed these two. > > Also, looking into this places I tend to think it's better to use > > b->sync = (b->last_buf || b->in_file) ? 0 : 1; > > instead, which is going to be simpler and more universal across > all uses. I agree. > > # HG changeset patch > # User Maxim Dounin > # Date 1674872423 -10800 > # Sat Jan 28 05:20:23 2023 +0300 > # Node ID 1c3b78d7cdc9e7059e90aa1670d58fd37927a1a2 > # Parent 106328a70f4ecb32f828d33e5cd66c861e455f92 > Style. > > diff --git a/src/event/ngx_event_connectex.c b/src/event/ngx_event_connectex.c > --- a/src/event/ngx_event_connectex.c > +++ b/src/event/ngx_event_connectex.c > @@ -127,8 +127,8 @@ void ngx_iocp_wait_events(int main) > conn[0] = NULL; > > for ( ;; ) { > - offset = (nevents == WSA_MAXIMUM_WAIT_EVENTS + 1) ? 1: 0; > - timeout = (nevents == 1 && !first) ? 60000: INFINITE; > + offset = (nevents == WSA_MAXIMUM_WAIT_EVENTS + 1) ? 1 : 0; > + timeout = (nevents == 1 && !first) ? 60000 : INFINITE; > > n = WSAWaitForMultipleEvents(nevents - offset, events[offset], > 0, timeout, 0); > diff --git a/src/http/modules/ngx_http_flv_module.c b/src/http/modules/ngx_http_flv_module.c > --- a/src/http/modules/ngx_http_flv_module.c > +++ b/src/http/modules/ngx_http_flv_module.c > @@ -232,7 +232,7 @@ ngx_http_flv_handler(ngx_http_request_t > b->file_pos = start; > b->file_last = of.size; > > - b->in_file = b->file_last ? 1: 0; > + b->in_file = b->file_last ? 1 : 0; > b->last_buf = (r == r->main) ? 1 : 0; > b->last_in_chain = 1; > > diff --git a/src/http/modules/ngx_http_static_module.c b/src/http/modules/ngx_http_static_module.c > --- a/src/http/modules/ngx_http_static_module.c > +++ b/src/http/modules/ngx_http_static_module.c > @@ -265,8 +265,8 @@ ngx_http_static_handler(ngx_http_request > b->file_pos = 0; > b->file_last = of.size; > > - b->in_file = b->file_last ? 1: 0; > - b->last_buf = (r == r->main) ? 1: 0; > + b->in_file = b->file_last ? 1 : 0; > + b->last_buf = (r == r->main) ? 1 : 0; > b->last_in_chain = 1; > > b->file->fd = of.fd; > diff --git a/src/http/ngx_http_file_cache.c b/src/http/ngx_http_file_cache.c > --- a/src/http/ngx_http_file_cache.c > +++ b/src/http/ngx_http_file_cache.c > @@ -1600,8 +1600,8 @@ ngx_http_cache_send(ngx_http_request_t * > b->file_pos = c->body_start; > b->file_last = c->length; > > - b->in_file = (c->length - c->body_start) ? 1: 0; > - b->last_buf = (r == r->main) ? 1: 0; > + b->in_file = (c->length - c->body_start) ? 1 : 0; > + b->last_buf = (r == r->main) ? 1 : 0; > b->last_in_chain = 1; > > b->file->fd = c->file.fd; > # HG changeset patch > # User Maxim Dounin > # Date 1674872613 -10800 > # Sat Jan 28 05:23:33 2023 +0300 > # Node ID f5515e727656b2cc529b26a0fe6b4496c62e32b2 > # Parent 1c3b78d7cdc9e7059e90aa1670d58fd37927a1a2 > Fixed "zero size buf" alerts with subrequests. > > Since 4611:2b6cb7528409 responses from the gzip static, flv, and mp4 modules > can be used with subrequests, though empty files were not properly handled. > Empty gzipped, flv, and mp4 files thus resulted in "zero size buf in output" > alerts. While valid corresponding files are not expected to be empty, such > files shouldn't result in alerts. > > Fix is to set b->sync on such empty subrequest responses, similarly to what > ngx_http_send_special() does. > > Additionally, the static module, the ngx_http_send_response() function, and > file cache are modified to do the same instead of not sending the response > body at all in such cases, since not sending the response body at all is > believed to be at least questionable, and might break various filters > which do not expect such behaviour. > > diff --git a/src/http/modules/ngx_http_flv_module.c b/src/http/modules/ngx_http_flv_module.c > --- a/src/http/modules/ngx_http_flv_module.c > +++ b/src/http/modules/ngx_http_flv_module.c > @@ -235,6 +235,7 @@ ngx_http_flv_handler(ngx_http_request_t > b->in_file = b->file_last ? 1 : 0; > b->last_buf = (r == r->main) ? 1 : 0; > b->last_in_chain = 1; > + b->sync = (b->last_buf || b->in_file) ? 0 : 1; > > b->file->fd = of.fd; > b->file->name = path; > diff --git a/src/http/modules/ngx_http_gzip_static_module.c b/src/http/modules/ngx_http_gzip_static_module.c > --- a/src/http/modules/ngx_http_gzip_static_module.c > +++ b/src/http/modules/ngx_http_gzip_static_module.c > @@ -273,6 +273,7 @@ ngx_http_gzip_static_handler(ngx_http_re > b->in_file = b->file_last ? 1 : 0; > b->last_buf = (r == r->main) ? 1 : 0; > b->last_in_chain = 1; > + b->sync = (b->last_buf || b->in_file) ? 0 : 1; > > b->file->fd = of.fd; > b->file->name = path; > diff --git a/src/http/modules/ngx_http_mp4_module.c b/src/http/modules/ngx_http_mp4_module.c > --- a/src/http/modules/ngx_http_mp4_module.c > +++ b/src/http/modules/ngx_http_mp4_module.c > @@ -714,6 +714,7 @@ ngx_http_mp4_handler(ngx_http_request_t > b->in_file = b->file_last ? 1 : 0; > b->last_buf = (r == r->main) ? 1 : 0; > b->last_in_chain = 1; > + b->sync = (b->last_buf || b->in_file) ? 0 : 1; > > b->file->fd = of.fd; > b->file->name = path; > diff --git a/src/http/modules/ngx_http_static_module.c b/src/http/modules/ngx_http_static_module.c > --- a/src/http/modules/ngx_http_static_module.c > +++ b/src/http/modules/ngx_http_static_module.c > @@ -238,10 +238,6 @@ ngx_http_static_handler(ngx_http_request > return NGX_HTTP_INTERNAL_SERVER_ERROR; > } > > - if (r != r->main && of.size == 0) { > - return ngx_http_send_header(r); > - } > - > r->allow_ranges = 1; > > /* we need to allocate all before the header would be sent */ > @@ -268,6 +264,7 @@ ngx_http_static_handler(ngx_http_request > b->in_file = b->file_last ? 1 : 0; > b->last_buf = (r == r->main) ? 1 : 0; > b->last_in_chain = 1; > + b->sync = (b->last_buf || b->in_file) ? 0 : 1; > > b->file->fd = of.fd; > b->file->name = path; > diff --git a/src/http/ngx_http_core_module.c b/src/http/ngx_http_core_module.c > --- a/src/http/ngx_http_core_module.c > +++ b/src/http/ngx_http_core_module.c > @@ -1803,10 +1803,6 @@ ngx_http_send_response(ngx_http_request_ > } > } > > - if (r != r->main && val.len == 0) { > - return ngx_http_send_header(r); > - } > - > b = ngx_calloc_buf(r->pool); > if (b == NULL) { > return NGX_HTTP_INTERNAL_SERVER_ERROR; > @@ -1817,6 +1813,7 @@ ngx_http_send_response(ngx_http_request_ > b->memory = val.len ? 1 : 0; > b->last_buf = (r == r->main) ? 1 : 0; > b->last_in_chain = 1; > + b->sync = (b->last_buf || b->memory) ? 0 : 1; > > out.buf = b; > out.next = NULL; > diff --git a/src/http/ngx_http_file_cache.c b/src/http/ngx_http_file_cache.c > --- a/src/http/ngx_http_file_cache.c > +++ b/src/http/ngx_http_file_cache.c > @@ -1575,10 +1575,6 @@ ngx_http_cache_send(ngx_http_request_t * > ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, > "http file cache send: %s", c->file.name.data); > > - if (r != r->main && c->length - c->body_start == 0) { > - return ngx_http_send_header(r); > - } > - > /* we need to allocate all before the header would be sent */ > > b = ngx_calloc_buf(r->pool); > @@ -1603,6 +1599,7 @@ ngx_http_cache_send(ngx_http_request_t * > b->in_file = (c->length - c->body_start) ? 1 : 0; > b->last_buf = (r == r->main) ? 1 : 0; > b->last_in_chain = 1; > + b->sync = (b->last_buf || b->in_file) ? 0 : 1; > > b->file->fd = c->file.fd; > b->file->name = c->file.name; > Both changes look good to me, please commit. -- Sergey Kandaurov From pluknet at nginx.com Thu Feb 2 14:53:20 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Thu, 2 Feb 2023 18:53:20 +0400 Subject: [PATCH] HTTP: trigger lingering close when keepalive connection will be closed In-Reply-To: References: <699749D2-D96F-47D8-85DA-74E7F97682FF@gmail.com> <8D59E87C-2DC3-43AB-9067-019CA9ACE5C4@gmail.com> Message-ID: <67831923-6BAE-4EF0-8457-46128FEB9CF5@nginx.com> > On 27 Jan 2023, at 08:01, Maxim Dounin wrote: > > Hello! > > [..] > Overall, after looking into logs and tcpdump you've provided I > tend to think that the only working fix would be to introduce > c->pipeline flag, and force lingering close if there were any > pipelined requests on the connection. > > Below is the patch which implements this approach. Review and > testing appreciated. It can be used either separately or with the > previously provided patch to use posted next events. > > # HG changeset patch > # User Maxim Dounin > # Date 1674790916 -10800 > # Fri Jan 27 06:41:56 2023 +0300 > # Node ID 784d0fa0b5a0796561642a5a64dc4e9e07592852 > # Parent 4eb1383f6432b034630e6de18739b817f6565c8c > Lingering close for connections with pipelined requests. > > This is expected to help with clients using pipelining with some constant > depth, such as apt[1][2]. > > When downloading many resources, apt uses pipelining with some constant > depth, a number of requests in flight[1][2]. This essentially means that Are links repeated on purpose? > after receiving a response it sends an additional request to the server, > and this can result in requests arriving to the server at any time. Further, > additional requests are sent one-by-one, and can be easily seen as such > (neither as pipelined, nor followed by pipelined requests). > > The only safe approach to close such connections (for example, when > keepalive_requests is reached) is with lingering. To do so, now nginx > monitors if pipelining was used on the connection, and if it was, closes > the connection with lingering. > > [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=973861#10 > [2] https://mailman.nginx.org/pipermail/nginx-devel/2023-January/ZA2SP5SJU55LHEBCJMFDB2AZVELRLTHI.html > > diff --git a/src/core/ngx_connection.h b/src/core/ngx_connection.h > --- a/src/core/ngx_connection.h > +++ b/src/core/ngx_connection.h > @@ -172,6 +172,7 @@ struct ngx_connection_s { > unsigned timedout:1; > unsigned error:1; > unsigned destroyed:1; > + unsigned pipeline:1; > > unsigned idle:1; > unsigned reusable:1; > diff --git a/src/http/ngx_http_request.c b/src/http/ngx_http_request.c > --- a/src/http/ngx_http_request.c > +++ b/src/http/ngx_http_request.c > @@ -2753,7 +2753,8 @@ ngx_http_finalize_connection(ngx_http_re > || (clcf->lingering_close == NGX_HTTP_LINGERING_ON > && (r->lingering_close > || r->header_in->pos < r->header_in->last > - || r->connection->read->ready))) > + || r->connection->read->ready > + || r->connection->pipeline))) > { > ngx_http_set_lingering_close(r->connection); > return; > @@ -3123,6 +3124,7 @@ ngx_http_set_keepalive(ngx_http_request_ > > c->sent = 0; > c->destroyed = 0; > + c->pipeline = 1; > > if (rev->timer_set) { > ngx_del_timer(rev); > Looks good to me. Further, as request pipelining implementation status is quite limited (often disabled by default or unimplemented, notably in browsers), I believe the change is fine to not cause noticeable legit resource usage. Still, old behaviour can be reverted with "lingering_close off;". -- Sergey Kandaurov From v.zhestikov at f5.com Thu Feb 2 18:04:10 2023 From: v.zhestikov at f5.com (Vadim Zhestikov) Date: Thu, 02 Feb 2023 18:04:10 +0000 Subject: [njs] Fixed RegExp.prototype[@@replace](). Message-ID: details: https://hg.nginx.org/njs/rev/286675dcfbc5 branches: changeset: 2037:286675dcfbc5 user: Vadim Zhestikov date: Thu Feb 02 10:01:26 2023 -0800 description: Fixed RegExp.prototype[@@replace](). Previously, when RegExpExec() returned a fast-array with gaps String.prototype.replace() might return erroneous exception TypeError: Cannot convert object to primitive value. diffstat: src/njs_regexp.c | 4 ++++ src/test/njs_unit_test.c | 42 ++++++++++++++++++++++++++++++++++++++---- 2 files changed, 42 insertions(+), 4 deletions(-) diffs (79 lines): diff -r 0f7aedb08056 -r 286675dcfbc5 src/njs_regexp.c --- a/src/njs_regexp.c Wed Feb 01 19:06:22 2023 -0800 +++ b/src/njs_regexp.c Thu Feb 02 10:01:26 2023 -0800 @@ -1288,6 +1288,10 @@ njs_regexp_prototype_symbol_replace(njs_ if (njs_fast_path(njs_is_fast_array(r) && njs_array_len(r) != 0)) { value = njs_array_start(r)[0]; + if (!njs_is_valid(&value)) { + njs_set_undefined(&value); + } + } else { ret = njs_value_property_i64(vm, r, 0, &value); if (njs_slow_path(ret == NJS_ERROR)) { diff -r 0f7aedb08056 -r 286675dcfbc5 src/test/njs_unit_test.c --- a/src/test/njs_unit_test.c Wed Feb 01 19:06:22 2023 -0800 +++ b/src/test/njs_unit_test.c Thu Feb 02 10:01:26 2023 -0800 @@ -9063,8 +9063,19 @@ static njs_unit_test_t njs_test[] = "re.exec = function () {" " return a;" "};" - "var r = 'any_string'.replace(re);"), - njs_str("undefined") }, + "'any_string'.replace(re)"), + njs_str("undefinedg") }, + + { njs_str("var cnt = 0;" + "var a = [];" + "a[2] = '';" + "var re = /any_regexp/g;" + "re.exec = function () {" + " if (cnt++ > 1) return null;" + " return a;" + "};" + "'any_string'.replace(re)"), + njs_str("undefinedg") }, { njs_str("var a = [];" "a[2] = {toString() {a[2**20] = 1; return 'X';}}; " @@ -9077,14 +9088,37 @@ static njs_unit_test_t njs_test[] = "'abc'.replace(re, '@$1|$2|$3|$4|$99|$100|@')"), njs_str("@|X||Y|Z|0|@") }, + { njs_str("var cnt = 0;" + "var a = [];" + "a[2] = {toString() {a[2**20] = 1; return 'X';}}; " + "a[4] = 'Y';" + "a[99] = 'Z';" + "a[100] = '*';" + "a[200] = '!';" + "var re = /b/g;" + "re.exec = () => {if (cnt++ > 1) return null; return a};" + "'abc'.replace(re, '@$1|$2|$3|$4|$99|$100|@')"), + njs_str("@|X||Y|Z|0|@") }, + { njs_str("var a = [];" "Object.defineProperty(a, 32768, {});" "var re = /any_regexp/;" "re.exec = function () {" " return a;" "};" - "var r = 'any_string'.replace(re);"), - njs_str("undefined") }, + "'any_string'.replace(re)"), + njs_str("undefinedg") }, + + { njs_str("var cnt = 0;" + "var a = [];" + "Object.defineProperty(a, 32768, {});" + "var re = /any_regexp/g;" + "re.exec = function () {" + " if (cnt++ > 1) return null;" + " return a;" + "};" + "'any_string'.replace(re)"), + njs_str("undefinedg") }, { njs_str("/=/"), njs_str("/=/") }, From pluknet at nginx.com Thu Feb 2 18:35:22 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Thu, 2 Feb 2023 22:35:22 +0400 Subject: [PATCH] QUIC: OpenSSL compatibility layer In-Reply-To: <20230109111316.yfqj24unoubibihz@N00W24XTQX> References: <64a365dcb52503e91d91.1671606452@arut-laptop> <20230109111316.yfqj24unoubibihz@N00W24XTQX> Message-ID: <20230202183522.o4w5xbjvegbchts4@Y9MQ9X2QVV> On Mon, Jan 09, 2023 at 03:13:16PM +0400, Roman Arutyunyan wrote: > # HG changeset patch > # User Roman Arutyunyan > # Date 1673262402 -14400 > # Mon Jan 09 15:06:42 2023 +0400 > # Branch quic > # Node ID 4e5dfe13c84fe50bec639f1b7dcc81604378a42b > # Parent aaa2a3831eefe4315dfb8a9be7178c79ff67f163 > QUIC: OpenSSL compatibility layer. > > The change allows to compile QUIC with OpenSSL which lacks BoringSSL QUIC API. > > This implementation does not support 0-RTT. > > diff --git a/README b/README > --- a/README > +++ b/README > @@ -53,7 +53,7 @@ 1. Introduction > > 2. Installing > > - A library that provides QUIC support is required to build nginx, there > + A library that provides QUIC support is recommended to build nginx, there > are several of those available on the market: > + BoringSSL [4] > + LibreSSL [5] > @@ -85,6 +85,10 @@ 2. Installing > --with-cc-opt="-I../libressl/build/include" \ > --with-ld-opt="-L../libressl/build/lib" > > + Alternatively, nginx can be configured with OpenSSL compatibility > + layer, which emulates BoringSSL QUIC API for OpenSSL. This mode is > + enabled by default if native QUIC support is not detected. > + > When configuring nginx, it's possible to enable QUIC and HTTP/3 > using the following new configuration options: > > diff --git a/auto/lib/openssl/conf b/auto/lib/openssl/conf > --- a/auto/lib/openssl/conf > +++ b/auto/lib/openssl/conf > @@ -10,6 +10,7 @@ if [ $OPENSSL != NONE ]; then > > if [ $USE_OPENSSL_QUIC = YES ]; then > have=NGX_QUIC . auto/have > + have=NGX_QUIC_OPENSSL_COMPAT . auto/have This won't build with QuicTLS sources specified in --with-openssl, due to type/function redefinitions. The patch to address this: diff --git a/src/event/quic/ngx_event_quic_openssl_compat.h b/src/event/quic/ngx_event_quic_openssl_compat.h --- a/src/event/quic/ngx_event_quic_openssl_compat.h +++ b/src/event/quic/ngx_event_quic_openssl_compat.h @@ -7,6 +7,10 @@ #ifndef _NGX_EVENT_QUIC_OPENSSL_COMPAT_H_INCLUDED_ #define _NGX_EVENT_QUIC_OPENSSL_COMPAT_H_INCLUDED_ +#ifdef TLSEXT_TYPE_quic_transport_parameters +#undef NGX_QUIC_OPENSSL_COMPAT +#else + #include #include @@ -48,4 +52,6 @@ void SSL_get_peer_quic_transport_params( const uint8_t **out_params, size_t *out_params_len); +#endif /* TLSEXT_TYPE_quic_transport_parameters */ + #endif /* _NGX_EVENT_QUIC_OPENSSL_COMPAT_H_INCLUDED_ */ > fi > > case "$CC" in > @@ -124,6 +125,45 @@ else > CORE_INCS="$CORE_INCS $ngx_feature_path" > CORE_LIBS="$CORE_LIBS $ngx_feature_libs" > OPENSSL=YES > + > + if [ $USE_OPENSSL_QUIC = YES ]; then > + > + ngx_feature="OpenSSL QUIC support" > + ngx_feature_name="NGX_OPENSSL_QUIC" > + ngx_feature_run=no > + ngx_feature_incs="#include " > + ngx_feature_test="SSL_set_quic_method(NULL, NULL)" > + . auto/feature > + > + if [ $ngx_found = no ]; then bad indent > + > + ngx_feature="OpenSSL QUIC compatibility" > + ngx_feature_name="NGX_QUIC_OPENSSL_COMPAT" > + ngx_feature_run=no > + ngx_feature_incs="#include " > + ngx_feature_test=" > + SSL_set_max_early_data(NULL, TLS1_3_VERSION); While practicaly useful to check for TLSv1.3-aware library, this abuses SSL_set_max_early_data() API. > + SSL_CTX_set_msg_callback(NULL, NULL); > + SSL_CTX_set_keylog_callback(NULL, NULL); > + SSL_CTX_add_custom_ext(NULL, 0, 0, NULL, NULL, > + NULL, NULL, NULL)" Just using SSL_CTX_add_custom_ext() seems to be sufficient to ensure this is OpenSSL version 1.1.1+. > + . auto/feature > + fi > + > + if [ $ngx_found = no ]; then > +cat << END > + > +$0: error: certain modules require OpenSSL QUIC support. > +You can either do not enable the modules, or install the OpenSSL library with > +QUIC support into the system, or build the OpenSSL library with QUIC support > +statically from the source with nginx by using --with-openssl= option. > + > +END > + exit 1 > + fi > + > + have=NGX_QUIC . auto/have > + fi > fi > fi > > @@ -139,29 +179,4 @@ with nginx by using --with-openssl= END > exit 1 > fi > - > - if [ $USE_OPENSSL_QUIC = YES ]; then > - > - ngx_feature="OpenSSL QUIC support" > - ngx_feature_name="NGX_QUIC" > - ngx_feature_run=no > - ngx_feature_incs="#include " > - ngx_feature_path= > - ngx_feature_libs="-lssl -lcrypto $NGX_LIBDL $NGX_LIBPTHREAD" > - ngx_feature_test="SSL_set_quic_method(NULL, NULL)" > - . auto/feature > - > - if [ $ngx_found = no ]; then > - > -cat << END > - > -$0: error: certain modules require OpenSSL QUIC support. > -You can either do not enable the modules, or install the OpenSSL library with > -QUIC support into the system, or build the OpenSSL library with QUIC support > -statically from the source with nginx by using --with-openssl= option. > - > -END > - exit 1 > - fi > - fi > fi > diff --git a/auto/modules b/auto/modules > --- a/auto/modules > +++ b/auto/modules > @@ -1342,7 +1342,8 @@ if [ $USE_OPENSSL_QUIC = YES ]; then > src/event/quic/ngx_event_quic_tokens.h \ > src/event/quic/ngx_event_quic_ack.h \ > src/event/quic/ngx_event_quic_output.h \ > - src/event/quic/ngx_event_quic_socket.h" > + src/event/quic/ngx_event_quic_socket.h \ > + src/event/quic/ngx_event_quic_openssl_compat.h" > ngx_module_srcs="src/event/quic/ngx_event_quic.c \ > src/event/quic/ngx_event_quic_udp.c \ > src/event/quic/ngx_event_quic_transport.c \ > @@ -1355,7 +1356,8 @@ if [ $USE_OPENSSL_QUIC = YES ]; then > src/event/quic/ngx_event_quic_tokens.c \ > src/event/quic/ngx_event_quic_ack.c \ > src/event/quic/ngx_event_quic_output.c \ > - src/event/quic/ngx_event_quic_socket.c" > + src/event/quic/ngx_event_quic_socket.c \ > + src/event/quic/ngx_event_quic_openssl_compat.c" > > ngx_module_libs= > ngx_module_link=YES > diff --git a/src/event/ngx_event_openssl.c b/src/event/ngx_event_openssl.c > --- a/src/event/ngx_event_openssl.c > +++ b/src/event/ngx_event_openssl.c > @@ -9,6 +9,10 @@ > #include > #include > > +#if (NGX_QUIC_OPENSSL_COMPAT) > +#include > +#endif > + > > #define NGX_SSL_PASSWORD_BUFFER_SIZE 4096 > > @@ -392,6 +396,10 @@ ngx_ssl_create(ngx_ssl_t *ssl, ngx_uint_ > > SSL_CTX_set_info_callback(ssl->ctx, ngx_ssl_info_callback); > > +#if (NGX_QUIC_OPENSSL_COMPAT) > + ngx_quic_compat_init(ssl->ctx); > +#endif > + This enables compatibility unconditionally for all TLS versions and consumers, including such modules as the mail module and connections to upstream. While SSL_CTX_add_custom_ext() to enable the transport parameters extension in TLS < 1.3 seems to be harmless, and enabling the keylog callback just leads to redundant function calls, I propose to move this close to consumers, e.g.: diff --git a/src/http/modules/ngx_http_ssl_module.c b/src/http/modules/ngx_http_ssl_module.c --- a/src/http/modules/ngx_http_ssl_module.c +++ b/src/http/modules/ngx_http_ssl_module.c @@ -9,6 +9,10 @@ #include #include +#if (NGX_QUIC_OPENSSL_COMPAT) +#include +#endif + typedef ngx_int_t (*ngx_ssl_variable_handler_pt)(ngx_connection_t *c, ngx_pool_t *pool, ngx_str_t *s); @@ -1317,16 +1321,19 @@ ngx_http_ssl_init(ngx_conf_t *cf) continue; } + cscf = addr[a].default_server; + sscf = cscf->ctx->srv_conf[ngx_http_ssl_module.ctx_index]; + if (addr[a].opt.http3) { name = "http3"; +#if (NGX_QUIC_OPENSSL_COMPAT) + ngx_quic_compat_init(sscf->ssl.ctx); +#endif } else { name = "ssl"; } - cscf = addr[a].default_server; - sscf = cscf->ctx->srv_conf[ngx_http_ssl_module.ctx_index]; - if (sscf->certificates) { if (addr[a].opt.http3 && !(sscf->protocols & NGX_SSL_TLSv1_3)) { (with similar change in the stream module) So we could remove SOCK_DGRAM checks. > return NGX_OK; > } > > diff --git a/src/event/quic/ngx_event_quic_connection.h b/src/event/quic/ngx_event_quic_connection.h > --- a/src/event/quic/ngx_event_quic_connection.h > +++ b/src/event/quic/ngx_event_quic_connection.h > @@ -24,6 +24,9 @@ typedef struct ngx_quic_send_ctx_s ng > typedef struct ngx_quic_socket_s ngx_quic_socket_t; > typedef struct ngx_quic_path_s ngx_quic_path_t; > typedef struct ngx_quic_keys_s ngx_quic_keys_t; > +#if (NGX_QUIC_OPENSSL_COMPAT) > +typedef struct ngx_quic_compat_s ngx_quic_compat_t; > +#endif > > #include > #include > @@ -36,6 +39,9 @@ typedef struct ngx_quic_keys_s ng > #include > #include > #include > +#if (NGX_QUIC_OPENSSL_COMPAT) > +#include > +#endif > > > /* RFC 9002, 6.2.2. Handshakes and New Paths: kInitialRtt */ > @@ -236,6 +242,10 @@ struct ngx_quic_connection_s { > ngx_uint_t nshadowbufs; > #endif > > +#if (NGX_QUIC_OPENSSL_COMPAT) > + ngx_quic_compat_t *compat; > +#endif > + > ngx_quic_streams_t streams; > ngx_quic_congestion_t congestion; > > diff --git a/src/event/quic/ngx_event_quic_openssl_compat.c b/src/event/quic/ngx_event_quic_openssl_compat.c > new file mode 100644 > --- /dev/null > +++ b/src/event/quic/ngx_event_quic_openssl_compat.c > @@ -0,0 +1,656 @@ > + > +/* > + * Copyright (C) Nginx, Inc. > + */ > + > + > +#include > +#include > +#include > +#include > + > + > +#if (NGX_QUIC_OPENSSL_COMPAT) > + > +#define NGX_QUIC_COMPAT_RECORD_SIZE 1024 > + > +#define NGX_QUIC_COMPAT_SSL_TP_EXT 0x39 For the record: SSL library sources have the TLSEXT_TYPE_quic_transport_parameters extension value written in the decimal form, so could we. An exception is QuicTLS, which seems to derive hex form from draft days. > + > +#define NGX_QUIC_COMPAT_CLIENT_EARLY "CLIENT_EARLY_TRAFFIC_SECRET" > +#define NGX_QUIC_COMPAT_CLIENT_HANDSHAKE "CLIENT_HANDSHAKE_TRAFFIC_SECRET" > +#define NGX_QUIC_COMPAT_SERVER_HANDSHAKE "SERVER_HANDSHAKE_TRAFFIC_SECRET" > +#define NGX_QUIC_COMPAT_CLIENT_APPLICATION "CLIENT_TRAFFIC_SECRET_0" > +#define NGX_QUIC_COMPAT_SERVER_APPLICATION "SERVER_TRAFFIC_SECRET_0" > + > + > +typedef struct { > + ngx_quic_secret_t secret; > + ngx_uint_t cipher; > +} ngx_quic_compat_keys_t; > + > + > +typedef struct { > + ngx_log_t *log; > + > + u_char type; > + ngx_str_t payload; > + uint64_t number; > + ngx_quic_compat_keys_t *keys; > + > + enum ssl_encryption_level_t level; > +} ngx_quic_compat_record_t; > + > + > +struct ngx_quic_compat_s { > + const SSL_QUIC_METHOD *method; > + > + enum ssl_encryption_level_t write_level; > + enum ssl_encryption_level_t read_level; > + > + uint64_t read_record; > + ngx_quic_compat_keys_t keys; > + > + ngx_str_t tp; > + ngx_str_t ctp; > +}; > + > + > +static void ngx_quic_compat_keylog_callback(const SSL *ssl, const char *line); > +static ngx_int_t ngx_quic_compat_set_encryption_secret(ngx_log_t *log, > + ngx_quic_compat_keys_t *keys, enum ssl_encryption_level_t level, > + const SSL_CIPHER *cipher, const uint8_t *secret, size_t secret_len); > +static int ngx_quic_compat_add_transport_params_callback(SSL *ssl, > + unsigned int ext_type, unsigned int context, const unsigned char **out, > + size_t *outlen, X509 *x, size_t chainidx, int *al, void *add_arg); > +static int ngx_quic_compat_parse_transport_params_callback(SSL *ssl, > + unsigned int ext_type, unsigned int context, const unsigned char *in, > + size_t inlen, X509 *x, size_t chainidx, int *al, void *parse_arg); > +static void ngx_quic_compat_message_callback(int write_p, int version, > + int content_type, const void *buf, size_t len, SSL *ssl, void *arg); > +static size_t ngx_quic_compat_create_header(ngx_quic_compat_record_t *rec, > + u_char *out, ngx_uint_t plain); > +static ngx_int_t ngx_quic_compat_create_record(ngx_quic_compat_record_t *rec, > + ngx_str_t *res); > + > + > +ngx_int_t > +ngx_quic_compat_init(SSL_CTX *ctx) > +{ > + SSL_CTX_set_keylog_callback(ctx, ngx_quic_compat_keylog_callback); > + > + if (SSL_CTX_add_custom_ext(ctx, NGX_QUIC_COMPAT_SSL_TP_EXT, > + SSL_EXT_CLIENT_HELLO > + |SSL_EXT_TLS1_3_ENCRYPTED_EXTENSIONS, > + ngx_quic_compat_add_transport_params_callback, > + NULL, > + NULL, > + ngx_quic_compat_parse_transport_params_callback, > + NULL) > + == 0) > + { > + return NGX_ERROR; > + } > + > + return NGX_OK; > +} > + > + > +static void > +ngx_quic_compat_keylog_callback(const SSL *ssl, const char *line) > +{ > + u_char ch, *p, *start, value; > + size_t n; > + ngx_uint_t write; > + const SSL_CIPHER *cipher; > + ngx_quic_compat_t *com; > + ngx_connection_t *c; > + ngx_quic_connection_t *qc; > + enum ssl_encryption_level_t level; > + u_char secret[EVP_MAX_MD_SIZE]; > + > + c = ngx_ssl_get_connection(ssl); > + if (c->type != SOCK_DGRAM) { > + return; > + } > + > + p = (u_char *) line; > + > + for (start = p; *p && *p != ' '; p++); > + > + n = p - start; > + > + ngx_log_debug2(NGX_LOG_DEBUG_EVENT, c->log, 0, > + "quic compat secret %*s", n, start); > + > + if (n == sizeof(NGX_QUIC_COMPAT_CLIENT_EARLY) - 1 > + && ngx_strncmp(start, NGX_QUIC_COMPAT_CLIENT_EARLY, n) == 0) For the record, with the current approach, this condition will never match: > + { > + level = ssl_encryption_early_data; > + write = 0; > + > + } else if (n == sizeof(NGX_QUIC_COMPAT_CLIENT_HANDSHAKE) - 1 > + && ngx_strncmp(start, NGX_QUIC_COMPAT_CLIENT_HANDSHAKE, n) == 0) > + { > + level = ssl_encryption_handshake; > + write = 0; > + > + } else if (n == sizeof(NGX_QUIC_COMPAT_SERVER_HANDSHAKE) - 1 > + && ngx_strncmp(start, NGX_QUIC_COMPAT_SERVER_HANDSHAKE, n) == 0) > + { > + level = ssl_encryption_handshake; > + write = 1; > + > + } else if (n == sizeof(NGX_QUIC_COMPAT_CLIENT_APPLICATION) - 1 > + && ngx_strncmp(start, NGX_QUIC_COMPAT_CLIENT_APPLICATION, n) > + == 0) > + { > + level = ssl_encryption_application; > + write = 0; > + > + } else if (n == sizeof(NGX_QUIC_COMPAT_SERVER_APPLICATION) - 1 > + && ngx_strncmp(start, NGX_QUIC_COMPAT_SERVER_APPLICATION, n) > + == 0) > + { > + level = ssl_encryption_application; > + write = 1; > + > + } else { > + return; > + } > + > + if (*p++ == '\0') { > + return; > + } > + > + for ( /* void */ ; *p && *p != ' '; p++); > + > + if (*p++ == '\0') { > + return; > + } > + > + for (n = 0, start = p; *p; p++) { > + ch = *p; > + > + if (ch >= '0' && ch <= '9') { > + value = ch - '0'; > + goto next; > + } > + > + ch = (u_char) (ch | 0x20); > + > + if (ch >= 'a' && ch <= 'f') { > + value = ch - 'a' + 10; > + goto next; > + } > + > + ngx_log_error(NGX_LOG_EMERG, c->log, 0, > + "invalid OpenSSL QUIC secret format"); > + > + return; > + > + next: > + > + if ((p - start) % 2) { > + secret[n++] += value; > + > + } else { > + if (n >= EVP_MAX_MD_SIZE) { > + ngx_log_error(NGX_LOG_EMERG, c->log, 0, > + "too big OpenSSL QUIC secret"); > + return; > + } > + > + secret[n] = (value << 4); > + } > + } > + > + qc = ngx_quic_get_connection(c); > + com = qc->compat; > + cipher = SSL_get_current_cipher(ssl); > + > + if (write) { > + com->method->set_write_secret((SSL *) ssl, level, cipher, secret, n); > + com->write_level = level; > + > + } else { > + com->method->set_read_secret((SSL *) ssl, level, cipher, secret, n); > + com->read_level = level; > + com->read_record = 0; > + > + (void) ngx_quic_compat_set_encryption_secret(c->log, &com->keys, level, > + cipher, secret, n); > + } > +} > + > + > +static ngx_int_t > +ngx_quic_compat_set_encryption_secret(ngx_log_t *log, > + ngx_quic_compat_keys_t *keys, enum ssl_encryption_level_t level, > + const SSL_CIPHER *cipher, const uint8_t *secret, size_t secret_len) > +{ > + ngx_int_t key_len; > + ngx_str_t secret_str; > + ngx_uint_t i; > + ngx_quic_hkdf_t seq[2]; > + ngx_quic_secret_t *peer_secret; > + ngx_quic_ciphers_t ciphers; > + > + peer_secret = &keys->secret; > + > + keys->cipher = SSL_CIPHER_get_id(cipher); > + > + key_len = ngx_quic_ciphers(keys->cipher, &ciphers, level); > + > + if (key_len == NGX_ERROR) { > + ngx_ssl_error(NGX_LOG_INFO, log, 0, "unexpected cipher"); > + return NGX_ERROR; > + } > + > + if (sizeof(peer_secret->secret.data) < secret_len) { > + ngx_log_error(NGX_LOG_ALERT, log, 0, > + "unexpected secret len: %uz", secret_len); > + return NGX_ERROR; > + } > + > + peer_secret->secret.len = secret_len; > + ngx_memcpy(peer_secret->secret.data, secret, secret_len); > + > + peer_secret->key.len = key_len; > + peer_secret->iv.len = NGX_QUIC_IV_LEN; > + > + secret_str.len = secret_len; > + secret_str.data = (u_char *) secret; > + > + ngx_quic_hkdf_set(&seq[0], "tls13 key", &peer_secret->key, &secret_str); > + ngx_quic_hkdf_set(&seq[1], "tls13 iv", &peer_secret->iv, &secret_str); > + > + for (i = 0; i < (sizeof(seq) / sizeof(seq[0])); i++) { > + if (ngx_quic_hkdf_expand(&seq[i], ciphers.d, log) != NGX_OK) { > + return NGX_ERROR; > + } > + } > + > + return NGX_OK; > +} > + > + > +static int > +ngx_quic_compat_add_transport_params_callback(SSL *ssl, unsigned int ext_type, > + unsigned int context, const unsigned char **out, size_t *outlen, X509 *x, > + size_t chainidx, int *al, void *add_arg) > +{ > + ngx_connection_t *c; > + ngx_quic_compat_t *com; > + ngx_quic_connection_t *qc; > + > + c = ngx_ssl_get_connection(ssl); > + if (c->type != SOCK_DGRAM) { > + return 0; > + } > + > + ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0, > + "quic compat add transport params"); > + > + qc = ngx_quic_get_connection(c); > + com = qc->compat; > + > + *out = com->tp.data; > + *outlen = com->tp.len; > + > + return 1; > +} > + > + > +static int > +ngx_quic_compat_parse_transport_params_callback(SSL *ssl, unsigned int ext_type, > + unsigned int context, const unsigned char *in, size_t inlen, X509 *x, > + size_t chainidx, int *al, void *parse_arg) > +{ > + u_char *p; > + ngx_connection_t *c; > + ngx_quic_compat_t *com; > + ngx_quic_connection_t *qc; > + > + c = ngx_ssl_get_connection(ssl); > + if (c->type != SOCK_DGRAM) { > + return 0; > + } > + > + ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0, > + "quic compat parse transport params"); > + > + qc = ngx_quic_get_connection(c); > + com = qc->compat; > + > + p = ngx_pnalloc(c->pool, inlen); > + if (p == NULL) { > + return 0; > + } > + > + ngx_memcpy(p, in, inlen); > + > + com->ctp.data = p; > + com->ctp.len = inlen; > + > + return 1; > +} > + > + > +int > +SSL_set_quic_method(SSL *ssl, const SSL_QUIC_METHOD *quic_method) > +{ > + BIO *rbio, *wbio; > + ngx_connection_t *c; > + ngx_quic_compat_t *com; > + ngx_quic_connection_t *qc; > + > + c = ngx_ssl_get_connection(ssl); > + > + ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0, "quic compat set method"); > + > + qc = ngx_quic_get_connection(c); > + > + qc->compat = ngx_pcalloc(c->pool, sizeof(ngx_quic_compat_t)); > + if (qc->compat == NULL) { > + return 0; > + } > + > + com = qc->compat; > + com->method = quic_method; > + > + rbio = BIO_new(BIO_s_mem()); > + if (rbio == NULL) { > + return 0; > + } > + > + wbio = BIO_new(BIO_s_null()); > + if (wbio == NULL) { > + return 0; > + } > + > + SSL_set_bio(ssl, rbio, wbio); > + > + SSL_set_msg_callback(ssl, ngx_quic_compat_message_callback); > + > + /* early data is not supported */ > + SSL_set_max_early_data(ssl, 0); > + > + return 1; > +} > + > + > +static void > +ngx_quic_compat_message_callback(int write_p, int version, int content_type, > + const void *buf, size_t len, SSL *ssl, void *arg) > +{ > + ngx_uint_t alert; > + ngx_connection_t *c; > + ngx_quic_compat_t *com; > + ngx_quic_connection_t *qc; > + enum ssl_encryption_level_t level; > + > + if (!write_p) { > + return; > + } > + > + c = ngx_ssl_get_connection(ssl); > + qc = ngx_quic_get_connection(c); > + > + if (qc == NULL) { > + /* closing */ > + return; > + } > + > + com = qc->compat; > + level = com->write_level; > + > + switch (content_type) { > + > + case SSL3_RT_HANDSHAKE: > + ngx_log_debug2(NGX_LOG_DEBUG_EVENT, c->log, 0, > + "quic compat tx %s len:%uz ", > + ngx_quic_level_name(level), len); > + > + (void) com->method->add_handshake_data(ssl, level, buf, len); > + > + break; > + > + case SSL3_RT_ALERT: > + if (len >= 2) { > + alert = ((u_char *) buf)[1]; > + > + ngx_log_debug3(NGX_LOG_DEBUG_EVENT, c->log, 0, > + "quic compat %s alert:%ui len:%uz ", > + ngx_quic_level_name(level), alert, len); > + > + (void) com->method->send_alert(ssl, level, alert); > + > + break; > + } > + > + /* fall through */ > + > + default: > + ngx_log_debug3(NGX_LOG_DEBUG_EVENT, c->log, 0, > + "quic compat %s ignore msg:%d len:%uz ", > + ngx_quic_level_name(level), content_type, len); Other content_type values are typically SSL3_RT_HEADER with its short header, and SSL3_RT_INNER_CONTENT_TYPE pseudo content type used to preface the inner TLSv1.3 content type. As such, I think such log should be removed for brevity as typically useless. > + > + break; > + } > +} > + > + > +int > +SSL_provide_quic_data(SSL *ssl, enum ssl_encryption_level_t level, > + const uint8_t *data, size_t len) > +{ > + BIO *rbio; > + size_t n; > + u_char *p; > + ngx_str_t res; > + ngx_connection_t *c; > + ngx_quic_compat_t *com; > + ngx_quic_connection_t *qc; > + ngx_quic_compat_record_t rec; > + u_char in[NGX_QUIC_COMPAT_RECORD_SIZE + 1]; > + u_char out[NGX_QUIC_COMPAT_RECORD_SIZE + 1 > + + SSL3_RT_HEADER_LENGTH > + + EVP_GCM_TLS_TAG_LEN]; > + > + c = ngx_ssl_get_connection(ssl); > + > + ngx_log_debug2(NGX_LOG_DEBUG_EVENT, c->log, 0, "quic compat rx %s len:%uz", > + ngx_quic_level_name(level), len); > + > + qc = ngx_quic_get_connection(c); > + com = qc->compat; > + rbio = SSL_get_rbio(ssl); > + > + while (len) { > + ngx_memzero(&rec, sizeof(ngx_quic_compat_record_t)); > + > + rec.type = SSL3_RT_HANDSHAKE; > + rec.log = c->log; > + rec.number = com->read_record++; > + rec.keys = &com->keys; > + > + if (level == ssl_encryption_initial) { > + n = ngx_min(len, 65535); > + > + rec.payload.len = n; > + rec.payload.data = (u_char *) data; > + > + ngx_quic_compat_create_header(&rec, out, 1); > + > + BIO_write(rbio, out, SSL3_RT_HEADER_LENGTH); > + BIO_write(rbio, data, n); > + > +#if defined(NGX_QUIC_DEBUG_CRYPTO) && defined(NGX_QUIC_DEBUG_PACKETS) > + ngx_log_debug5(NGX_LOG_DEBUG_EVENT, c->log, 0, > + "quic compat record len:%uz %*xs%*xs", > + n + SSL3_RT_HEADER_LENGTH, > + (size_t) SSL3_RT_HEADER_LENGTH, out, n, data); > +#endif > + > + } else { > + n = ngx_min(len, NGX_QUIC_COMPAT_RECORD_SIZE); > + > + p = ngx_cpymem(in, data, n); > + *p++ = SSL3_RT_HANDSHAKE; > + > + rec.payload.len = p - in; > + rec.payload.data = in; > + > + res.data = out; > + > + if (ngx_quic_compat_create_record(&rec, &res) != NGX_OK) { > + return 0; > + } > + > +#if defined(NGX_QUIC_DEBUG_CRYPTO) && defined(NGX_QUIC_DEBUG_PACKETS) > + ngx_log_debug2(NGX_LOG_DEBUG_EVENT, c->log, 0, > + "quic compat record len:%uz %xV", res.len, &res); > +#endif > + > + BIO_write(rbio, res.data, res.len); > + } > + > + data += n; > + len -= n; > + } > + > + return 1; > +} > + > + > +static size_t > +ngx_quic_compat_create_header(ngx_quic_compat_record_t *rec, u_char *out, > + ngx_uint_t plain) > +{ > + u_char type; > + size_t len; > + > + len = rec->payload.len; > + > + if (plain) { > + type = rec->type; > + > + } else { > + type = SSL3_RT_APPLICATION_DATA; > + len += EVP_GCM_TLS_TAG_LEN; > + } > + > + out[0] = type; > + out[1] = 0x03; > + out[2] = 0x03; > + out[3] = (len >> 8); > + out[4] = len; > + > + return 5; > +} > + > + > +static ngx_int_t > +ngx_quic_compat_create_record(ngx_quic_compat_record_t *rec, ngx_str_t *res) > +{ > + ngx_str_t ad, out; > + ngx_quic_secret_t *secret; > + ngx_quic_ciphers_t ciphers; > + u_char nonce[NGX_QUIC_IV_LEN]; > + > + ad.data = res->data; > + ad.len = ngx_quic_compat_create_header(rec, ad.data, 0); > + > + out.len = rec->payload.len + EVP_GCM_TLS_TAG_LEN; > + out.data = res->data + ad.len; > + > +#ifdef NGX_QUIC_DEBUG_CRYPTO > + ngx_log_debug2(NGX_LOG_DEBUG_EVENT, rec->log, 0, > + "quic compat ad len:%uz %xV", ad.len, &ad); > +#endif > + > + if (ngx_quic_ciphers(rec->keys->cipher, &ciphers, rec->level) == NGX_ERROR) > + { > + return NGX_ERROR; > + } > + > + secret = &rec->keys->secret; > + > + ngx_memcpy(nonce, secret->iv.data, secret->iv.len); > + ngx_quic_compute_nonce(nonce, sizeof(nonce), rec->number); > + > + if (ngx_quic_tls_seal(ciphers.c, secret, &out, > + nonce, &rec->payload, &ad, rec->log) > + != NGX_OK) > + { > + return NGX_ERROR; > + } > + > + res->len = ad.len + out.len; > + > + return NGX_OK; > +} > + > + > +enum ssl_encryption_level_t > +SSL_quic_read_level(const SSL *ssl) > +{ > + ngx_connection_t *c; > + ngx_quic_connection_t *qc; > + > + c = ngx_ssl_get_connection(ssl); > + qc = ngx_quic_get_connection(c); > + > + return qc->compat->read_level; > +} > + > + > +enum ssl_encryption_level_t > +SSL_quic_write_level(const SSL *ssl) > +{ > + ngx_connection_t *c; > + ngx_quic_connection_t *qc; > + > + c = ngx_ssl_get_connection(ssl); > + qc = ngx_quic_get_connection(c); > + > + return qc->compat->write_level; > +} > + > + > +int > +SSL_set_quic_transport_params(SSL *ssl, const uint8_t *params, > + size_t params_len) > +{ > + ngx_connection_t *c; > + ngx_quic_compat_t *com; > + ngx_quic_connection_t *qc; > + > + c = ngx_ssl_get_connection(ssl); > + qc = ngx_quic_get_connection(c); > + com = qc->compat; > + > + com->tp.len = params_len; > + com->tp.data = (u_char *) params; > + > + return 1; > +} > + > + > +void > +SSL_get_peer_quic_transport_params(const SSL *ssl, const uint8_t **out_params, > + size_t *out_params_len) > +{ > + ngx_connection_t *c; > + ngx_quic_compat_t *com; > + ngx_quic_connection_t *qc; > + > + c = ngx_ssl_get_connection(ssl); > + qc = ngx_quic_get_connection(c); > + com = qc->compat; > + > + *out_params = com->ctp.data; > + *out_params_len = com->ctp.len; > +} > + > +#endif /* NGX_QUIC_OPENSSL_COMPAT */ > diff --git a/src/event/quic/ngx_event_quic_openssl_compat.h b/src/event/quic/ngx_event_quic_openssl_compat.h > new file mode 100644 > --- /dev/null > +++ b/src/event/quic/ngx_event_quic_openssl_compat.h > @@ -0,0 +1,51 @@ > + > +/* > + * Copyright (C) Nginx, Inc. > + */ > + > + > +#ifndef _NGX_EVENT_QUIC_OPENSSL_COMPAT_H_INCLUDED_ > +#define _NGX_EVENT_QUIC_OPENSSL_COMPAT_H_INCLUDED_ > + > + > +#include > +#include > + > + > +enum ssl_encryption_level_t { > + ssl_encryption_initial = 0, > + ssl_encryption_early_data, > + ssl_encryption_handshake, > + ssl_encryption_application > +}; > + > + > +typedef struct ssl_quic_method_st { > + int (*set_read_secret)(SSL *ssl, enum ssl_encryption_level_t level, > + const SSL_CIPHER *cipher, > + const uint8_t *rsecret, size_t secret_len); > + int (*set_write_secret)(SSL *ssl, enum ssl_encryption_level_t level, > + const SSL_CIPHER *cipher, > + const uint8_t *wsecret, size_t secret_len); bad indent > + int (*add_handshake_data)(SSL *ssl, enum ssl_encryption_level_t level, > + const uint8_t *data, size_t len); > + int (*flush_flight)(SSL *ssl); > + int (*send_alert)(SSL *ssl, enum ssl_encryption_level_t level, > + uint8_t alert); > +} SSL_QUIC_METHOD; > + > + > +ngx_int_t ngx_quic_compat_init(SSL_CTX *ctx); > + > +int SSL_set_quic_method(SSL *ssl, const SSL_QUIC_METHOD *quic_method); > +int SSL_provide_quic_data(SSL *ssl, enum ssl_encryption_level_t level, > + const uint8_t *data, size_t len); > +enum ssl_encryption_level_t SSL_quic_read_level(const SSL *ssl); > +enum ssl_encryption_level_t SSL_quic_write_level(const SSL *ssl); > +int SSL_set_quic_transport_params(SSL *ssl, const uint8_t *params, > + size_t params_len); > +void SSL_get_peer_quic_transport_params(const SSL *ssl, > + const uint8_t **out_params, size_t *out_params_len); > + > + > +#endif /* _NGX_EVENT_QUIC_OPENSSL_COMPAT_H_INCLUDED_ */ > diff --git a/src/event/quic/ngx_event_quic_protection.c b/src/event/quic/ngx_event_quic_protection.c > --- a/src/event/quic/ngx_event_quic_protection.c > +++ b/src/event/quic/ngx_event_quic_protection.c > @@ -23,37 +23,6 @@ > #endif > > > -#ifdef OPENSSL_IS_BORINGSSL > -#define ngx_quic_cipher_t EVP_AEAD > -#else > -#define ngx_quic_cipher_t EVP_CIPHER > -#endif > - > - > -typedef struct { > - const ngx_quic_cipher_t *c; > - const EVP_CIPHER *hp; > - const EVP_MD *d; > -} ngx_quic_ciphers_t; > - > - > -typedef struct { > - size_t out_len; > - u_char *out; > - > - size_t prk_len; > - const uint8_t *prk; > - > - size_t label_len; > - const u_char *label; > -} ngx_quic_hkdf_t; > - > -#define ngx_quic_hkdf_set(seq, _label, _out, _prk) \ > - (seq)->out_len = (_out)->len; (seq)->out = (_out)->data; \ > - (seq)->prk_len = (_prk)->len, (seq)->prk = (_prk)->data, \ > - (seq)->label_len = (sizeof(_label) - 1); (seq)->label = (u_char *)(_label); > - > - > static ngx_int_t ngx_hkdf_expand(u_char *out_key, size_t out_len, > const EVP_MD *digest, const u_char *prk, size_t prk_len, > const u_char *info, size_t info_len); > @@ -63,20 +32,12 @@ static ngx_int_t ngx_hkdf_extract(u_char > > static uint64_t ngx_quic_parse_pn(u_char **pos, ngx_int_t len, u_char *mask, > uint64_t *largest_pn); > -static void ngx_quic_compute_nonce(u_char *nonce, size_t len, uint64_t pn); > -static ngx_int_t ngx_quic_ciphers(ngx_uint_t id, > - ngx_quic_ciphers_t *ciphers, enum ssl_encryption_level_t level); > > static ngx_int_t ngx_quic_tls_open(const ngx_quic_cipher_t *cipher, > ngx_quic_secret_t *s, ngx_str_t *out, u_char *nonce, ngx_str_t *in, > ngx_str_t *ad, ngx_log_t *log); > -static ngx_int_t ngx_quic_tls_seal(const ngx_quic_cipher_t *cipher, > - ngx_quic_secret_t *s, ngx_str_t *out, u_char *nonce, ngx_str_t *in, > - ngx_str_t *ad, ngx_log_t *log); > static ngx_int_t ngx_quic_tls_hp(ngx_log_t *log, const EVP_CIPHER *cipher, > ngx_quic_secret_t *s, u_char *out, u_char *in); > -static ngx_int_t ngx_quic_hkdf_expand(ngx_quic_hkdf_t *hkdf, > - const EVP_MD *digest, ngx_log_t *log); > > static ngx_int_t ngx_quic_create_packet(ngx_quic_header_t *pkt, > ngx_str_t *res); > @@ -84,7 +45,7 @@ static ngx_int_t ngx_quic_create_retry_p > ngx_str_t *res); > > > -static ngx_int_t > +ngx_int_t > ngx_quic_ciphers(ngx_uint_t id, ngx_quic_ciphers_t *ciphers, > enum ssl_encryption_level_t level) > { > @@ -221,7 +182,7 @@ ngx_quic_keys_set_initial_secret(ngx_qui > } > > > -static ngx_int_t > +ngx_int_t > ngx_quic_hkdf_expand(ngx_quic_hkdf_t *h, const EVP_MD *digest, ngx_log_t *log) > { > size_t info_len; > @@ -480,7 +441,7 @@ ngx_quic_tls_open(const ngx_quic_cipher_ > } > > > -static ngx_int_t > +ngx_int_t > ngx_quic_tls_seal(const ngx_quic_cipher_t *cipher, ngx_quic_secret_t *s, > ngx_str_t *out, u_char *nonce, ngx_str_t *in, ngx_str_t *ad, ngx_log_t *log) > { > @@ -961,7 +922,7 @@ ngx_quic_parse_pn(u_char **pos, ngx_int_ > } > > > -static void > +void > ngx_quic_compute_nonce(u_char *nonce, size_t len, uint64_t pn) > { > nonce[len - 8] ^= (pn >> 56) & 0x3f; > diff --git a/src/event/quic/ngx_event_quic_protection.h b/src/event/quic/ngx_event_quic_protection.h > --- a/src/event/quic/ngx_event_quic_protection.h > +++ b/src/event/quic/ngx_event_quic_protection.h > @@ -23,6 +23,13 @@ > #define NGX_QUIC_MAX_MD_SIZE 48 > > > +#ifdef OPENSSL_IS_BORINGSSL > +#define ngx_quic_cipher_t EVP_AEAD > +#else > +#define ngx_quic_cipher_t EVP_CIPHER > +#endif > + > + > typedef struct { > size_t len; > u_char data[NGX_QUIC_MAX_MD_SIZE]; > @@ -56,6 +63,30 @@ struct ngx_quic_keys_s { > }; > > > +typedef struct { > + const ngx_quic_cipher_t *c; > + const EVP_CIPHER *hp; > + const EVP_MD *d; > +} ngx_quic_ciphers_t; > + > + > +typedef struct { > + size_t out_len; > + u_char *out; > + > + size_t prk_len; > + const uint8_t *prk; > + > + size_t label_len; > + const u_char *label; > +} ngx_quic_hkdf_t; > + > +#define ngx_quic_hkdf_set(seq, _label, _out, _prk) \ > + (seq)->out_len = (_out)->len; (seq)->out = (_out)->data; \ > + (seq)->prk_len = (_prk)->len, (seq)->prk = (_prk)->data, \ > + (seq)->label_len = (sizeof(_label) - 1); (seq)->label = (u_char *)(_label); > + > + > ngx_int_t ngx_quic_keys_set_initial_secret(ngx_quic_keys_t *keys, > ngx_str_t *secret, ngx_log_t *log); > ngx_int_t ngx_quic_keys_set_encryption_secret(ngx_log_t *log, > @@ -70,6 +101,14 @@ void ngx_quic_keys_switch(ngx_connection > ngx_int_t ngx_quic_keys_update(ngx_connection_t *c, ngx_quic_keys_t *keys); > ngx_int_t ngx_quic_encrypt(ngx_quic_header_t *pkt, ngx_str_t *res); > ngx_int_t ngx_quic_decrypt(ngx_quic_header_t *pkt, uint64_t *largest_pn); > +void ngx_quic_compute_nonce(u_char *nonce, size_t len, uint64_t pn); > +ngx_int_t ngx_quic_ciphers(ngx_uint_t id, ngx_quic_ciphers_t *ciphers, > + enum ssl_encryption_level_t level); > +ngx_int_t ngx_quic_tls_seal(const ngx_quic_cipher_t *cipher, > + ngx_quic_secret_t *s, ngx_str_t *out, u_char *nonce, ngx_str_t *in, > + ngx_str_t *ad, ngx_log_t *log); > +ngx_int_t ngx_quic_hkdf_expand(ngx_quic_hkdf_t *hkdf, const EVP_MD *digest, > + ngx_log_t *log); > > > #endif /* _NGX_EVENT_QUIC_PROTECTION_H_INCLUDED_ */ > diff --git a/src/event/quic/ngx_event_quic_ssl.c b/src/event/quic/ngx_event_quic_ssl.c > --- a/src/event/quic/ngx_event_quic_ssl.c > +++ b/src/event/quic/ngx_event_quic_ssl.c > @@ -10,6 +10,13 @@ > #include > > > +#if defined OPENSSL_IS_BORINGSSL \ > + || defined LIBRESSL_VERSION_NUMBER \ > + || NGX_QUIC_OPENSSL_COMPAT > +#define NGX_QUIC_BORINGSSL_API 1 > +#endif > + > + > /* > * RFC 9000, 7.5. Cryptographic Message Buffering > * > @@ -18,7 +25,7 @@ > #define NGX_QUIC_MAX_BUFFERED 65535 > > > -#if defined OPENSSL_IS_BORINGSSL || defined LIBRESSL_VERSION_NUMBER > +#if (NGX_QUIC_BORINGSSL_API) > static int ngx_quic_set_read_secret(ngx_ssl_conn_t *ssl_conn, > enum ssl_encryption_level_t level, const SSL_CIPHER *cipher, > const uint8_t *secret, size_t secret_len); > @@ -39,7 +46,7 @@ static int ngx_quic_send_alert(ngx_ssl_c > static ngx_int_t ngx_quic_crypto_input(ngx_connection_t *c, ngx_chain_t *data); > > > -#if defined OPENSSL_IS_BORINGSSL || defined LIBRESSL_VERSION_NUMBER > +#if (NGX_QUIC_BORINGSSL_API) > > static int > ngx_quic_set_read_secret(ngx_ssl_conn_t *ssl_conn, > @@ -67,12 +74,6 @@ ngx_quic_set_read_secret(ngx_ssl_conn_t > return 0; > } > > - if (level == ssl_encryption_early_data) { > - if (ngx_quic_init_streams(c) != NGX_OK) { > - return 0; > - } > - } > - > return 1; > } > > @@ -138,10 +139,6 @@ ngx_quic_set_encryption_secrets(ngx_ssl_ > } > > if (level == ssl_encryption_early_data) { > - if (ngx_quic_init_streams(c) != NGX_OK) { > - return 0; > - } > - > return 1; > } > > @@ -456,6 +453,12 @@ ngx_quic_crypto_input(ngx_connection_t * > return NGX_ERROR; > } > > + if (ngx_quic_keys_available(qc->keys, ssl_encryption_early_data)) { > + if (ngx_quic_init_streams(c) != NGX_OK) { > + return NGX_ERROR; > + } > + } > + > return NGX_OK; > } > For the record, these chunks were merged. > @@ -540,7 +543,7 @@ ngx_quic_init_connection(ngx_connection_ > ssl_conn = c->ssl->connection; > > if (!quic_method.send_alert) { > -#if defined OPENSSL_IS_BORINGSSL || defined LIBRESSL_VERSION_NUMBER > +#if (NGX_QUIC_BORINGSSL_API) > quic_method.set_read_secret = ngx_quic_set_read_secret; > quic_method.set_write_secret = ngx_quic_set_write_secret; > #else > diff --git a/src/event/quic/ngx_event_quic_transport.h b/src/event/quic/ngx_event_quic_transport.h > --- a/src/event/quic/ngx_event_quic_transport.h > +++ b/src/event/quic/ngx_event_quic_transport.h > @@ -11,6 +11,10 @@ > #include > #include > > +#if (NGX_QUIC_OPENSSL_COMPAT) > +#include > +#endif > + > > /* > * RFC 9000, 17.2. Long Header Packets I think it make sense to adjust the compat header inclusion in ngx_event_quic_connection.h, in order to make this part unnecessary, and for more consistency: diff --git a/src/event/quic/ngx_event_quic_connection.h b/src/event/quic/ngx_eve nt_quic_connection.h --- a/src/event/quic/ngx_event_quic_connection.h +++ b/src/event/quic/ngx_event_quic_connection.h @@ -28,6 +28,9 @@ typedef struct ngx_quic_keys_s ng typedef struct ngx_quic_compat_s ngx_quic_compat_t; #endif +#if (NGX_QUIC_OPENSSL_COMPAT) +#include +#endif #include #include #include @@ -39,9 +42,6 @@ typedef struct ngx_quic_compat_s ng #include #include #include -#if (NGX_QUIC_OPENSSL_COMPAT) -#include -#endif /* RFC 9002, 6.2.2. Handshakes and New Paths: kInitialRtt */ diff --git a/src/event/quic/ngx_event_quic_transport.h b/src/event/quic/ngx_event_quic_transport.h --- a/src/event/quic/ngx_event_quic_transport.h +++ b/src/event/quic/ngx_event_quic_transport.h @@ -11,10 +11,6 @@ #include #include -#if (NGX_QUIC_OPENSSL_COMPAT) -#include -#endif - /* * RFC 9000, 17.2. Long Header Packets In other news, I looked how to obtain 0-RTT keys without interacting with SSL_read_early_data(), which enters the OpenSSL handshake state machine into a special mode and causes TLSv1.3/QUIC handshake message mismatches. This approach uses the SSL_SESSION_get_master_key() API. In TLSv1.3, it returns PSK, which accoring to TLSv1.3 key schedule is used as the input secret to extract Early Secret, then to derive client_early_traffic_secret with ClientHello message transcript digest, which we also control, so obtaining 0-RTT keys is relatively easy. Still, OpenSSL knows nothing about we accept 0-RTT, and as such it won't set the required early_data extension in ServerHello, which means we rejected it, even if 0-RTT was read and acknowledged. So the only way to switch the state seems to call SSL_read_early_data() with all its hard-ways. From mdounin at mdounin.ru Thu Feb 2 20:30:56 2023 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 2 Feb 2023 23:30:56 +0300 Subject: [PATCH] Gzip static: ranges support (ticket #2349) In-Reply-To: <33B9DA06-D8F7-4013-B696-5DB776D5DDAE@nginx.com> References: <262ABC58-7AB2-4B53-90D4-648464DE4F44@nginx.com> <33B9DA06-D8F7-4013-B696-5DB776D5DDAE@nginx.com> Message-ID: Hello! On Thu, Feb 02, 2023 at 01:05:19PM +0400, Sergey Kandaurov wrote: > > On 29 Jan 2023, at 00:03, Maxim Dounin wrote: > > > > Hello! > > > > On Wed, Jan 25, 2023 at 01:50:56PM +0400, Sergey Kandaurov wrote: > > > >>> On 24 Jan 2023, at 06:19, Maxim Dounin wrote: > >>> > >>>>> [..] > >>> > >>> diff --git a/src/http/modules/ngx_http_flv_module.c b/src/http/modules/ngx_http_flv_module.c > >>> --- a/src/http/modules/ngx_http_flv_module.c > >>> +++ b/src/http/modules/ngx_http_flv_module.c > >>> @@ -235,6 +235,7 @@ ngx_http_flv_handler(ngx_http_request_t > >>> b->in_file = b->file_last ? 1: 0; > >> > >> btw, there are old style issues originated in the static module > >> (it was fixed in gzip_static when it was copied from there) > > > > This is actually an (old) style, which was previously used in many > > places. Now it remains mostly in multi-line forms of the > > conditional operator, e.g.: > > > > r->loc_conf = (node->exact) ? node->exact->loc_conf: > > node->inclusive->loc_conf; > > > > Single-line form is indeed mostly unused now, and probably it's > > time to remove remaining occurrences. Added a patch for this. > > > > [...] > > > >>> @@ -238,10 +238,6 @@ ngx_http_static_handler(ngx_http_request > >>> return NGX_HTTP_INTERNAL_SERVER_ERROR; > >>> } > >>> > >>> - if (r != r->main && of.size == 0) { > >>> - return ngx_http_send_header(r); > >>> - } > >>> - > >>> r->allow_ranges = 1; > >>> > >>> /* we need to allocate all before the header would be sent */ > >>> @@ -268,6 +264,7 @@ ngx_http_static_handler(ngx_http_request > >>> b->in_file = b->file_last ? 1: 0; > >>> b->last_buf = (r == r->main) ? 1: 0; > >>> b->last_in_chain = 1; > >>> + b->sync = (r == r->main || b->file_last) ? 0 : 1; > >>> > >>> b->file->fd = of.fd; > >>> b->file->name = path; > >>> > >> > >> In addition to the static module change, any reason not to include > >> ngx_http_cache_send()? It has a similar behaviour to skip cached > >> empty body if served in subrequests. Yet another similar place is > >> subrequests processed with rewrite such as "return 204;". > > > > Indeed, thanks for catching, missed these two. > > > > Also, looking into this places I tend to think it's better to use > > > > b->sync = (b->last_buf || b->in_file) ? 0 : 1; > > > > instead, which is going to be simpler and more universal across > > all uses. > > I agree. > > > > > # HG changeset patch > > # User Maxim Dounin > > # Date 1674872423 -10800 > > # Sat Jan 28 05:20:23 2023 +0300 > > # Node ID 1c3b78d7cdc9e7059e90aa1670d58fd37927a1a2 > > # Parent 106328a70f4ecb32f828d33e5cd66c861e455f92 > > Style. > > > > diff --git a/src/event/ngx_event_connectex.c b/src/event/ngx_event_connectex.c > > --- a/src/event/ngx_event_connectex.c > > +++ b/src/event/ngx_event_connectex.c > > @@ -127,8 +127,8 @@ void ngx_iocp_wait_events(int main) > > conn[0] = NULL; > > > > for ( ;; ) { > > - offset = (nevents == WSA_MAXIMUM_WAIT_EVENTS + 1) ? 1: 0; > > - timeout = (nevents == 1 && !first) ? 60000: INFINITE; > > + offset = (nevents == WSA_MAXIMUM_WAIT_EVENTS + 1) ? 1 : 0; > > + timeout = (nevents == 1 && !first) ? 60000 : INFINITE; > > > > n = WSAWaitForMultipleEvents(nevents - offset, events[offset], > > 0, timeout, 0); > > diff --git a/src/http/modules/ngx_http_flv_module.c b/src/http/modules/ngx_http_flv_module.c > > --- a/src/http/modules/ngx_http_flv_module.c > > +++ b/src/http/modules/ngx_http_flv_module.c > > @@ -232,7 +232,7 @@ ngx_http_flv_handler(ngx_http_request_t > > b->file_pos = start; > > b->file_last = of.size; > > > > - b->in_file = b->file_last ? 1: 0; > > + b->in_file = b->file_last ? 1 : 0; > > b->last_buf = (r == r->main) ? 1 : 0; > > b->last_in_chain = 1; > > > > diff --git a/src/http/modules/ngx_http_static_module.c b/src/http/modules/ngx_http_static_module.c > > --- a/src/http/modules/ngx_http_static_module.c > > +++ b/src/http/modules/ngx_http_static_module.c > > @@ -265,8 +265,8 @@ ngx_http_static_handler(ngx_http_request > > b->file_pos = 0; > > b->file_last = of.size; > > > > - b->in_file = b->file_last ? 1: 0; > > - b->last_buf = (r == r->main) ? 1: 0; > > + b->in_file = b->file_last ? 1 : 0; > > + b->last_buf = (r == r->main) ? 1 : 0; > > b->last_in_chain = 1; > > > > b->file->fd = of.fd; > > diff --git a/src/http/ngx_http_file_cache.c b/src/http/ngx_http_file_cache.c > > --- a/src/http/ngx_http_file_cache.c > > +++ b/src/http/ngx_http_file_cache.c > > @@ -1600,8 +1600,8 @@ ngx_http_cache_send(ngx_http_request_t * > > b->file_pos = c->body_start; > > b->file_last = c->length; > > > > - b->in_file = (c->length - c->body_start) ? 1: 0; > > - b->last_buf = (r == r->main) ? 1: 0; > > + b->in_file = (c->length - c->body_start) ? 1 : 0; > > + b->last_buf = (r == r->main) ? 1 : 0; > > b->last_in_chain = 1; > > > > b->file->fd = c->file.fd; > > # HG changeset patch > > # User Maxim Dounin > > # Date 1674872613 -10800 > > # Sat Jan 28 05:23:33 2023 +0300 > > # Node ID f5515e727656b2cc529b26a0fe6b4496c62e32b2 > > # Parent 1c3b78d7cdc9e7059e90aa1670d58fd37927a1a2 > > Fixed "zero size buf" alerts with subrequests. > > > > Since 4611:2b6cb7528409 responses from the gzip static, flv, and mp4 modules > > can be used with subrequests, though empty files were not properly handled. > > Empty gzipped, flv, and mp4 files thus resulted in "zero size buf in output" > > alerts. While valid corresponding files are not expected to be empty, such > > files shouldn't result in alerts. > > > > Fix is to set b->sync on such empty subrequest responses, similarly to what > > ngx_http_send_special() does. > > > > Additionally, the static module, the ngx_http_send_response() function, and > > file cache are modified to do the same instead of not sending the response > > body at all in such cases, since not sending the response body at all is > > believed to be at least questionable, and might break various filters > > which do not expect such behaviour. > > > > diff --git a/src/http/modules/ngx_http_flv_module.c b/src/http/modules/ngx_http_flv_module.c > > --- a/src/http/modules/ngx_http_flv_module.c > > +++ b/src/http/modules/ngx_http_flv_module.c > > @@ -235,6 +235,7 @@ ngx_http_flv_handler(ngx_http_request_t > > b->in_file = b->file_last ? 1 : 0; > > b->last_buf = (r == r->main) ? 1 : 0; > > b->last_in_chain = 1; > > + b->sync = (b->last_buf || b->in_file) ? 0 : 1; > > > > b->file->fd = of.fd; > > b->file->name = path; > > diff --git a/src/http/modules/ngx_http_gzip_static_module.c b/src/http/modules/ngx_http_gzip_static_module.c > > --- a/src/http/modules/ngx_http_gzip_static_module.c > > +++ b/src/http/modules/ngx_http_gzip_static_module.c > > @@ -273,6 +273,7 @@ ngx_http_gzip_static_handler(ngx_http_re > > b->in_file = b->file_last ? 1 : 0; > > b->last_buf = (r == r->main) ? 1 : 0; > > b->last_in_chain = 1; > > + b->sync = (b->last_buf || b->in_file) ? 0 : 1; > > > > b->file->fd = of.fd; > > b->file->name = path; > > diff --git a/src/http/modules/ngx_http_mp4_module.c b/src/http/modules/ngx_http_mp4_module.c > > --- a/src/http/modules/ngx_http_mp4_module.c > > +++ b/src/http/modules/ngx_http_mp4_module.c > > @@ -714,6 +714,7 @@ ngx_http_mp4_handler(ngx_http_request_t > > b->in_file = b->file_last ? 1 : 0; > > b->last_buf = (r == r->main) ? 1 : 0; > > b->last_in_chain = 1; > > + b->sync = (b->last_buf || b->in_file) ? 0 : 1; > > > > b->file->fd = of.fd; > > b->file->name = path; > > diff --git a/src/http/modules/ngx_http_static_module.c b/src/http/modules/ngx_http_static_module.c > > --- a/src/http/modules/ngx_http_static_module.c > > +++ b/src/http/modules/ngx_http_static_module.c > > @@ -238,10 +238,6 @@ ngx_http_static_handler(ngx_http_request > > return NGX_HTTP_INTERNAL_SERVER_ERROR; > > } > > > > - if (r != r->main && of.size == 0) { > > - return ngx_http_send_header(r); > > - } > > - > > r->allow_ranges = 1; > > > > /* we need to allocate all before the header would be sent */ > > @@ -268,6 +264,7 @@ ngx_http_static_handler(ngx_http_request > > b->in_file = b->file_last ? 1 : 0; > > b->last_buf = (r == r->main) ? 1 : 0; > > b->last_in_chain = 1; > > + b->sync = (b->last_buf || b->in_file) ? 0 : 1; > > > > b->file->fd = of.fd; > > b->file->name = path; > > diff --git a/src/http/ngx_http_core_module.c b/src/http/ngx_http_core_module.c > > --- a/src/http/ngx_http_core_module.c > > +++ b/src/http/ngx_http_core_module.c > > @@ -1803,10 +1803,6 @@ ngx_http_send_response(ngx_http_request_ > > } > > } > > > > - if (r != r->main && val.len == 0) { > > - return ngx_http_send_header(r); > > - } > > - > > b = ngx_calloc_buf(r->pool); > > if (b == NULL) { > > return NGX_HTTP_INTERNAL_SERVER_ERROR; > > @@ -1817,6 +1813,7 @@ ngx_http_send_response(ngx_http_request_ > > b->memory = val.len ? 1 : 0; > > b->last_buf = (r == r->main) ? 1 : 0; > > b->last_in_chain = 1; > > + b->sync = (b->last_buf || b->memory) ? 0 : 1; > > > > out.buf = b; > > out.next = NULL; > > diff --git a/src/http/ngx_http_file_cache.c b/src/http/ngx_http_file_cache.c > > --- a/src/http/ngx_http_file_cache.c > > +++ b/src/http/ngx_http_file_cache.c > > @@ -1575,10 +1575,6 @@ ngx_http_cache_send(ngx_http_request_t * > > ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, > > "http file cache send: %s", c->file.name.data); > > > > - if (r != r->main && c->length - c->body_start == 0) { > > - return ngx_http_send_header(r); > > - } > > - > > /* we need to allocate all before the header would be sent */ > > > > b = ngx_calloc_buf(r->pool); > > @@ -1603,6 +1599,7 @@ ngx_http_cache_send(ngx_http_request_t * > > b->in_file = (c->length - c->body_start) ? 1 : 0; > > b->last_buf = (r == r->main) ? 1 : 0; > > b->last_in_chain = 1; > > + b->sync = (b->last_buf || b->in_file) ? 0 : 1; > > > > b->file->fd = c->file.fd; > > b->file->name = c->file.name; > > > > Both changes look good to me, please commit. Thanks for the review, pushed to http://mdounin.ru/hg/nginx. -- Maxim Dounin http://mdounin.ru/ From ryan at lahfa.xyz Thu Feb 2 22:06:25 2023 From: ryan at lahfa.xyz (=?iso-8859-1?q?Ryan_Lahfa?=) Date: Thu, 02 Feb 2023 23:06:25 +0100 Subject: [PATCH] HTTP: Support mixed addr configuration for PROXY protocol in RealIP module Message-ID: <53cf9a05e1ae1535166f.1675375585@localhost> # HG changeset patch # User Ryan Lahfa # Date 1675214187 -3600 # Wed Feb 01 02:16:27 2023 +0100 # Node ID 53cf9a05e1ae1535166f45582eb4bf5aa34c23ea # Parent 106328a70f4ecb32f828d33e5cd66c861e455f92 HTTP: Support mixed addr configuration for PROXY protocol in RealIP module This ensures that under `real_ip_header proxy_protocol`, if a request is received on a `listen` block which do not contain `proxy_protocol`, it is not rejected by NGINX. This enables the usecase where you want clients to let them access directly without going through a "load balancer" or a "proxy" while having PROXY protocol for the other clients. This do not introduce vulnerability per se, non-PROXY requests unsets all the RealIP module context when `real_ip_header` is set to proxy protocol. This is akin to not having RealIP module working for that HTTP request. diff -r 106328a70f4e -r 53cf9a05e1ae src/http/modules/ngx_http_realip_module.c --- a/src/http/modules/ngx_http_realip_module.c Sat Jan 28 01:29:45 2023 +0300 +++ b/src/http/modules/ngx_http_realip_module.c Wed Feb 01 02:16:27 2023 +0100 @@ -179,10 +179,24 @@ case NGX_HTTP_REALIP_PROXY: + // If the address configuration is not PROXY protocol-enabled + // We can ignore this request. + // http_connection is guaranteed to exist as we are + // in a HTTP context. + if (!r->http_connection->addr_conf->proxy_protocol) { + // Unset context to have valid $remote_addr, $remote_port vars. + ngx_http_set_ctx(r, NULL, ngx_http_realip_module); + return NGX_OK; + } + + // We are supposed to receive a PROXY protocol-enabled request + // and this is not the case. + // We reject this request. if (r->connection->proxy_protocol == NULL) { return NGX_DECLINED; } + // This value is now guaranteed to exist. value = &r->connection->proxy_protocol->src_addr; xfwd = NULL; @@ -242,6 +256,8 @@ return ngx_http_realip_set_addr(r, &addr); } + ngx_log_error(NGX_LOG_ALERT, r->connection->log, 0, + "'%V' is not an allowed proxy, declining request", &c->addr_text); return NGX_DECLINED; } From mdounin at mdounin.ru Fri Feb 3 01:50:53 2023 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 3 Feb 2023 04:50:53 +0300 Subject: [PATCH] HTTP: trigger lingering close when keepalive connection will be closed In-Reply-To: <67831923-6BAE-4EF0-8457-46128FEB9CF5@nginx.com> References: <699749D2-D96F-47D8-85DA-74E7F97682FF@gmail.com> <8D59E87C-2DC3-43AB-9067-019CA9ACE5C4@gmail.com> <67831923-6BAE-4EF0-8457-46128FEB9CF5@nginx.com> Message-ID: Hello! On Thu, Feb 02, 2023 at 06:53:20PM +0400, Sergey Kandaurov wrote: > > On 27 Jan 2023, at 08:01, Maxim Dounin wrote: > > > > Hello! > > > > [..] > > Overall, after looking into logs and tcpdump you've provided I > > tend to think that the only working fix would be to introduce > > c->pipeline flag, and force lingering close if there were any > > pipelined requests on the connection. > > > > Below is the patch which implements this approach. Review and > > testing appreciated. It can be used either separately or with the > > previously provided patch to use posted next events. > > > > # HG changeset patch > > # User Maxim Dounin > > # Date 1674790916 -10800 > > # Fri Jan 27 06:41:56 2023 +0300 > > # Node ID 784d0fa0b5a0796561642a5a64dc4e9e07592852 > > # Parent 4eb1383f6432b034630e6de18739b817f6565c8c > > Lingering close for connections with pipelined requests. > > > > This is expected to help with clients using pipelining with some constant > > depth, such as apt[1][2]. > > > > When downloading many resources, apt uses pipelining with some constant > > depth, a number of requests in flight[1][2]. This essentially means that > > Are links repeated on purpose? No, thanks for catching. Removed the second pair. > > after receiving a response it sends an additional request to the server, > > and this can result in requests arriving to the server at any time. Further, > > additional requests are sent one-by-one, and can be easily seen as such > > (neither as pipelined, nor followed by pipelined requests). > > > > The only safe approach to close such connections (for example, when > > keepalive_requests is reached) is with lingering. To do so, now nginx > > monitors if pipelining was used on the connection, and if it was, closes > > the connection with lingering. > > > > [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=973861#10 > > [2] https://mailman.nginx.org/pipermail/nginx-devel/2023-January/ZA2SP5SJU55LHEBCJMFDB2AZVELRLTHI.html > > > > diff --git a/src/core/ngx_connection.h b/src/core/ngx_connection.h > > --- a/src/core/ngx_connection.h > > +++ b/src/core/ngx_connection.h > > @@ -172,6 +172,7 @@ struct ngx_connection_s { > > unsigned timedout:1; > > unsigned error:1; > > unsigned destroyed:1; > > + unsigned pipeline:1; > > > > unsigned idle:1; > > unsigned reusable:1; > > diff --git a/src/http/ngx_http_request.c b/src/http/ngx_http_request.c > > --- a/src/http/ngx_http_request.c > > +++ b/src/http/ngx_http_request.c > > @@ -2753,7 +2753,8 @@ ngx_http_finalize_connection(ngx_http_re > > || (clcf->lingering_close == NGX_HTTP_LINGERING_ON > > && (r->lingering_close > > || r->header_in->pos < r->header_in->last > > - || r->connection->read->ready))) > > + || r->connection->read->ready > > + || r->connection->pipeline))) > > { > > ngx_http_set_lingering_close(r->connection); > > return; > > @@ -3123,6 +3124,7 @@ ngx_http_set_keepalive(ngx_http_request_ > > > > c->sent = 0; > > c->destroyed = 0; > > + c->pipeline = 1; > > > > if (rev->timer_set) { > > ngx_del_timer(rev); > > > > Looks good to me. > > Further, as request pipelining implementation status is quite limited > (often disabled by default or unimplemented, notably in browsers), > I believe the change is fine to not cause noticeable legit resource usage. > Still, old behaviour can be reverted with "lingering_close off;". Well, previous behaviour is a bit better than "lingering_close off;", but indeed, the idea behind the patch is that pipelining is mostly unused, and hence the change shouldn't affect most of the workloads at all. On the other hand, if pipelining is used, there is good chance that lingering close is actually needed. Further, since 1.19.7 nginx is able to force-close connections in lingering close state when free worker connections are exhausted, so some additional lingering shouldn't be problematic. Thanks for the review, pushed to http://mdounin.ru/hg/nginx. -- Maxim Dounin http://mdounin.ru/ From mdounin at mdounin.ru Fri Feb 3 02:51:48 2023 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 3 Feb 2023 05:51:48 +0300 Subject: [PATCH] HTTP: Support mixed addr configuration for PROXY protocol in RealIP module In-Reply-To: <53cf9a05e1ae1535166f.1675375585@localhost> References: <53cf9a05e1ae1535166f.1675375585@localhost> Message-ID: Hello! On Thu, Feb 02, 2023 at 11:06:25PM +0100, Ryan Lahfa wrote: > # HG changeset patch > # User Ryan Lahfa > # Date 1675214187 -3600 > # Wed Feb 01 02:16:27 2023 +0100 > # Node ID 53cf9a05e1ae1535166f45582eb4bf5aa34c23ea > # Parent 106328a70f4ecb32f828d33e5cd66c861e455f92 > HTTP: Support mixed addr configuration for PROXY protocol in RealIP module > > This ensures that under `real_ip_header proxy_protocol`, if a request is > received on a `listen` block which do not contain `proxy_protocol`, it > is not rejected by NGINX. Such requests are not currently rejected by nginx. Rather, the request is ignored by the realip module (which returns NGX_DECLINED). For example, for the following configuration: server { listen 8080; listen 8081 proxy_protocol; set_real_ip_from 127.0.0.1/32; real_ip_header proxy_protocol; return 200 "remote_addr: $remote_addr\n"; } will accept requests on port 8080 without PROXY protocol, and on port 8081 with PROXY protocol. Requests on port 8081 will be handled by the realip module, and the remote address will be set to the one obtained via PROXY protocol (as long as allowed by set_real_ip_from). Requests on port 8080 will be handled without additional processing by the realip module. Could you please clarify what are you trying to fix here? > > This enables the usecase where you want clients to let them access > directly without going through a "load balancer" or a "proxy" while > having PROXY protocol for the other clients. > > This do not introduce vulnerability per se, non-PROXY requests > unsets all the RealIP module context when `real_ip_header` is set to proxy protocol. > This is akin to not having RealIP module working for that HTTP request. > > diff -r 106328a70f4e -r 53cf9a05e1ae src/http/modules/ngx_http_realip_module.c > --- a/src/http/modules/ngx_http_realip_module.c Sat Jan 28 01:29:45 2023 +0300 > +++ b/src/http/modules/ngx_http_realip_module.c Wed Feb 01 02:16:27 2023 +0100 > @@ -179,10 +179,24 @@ > > case NGX_HTTP_REALIP_PROXY: > > + // If the address configuration is not PROXY protocol-enabled > + // We can ignore this request. > + // http_connection is guaranteed to exist as we are > + // in a HTTP context. > + if (!r->http_connection->addr_conf->proxy_protocol) { > + // Unset context to have valid $remote_addr, $remote_port vars. > + ngx_http_set_ctx(r, NULL, ngx_http_realip_module); > + return NGX_OK; > + } > + > + // We are supposed to receive a PROXY protocol-enabled request > + // and this is not the case. > + // We reject this request. > if (r->connection->proxy_protocol == NULL) { > return NGX_DECLINED; > } It looks like these changes essentially do nothing, see above. > > + // This value is now guaranteed to exist. > value = &r->connection->proxy_protocol->src_addr; > xfwd = NULL; > > @@ -242,6 +256,8 @@ > return ngx_http_realip_set_addr(r, &addr); > } > > + ngx_log_error(NGX_LOG_ALERT, r->connection->log, 0, > + "'%V' is not an allowed proxy, declining request", &c->addr_text); And that's certainly an incorrect change. It is quite normal that for some requests the realip module returns NGX_DECLINED, since this essentially means that the realip module is not configured to work with such requests. There is no need to spam logs with alerts about it. > return NGX_DECLINED; > } > Please also not that it's usually a good idea to follow nginx style in patches you submit. See http://nginx.org/en/docs/contributing_changes.html for some basic tips. -- Maxim Dounin http://mdounin.ru/ From pluknet at nginx.com Fri Feb 3 11:12:31 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Fri, 03 Feb 2023 11:12:31 +0000 Subject: [nginx] Style. Message-ID: details: https://hg.nginx.org/nginx/rev/1c3b78d7cdc9 branches: changeset: 8123:1c3b78d7cdc9 user: Maxim Dounin date: Sat Jan 28 05:20:23 2023 +0300 description: Style. diffstat: src/event/ngx_event_connectex.c | 4 ++-- src/http/modules/ngx_http_flv_module.c | 2 +- src/http/modules/ngx_http_static_module.c | 4 ++-- src/http/ngx_http_file_cache.c | 4 ++-- 4 files changed, 7 insertions(+), 7 deletions(-) diffs (54 lines): diff -r 106328a70f4e -r 1c3b78d7cdc9 src/event/ngx_event_connectex.c --- a/src/event/ngx_event_connectex.c Sat Jan 28 01:29:45 2023 +0300 +++ b/src/event/ngx_event_connectex.c Sat Jan 28 05:20:23 2023 +0300 @@ -127,8 +127,8 @@ void ngx_iocp_wait_events(int main) conn[0] = NULL; for ( ;; ) { - offset = (nevents == WSA_MAXIMUM_WAIT_EVENTS + 1) ? 1: 0; - timeout = (nevents == 1 && !first) ? 60000: INFINITE; + offset = (nevents == WSA_MAXIMUM_WAIT_EVENTS + 1) ? 1 : 0; + timeout = (nevents == 1 && !first) ? 60000 : INFINITE; n = WSAWaitForMultipleEvents(nevents - offset, events[offset], 0, timeout, 0); diff -r 106328a70f4e -r 1c3b78d7cdc9 src/http/modules/ngx_http_flv_module.c --- a/src/http/modules/ngx_http_flv_module.c Sat Jan 28 01:29:45 2023 +0300 +++ b/src/http/modules/ngx_http_flv_module.c Sat Jan 28 05:20:23 2023 +0300 @@ -232,7 +232,7 @@ ngx_http_flv_handler(ngx_http_request_t b->file_pos = start; b->file_last = of.size; - b->in_file = b->file_last ? 1: 0; + b->in_file = b->file_last ? 1 : 0; b->last_buf = (r == r->main) ? 1 : 0; b->last_in_chain = 1; diff -r 106328a70f4e -r 1c3b78d7cdc9 src/http/modules/ngx_http_static_module.c --- a/src/http/modules/ngx_http_static_module.c Sat Jan 28 01:29:45 2023 +0300 +++ b/src/http/modules/ngx_http_static_module.c Sat Jan 28 05:20:23 2023 +0300 @@ -265,8 +265,8 @@ ngx_http_static_handler(ngx_http_request b->file_pos = 0; b->file_last = of.size; - b->in_file = b->file_last ? 1: 0; - b->last_buf = (r == r->main) ? 1: 0; + b->in_file = b->file_last ? 1 : 0; + b->last_buf = (r == r->main) ? 1 : 0; b->last_in_chain = 1; b->file->fd = of.fd; diff -r 106328a70f4e -r 1c3b78d7cdc9 src/http/ngx_http_file_cache.c --- a/src/http/ngx_http_file_cache.c Sat Jan 28 01:29:45 2023 +0300 +++ b/src/http/ngx_http_file_cache.c Sat Jan 28 05:20:23 2023 +0300 @@ -1600,8 +1600,8 @@ ngx_http_cache_send(ngx_http_request_t * b->file_pos = c->body_start; b->file_last = c->length; - b->in_file = (c->length - c->body_start) ? 1: 0; - b->last_buf = (r == r->main) ? 1: 0; + b->in_file = (c->length - c->body_start) ? 1 : 0; + b->last_buf = (r == r->main) ? 1 : 0; b->last_in_chain = 1; b->file->fd = c->file.fd; From pluknet at nginx.com Fri Feb 3 11:12:34 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Fri, 03 Feb 2023 11:12:34 +0000 Subject: [nginx] Fixed "zero size buf" alerts with subrequests. Message-ID: details: https://hg.nginx.org/nginx/rev/f5515e727656 branches: changeset: 8124:f5515e727656 user: Maxim Dounin date: Sat Jan 28 05:23:33 2023 +0300 description: Fixed "zero size buf" alerts with subrequests. Since 4611:2b6cb7528409 responses from the gzip static, flv, and mp4 modules can be used with subrequests, though empty files were not properly handled. Empty gzipped, flv, and mp4 files thus resulted in "zero size buf in output" alerts. While valid corresponding files are not expected to be empty, such files shouldn't result in alerts. Fix is to set b->sync on such empty subrequest responses, similarly to what ngx_http_send_special() does. Additionally, the static module, the ngx_http_send_response() function, and file cache are modified to do the same instead of not sending the response body at all in such cases, since not sending the response body at all is believed to be at least questionable, and might break various filters which do not expect such behaviour. diffstat: src/http/modules/ngx_http_flv_module.c | 1 + src/http/modules/ngx_http_gzip_static_module.c | 1 + src/http/modules/ngx_http_mp4_module.c | 1 + src/http/modules/ngx_http_static_module.c | 5 +---- src/http/ngx_http_core_module.c | 5 +---- src/http/ngx_http_file_cache.c | 5 +---- 6 files changed, 6 insertions(+), 12 deletions(-) diffs (99 lines): diff -r 1c3b78d7cdc9 -r f5515e727656 src/http/modules/ngx_http_flv_module.c --- a/src/http/modules/ngx_http_flv_module.c Sat Jan 28 05:20:23 2023 +0300 +++ b/src/http/modules/ngx_http_flv_module.c Sat Jan 28 05:23:33 2023 +0300 @@ -235,6 +235,7 @@ ngx_http_flv_handler(ngx_http_request_t b->in_file = b->file_last ? 1 : 0; b->last_buf = (r == r->main) ? 1 : 0; b->last_in_chain = 1; + b->sync = (b->last_buf || b->in_file) ? 0 : 1; b->file->fd = of.fd; b->file->name = path; diff -r 1c3b78d7cdc9 -r f5515e727656 src/http/modules/ngx_http_gzip_static_module.c --- a/src/http/modules/ngx_http_gzip_static_module.c Sat Jan 28 05:20:23 2023 +0300 +++ b/src/http/modules/ngx_http_gzip_static_module.c Sat Jan 28 05:23:33 2023 +0300 @@ -273,6 +273,7 @@ ngx_http_gzip_static_handler(ngx_http_re b->in_file = b->file_last ? 1 : 0; b->last_buf = (r == r->main) ? 1 : 0; b->last_in_chain = 1; + b->sync = (b->last_buf || b->in_file) ? 0 : 1; b->file->fd = of.fd; b->file->name = path; diff -r 1c3b78d7cdc9 -r f5515e727656 src/http/modules/ngx_http_mp4_module.c --- a/src/http/modules/ngx_http_mp4_module.c Sat Jan 28 05:20:23 2023 +0300 +++ b/src/http/modules/ngx_http_mp4_module.c Sat Jan 28 05:23:33 2023 +0300 @@ -714,6 +714,7 @@ ngx_http_mp4_handler(ngx_http_request_t b->in_file = b->file_last ? 1 : 0; b->last_buf = (r == r->main) ? 1 : 0; b->last_in_chain = 1; + b->sync = (b->last_buf || b->in_file) ? 0 : 1; b->file->fd = of.fd; b->file->name = path; diff -r 1c3b78d7cdc9 -r f5515e727656 src/http/modules/ngx_http_static_module.c --- a/src/http/modules/ngx_http_static_module.c Sat Jan 28 05:20:23 2023 +0300 +++ b/src/http/modules/ngx_http_static_module.c Sat Jan 28 05:23:33 2023 +0300 @@ -238,10 +238,6 @@ ngx_http_static_handler(ngx_http_request return NGX_HTTP_INTERNAL_SERVER_ERROR; } - if (r != r->main && of.size == 0) { - return ngx_http_send_header(r); - } - r->allow_ranges = 1; /* we need to allocate all before the header would be sent */ @@ -268,6 +264,7 @@ ngx_http_static_handler(ngx_http_request b->in_file = b->file_last ? 1 : 0; b->last_buf = (r == r->main) ? 1 : 0; b->last_in_chain = 1; + b->sync = (b->last_buf || b->in_file) ? 0 : 1; b->file->fd = of.fd; b->file->name = path; diff -r 1c3b78d7cdc9 -r f5515e727656 src/http/ngx_http_core_module.c --- a/src/http/ngx_http_core_module.c Sat Jan 28 05:20:23 2023 +0300 +++ b/src/http/ngx_http_core_module.c Sat Jan 28 05:23:33 2023 +0300 @@ -1803,10 +1803,6 @@ ngx_http_send_response(ngx_http_request_ } } - if (r != r->main && val.len == 0) { - return ngx_http_send_header(r); - } - b = ngx_calloc_buf(r->pool); if (b == NULL) { return NGX_HTTP_INTERNAL_SERVER_ERROR; @@ -1817,6 +1813,7 @@ ngx_http_send_response(ngx_http_request_ b->memory = val.len ? 1 : 0; b->last_buf = (r == r->main) ? 1 : 0; b->last_in_chain = 1; + b->sync = (b->last_buf || b->memory) ? 0 : 1; out.buf = b; out.next = NULL; diff -r 1c3b78d7cdc9 -r f5515e727656 src/http/ngx_http_file_cache.c --- a/src/http/ngx_http_file_cache.c Sat Jan 28 05:20:23 2023 +0300 +++ b/src/http/ngx_http_file_cache.c Sat Jan 28 05:23:33 2023 +0300 @@ -1575,10 +1575,6 @@ ngx_http_cache_send(ngx_http_request_t * ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, "http file cache send: %s", c->file.name.data); - if (r != r->main && c->length - c->body_start == 0) { - return ngx_http_send_header(r); - } - /* we need to allocate all before the header would be sent */ b = ngx_calloc_buf(r->pool); @@ -1603,6 +1599,7 @@ ngx_http_cache_send(ngx_http_request_t * b->in_file = (c->length - c->body_start) ? 1 : 0; b->last_buf = (r == r->main) ? 1 : 0; b->last_in_chain = 1; + b->sync = (b->last_buf || b->in_file) ? 0 : 1; b->file->fd = c->file.fd; b->file->name = c->file.name; From pluknet at nginx.com Fri Feb 3 11:12:37 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Fri, 03 Feb 2023 11:12:37 +0000 Subject: [nginx] Lingering close for connections with pipelined requests. Message-ID: details: https://hg.nginx.org/nginx/rev/cffaf3f2eec8 branches: changeset: 8125:cffaf3f2eec8 user: Maxim Dounin date: Thu Feb 02 23:38:48 2023 +0300 description: Lingering close for connections with pipelined requests. This is expected to help with clients using pipelining with some constant depth, such as apt[1][2]. When downloading many resources, apt uses pipelining with some constant depth, a number of requests in flight. This essentially means that after receiving a response it sends an additional request to the server, and this can result in requests arriving to the server at any time. Further, additional requests are sent one-by-one, and can be easily seen as such (neither as pipelined, nor followed by pipelined requests). The only safe approach to close such connections (for example, when keepalive_requests is reached) is with lingering. To do so, now nginx monitors if pipelining was used on the connection, and if it was, closes the connection with lingering. [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=973861#10 [2] https://mailman.nginx.org/pipermail/nginx-devel/2023-January/ZA2SP5SJU55LHEBCJMFDB2AZVELRLTHI.html diffstat: src/core/ngx_connection.h | 1 + src/http/ngx_http_request.c | 4 +++- 2 files changed, 4 insertions(+), 1 deletions(-) diffs (32 lines): diff -r f5515e727656 -r cffaf3f2eec8 src/core/ngx_connection.h --- a/src/core/ngx_connection.h Sat Jan 28 05:23:33 2023 +0300 +++ b/src/core/ngx_connection.h Thu Feb 02 23:38:48 2023 +0300 @@ -172,6 +172,7 @@ struct ngx_connection_s { unsigned timedout:1; unsigned error:1; unsigned destroyed:1; + unsigned pipeline:1; unsigned idle:1; unsigned reusable:1; diff -r f5515e727656 -r cffaf3f2eec8 src/http/ngx_http_request.c --- a/src/http/ngx_http_request.c Sat Jan 28 05:23:33 2023 +0300 +++ b/src/http/ngx_http_request.c Thu Feb 02 23:38:48 2023 +0300 @@ -2753,7 +2753,8 @@ ngx_http_finalize_connection(ngx_http_re || (clcf->lingering_close == NGX_HTTP_LINGERING_ON && (r->lingering_close || r->header_in->pos < r->header_in->last - || r->connection->read->ready))) + || r->connection->read->ready + || r->connection->pipeline))) { ngx_http_set_lingering_close(r->connection); return; @@ -3123,6 +3124,7 @@ ngx_http_set_keepalive(ngx_http_request_ c->sent = 0; c->destroyed = 0; + c->pipeline = 1; if (rev->timer_set) { ngx_del_timer(rev); From v.zhestikov at f5.com Fri Feb 3 14:42:10 2023 From: v.zhestikov at f5.com (Vadim Zhestikov) Date: Fri, 03 Feb 2023 14:42:10 +0000 Subject: [njs] Added String.prototype.replaceAll(). Message-ID: details: https://hg.nginx.org/njs/rev/69547623da14 branches: changeset: 2038:69547623da14 user: Vadim Zhestikov date: Fri Feb 03 06:41:01 2023 -0800 description: Added String.prototype.replaceAll(). diffstat: src/njs_regexp.c | 2 +- src/njs_regexp.h | 2 + src/njs_string.c | 218 +++++++++++++++++++++++-------- src/test/njs_unit_test.c | 319 +++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 484 insertions(+), 57 deletions(-) diffs (927 lines): diff -r 286675dcfbc5 -r 69547623da14 src/njs_regexp.c --- a/src/njs_regexp.c Thu Feb 02 10:01:26 2023 -0800 +++ b/src/njs_regexp.c Fri Feb 03 06:41:01 2023 -0800 @@ -1193,7 +1193,7 @@ njs_regexp_string_create(njs_vm_t *vm, n } -static njs_int_t +njs_int_t njs_regexp_prototype_symbol_replace(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, njs_index_t unused) { diff -r 286675dcfbc5 -r 69547623da14 src/njs_regexp.h --- a/src/njs_regexp.h Thu Feb 02 10:01:26 2023 -0800 +++ b/src/njs_regexp.h Fri Feb 03 06:41:01 2023 -0800 @@ -21,6 +21,8 @@ njs_int_t njs_regexp_exec(njs_vm_t *vm, njs_value_t *retval); njs_int_t njs_regexp_prototype_exec(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, njs_index_t unused); +njs_int_t njs_regexp_prototype_symbol_replace(njs_vm_t *vm, + njs_value_t *args, njs_uint_t nargs, njs_index_t unused); njs_int_t njs_regexp_to_string(njs_vm_t *vm, njs_value_t *retval, const njs_value_t *regexp); diff -r 286675dcfbc5 -r 69547623da14 src/njs_string.c --- a/src/njs_string.c Thu Feb 02 10:01:26 2023 -0800 +++ b/src/njs_string.c Fri Feb 03 06:41:01 2023 -0800 @@ -3667,21 +3667,25 @@ exception: static njs_int_t njs_string_prototype_replace(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, - njs_index_t unused) + njs_index_t replaceAll) { u_char *r; - size_t length, size; + size_t length, size, increment, end_of_last_match; int64_t pos; njs_int_t ret; + njs_str_t str; + njs_chb_t chain; + njs_bool_t is_byte_or_ascii_string; njs_value_t *this, *search, *replace; njs_value_t search_lvalue, replace_lvalue, replacer, retval, arguments[3]; - const u_char *p; + const u_char *p, *p_start; njs_function_t *func_replace; njs_string_prop_t string, s, ret_string; static const njs_value_t replace_key = njs_wellknown_symbol(NJS_SYMBOL_REPLACE); + static const njs_value_t string_flags = njs_string("flags"); this = njs_argument(args, 0); @@ -3702,8 +3706,35 @@ njs_string_prototype_replace(njs_vm_t *v } if (njs_is_defined(&replacer)) { - arguments[0] = *this; - arguments[1] = *replace; + njs_value_assign(&arguments[0], this); + njs_value_assign(&arguments[1], replace); + + if (replaceAll && njs_function(&replacer)->native + && njs_function(&replacer)->u.native == + njs_regexp_prototype_symbol_replace) + { + ret = njs_value_property(vm, search, + njs_value_arg(&string_flags), &retval); + if (njs_slow_path(ret == NJS_ERROR)) { + return NJS_ERROR; + } + + ret = njs_value_to_string(vm, &retval, &retval); + if (njs_slow_path(ret != NJS_OK)) { + return NJS_ERROR; + } + + njs_string_get(&retval, &str); + + if (njs_strlchr(str.start, str.start + str.length, 'g') + == NULL) + { + njs_type_error(vm, "String.prototype.replaceAll" + " called with a non-global RegExp argument", + njs_type_string(this->type)); + return NJS_ERROR; + } + } return njs_function_call(vm, njs_function(&replacer), search, arguments, 2, &vm->retval); @@ -3727,6 +3758,10 @@ njs_string_prototype_replace(njs_vm_t *v if (njs_slow_path(ret != NJS_OK)) { return ret; } + + } else { + njs_value_assign(&arguments[0], search); + njs_value_assign(&arguments[2], this); } (void) njs_string_prop(&string, this); @@ -3734,69 +3769,138 @@ njs_string_prototype_replace(njs_vm_t *v pos = njs_string_index_of(&string, &s, 0); if (pos < 0) { - vm->retval = *this; + njs_value_assign(&vm->retval, this); return NJS_OK; } - if (func_replace == NULL) { - ret = njs_string_get_substitution(vm, search, this, pos, NULL, 0, NULL, - replace, &retval); - if (njs_slow_path(ret != NJS_OK)) { - return ret; + if (!replaceAll) { + + if (func_replace == NULL) { + ret = njs_string_get_substitution(vm, search, this, pos, NULL, 0, + NULL, replace, &retval); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } + + } else { + njs_set_number(&arguments[1], pos); + + ret = njs_function_call(vm, func_replace, + njs_value_arg(&njs_value_undefined), + arguments, 3, &retval); + + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } + + ret = njs_value_to_string(vm, &retval, &retval); + if (njs_slow_path(ret != NJS_OK)) { + return NJS_ERROR; + } } - } else { - arguments[0] = *search; - njs_set_number(&arguments[1], pos); - arguments[2] = *this; - - ret = njs_function_call(vm, func_replace, - njs_value_arg(&njs_value_undefined), - arguments, 3, &retval); - - if (njs_slow_path(ret != NJS_OK)) { - return ret; + if (njs_is_byte_or_ascii_string(&string)) { + p = string.start + pos; + + } else { + /* UTF-8 string. */ + p = njs_string_offset(string.start, string.start + string.size, + pos); } - ret = njs_value_to_string(vm, &retval, &retval); - if (njs_slow_path(ret != NJS_OK)) { + (void) njs_string_prop(&ret_string, &retval); + + size = string.size + ret_string.size - s.size; + length = string.length + ret_string.length - s.length; + + if (njs_is_byte_string(&string) + || njs_is_byte_string(&s) + || njs_is_byte_string(&ret_string)) + { + length = 0; + } + + r = njs_string_alloc(vm, &vm->retval, size, length); + if (njs_slow_path(r == NULL)) { return NJS_ERROR; } - } - - if (njs_is_byte_or_ascii_string(&string)) { - p = string.start + pos; - - } else { - /* UTF-8 string. */ - p = njs_string_offset(string.start, string.start + string.size, pos); - } - - (void) njs_string_prop(&ret_string, &retval); - - size = string.size + ret_string.size - s.size; - length = string.length + ret_string.length - s.length; - - if (njs_is_byte_string(&string) - || njs_is_byte_string(&s) - || njs_is_byte_string(&ret_string)) - { - length = 0; - } - - r = njs_string_alloc(vm, &vm->retval, size, length); - if (njs_slow_path(r == NULL)) { - return NJS_ERROR; - } - - r = njs_cpymem(r, string.start, p - string.start); - r = njs_cpymem(r, ret_string.start, ret_string.size); - memcpy(r, p + s.size, string.size - s.size - (p - string.start)); - - return NJS_OK; + + r = njs_cpymem(r, string.start, p - string.start); + r = njs_cpymem(r, ret_string.start, ret_string.size); + memcpy(r, p + s.size, string.size - s.size - (p - string.start)); + + return NJS_OK; + } + + njs_chb_init(&chain, vm->mem_pool); + + p_start = string.start; + increment = s.length != 0 ? s.length : 1; + is_byte_or_ascii_string = njs_is_byte_or_ascii_string(&string); + + do { + if (func_replace == NULL) { + ret = njs_string_get_substitution(vm, search, this, pos, NULL, 0, + NULL, replace, &retval); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } + + } else { + njs_set_number(&arguments[1], pos); + + ret = njs_function_call(vm, func_replace, + njs_value_arg(&njs_value_undefined), + arguments, 3, &retval); + + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } + + ret = njs_value_to_string(vm, &retval, &retval); + if (njs_slow_path(ret != NJS_OK)) { + return NJS_ERROR; + } + } + + if (is_byte_or_ascii_string) { + p = string.start + pos; + + } else { + /* UTF-8 string. */ + p = njs_string_offset(string.start, string.start + string.size, + pos); + } + + (void) njs_string_prop(&ret_string, &retval); + + njs_chb_append(&chain, p_start, p - p_start); + njs_chb_append(&chain, ret_string.start, ret_string.size); + + p_start = p + s.size; + end_of_last_match = pos + increment; + + pos = njs_string_index_of(&string, &s, end_of_last_match); + + } while (pos >= 0 && end_of_last_match <= string.length); + + njs_chb_append(&chain, p_start, string.start + string.size - p_start); + + ret = njs_string_create_chb(vm, &vm->retval, &chain); + if (njs_slow_path(ret != NJS_OK)) { + ret = NJS_ERROR; + goto exception; + } + +exception: + + njs_chb_destroy(&chain); + + return ret; } + static njs_int_t njs_string_prototype_iterator_obj(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, njs_index_t kind) @@ -4090,6 +4194,8 @@ static const njs_object_prop_t njs_stri NJS_DECLARE_PROP_NATIVE("replace", njs_string_prototype_replace, 2, 0), + NJS_DECLARE_PROP_NATIVE("replaceAll", njs_string_prototype_replace, 2, 1), + { .type = NJS_PROPERTY, .name = njs_wellknown_symbol(NJS_SYMBOL_ITERATOR), diff -r 286675dcfbc5 -r 69547623da14 src/test/njs_unit_test.c --- a/src/test/njs_unit_test.c Thu Feb 02 10:01:26 2023 -0800 +++ b/src/test/njs_unit_test.c Fri Feb 03 06:41:01 2023 -0800 @@ -8750,198 +8750,397 @@ static njs_unit_test_t njs_test[] = { njs_str("'abc'.replace()"), njs_str("abc") }, + { njs_str("'abc'.replaceAll()"), + njs_str("abc") }, + { njs_str("'ABC'.replace('B')"), njs_str("AundefinedC") }, + { njs_str("'ABCB'.replaceAll('B')"), + njs_str("AundefinedCundefined") }, + { njs_str("'ABC'.replace('B', undefined)"), njs_str("AundefinedC") }, + { njs_str("'ABBC'.replaceAll('B', undefined)"), + njs_str("AundefinedundefinedC") }, + { njs_str("'a'.repeat(16).replace('a'.repeat(17)) === 'a'.repeat(16)"), njs_str("true") }, + { njs_str("'a'.repeat(16).replaceAll('a'.repeat(17)) === 'a'.repeat(16)"), + njs_str("true") }, + { njs_str("'α'.repeat(16).replace('α'.repeat(17)) === 'α'.repeat(16)"), njs_str("true") }, + { njs_str("'α'.repeat(16).replaceAll('α'.repeat(17)) === 'α'.repeat(16)"), + njs_str("true") }, + { njs_str("'ABC'.replace('B', null)"), njs_str("AnullC") }, + { njs_str("'BABCB'.replaceAll('B', null)"), + njs_str("nullAnullCnull") }, + { njs_str("'abc'.replace('c', 1)"), njs_str("ab1") }, + { njs_str("'cacbc'.replaceAll('c', 1)"), + njs_str("1a1b1") }, + { njs_str("'abc'.replace('a', 'X')"), njs_str("Xbc") }, + { njs_str("'cabac'.replaceAll('a', 'X')"), + njs_str("cXbXc") }, + { njs_str("'abc'.replace('b', 'X')"), njs_str("aXc") }, + { njs_str("'abc'.replaceAll('b', 'X')"), + njs_str("aXc") }, + { njs_str("('a'.repeat(33) + 'bb').replace('bb', 'CC').slice(31)"), njs_str("aaCC") }, + { njs_str("('a'.repeat(33) + 'bb').replaceAll('b', 'C').slice(31)"), + njs_str("aaCC") }, + { njs_str("var r = 'abc'.replace('c', 'X'); [r, r.length]"), njs_str("abX,3") }, + { njs_str("var r = 'acbc'.replaceAll('c', 'X'); [r, r.length]"), + njs_str("aXbX,4") }, + { njs_str("var r = 'αβγ'.replace('α', 'X'); [r, r.length]"), njs_str("Xβγ,3") }, + { njs_str("var r = 'αβαγα'.replaceAll('α', 'X'); [r, r.length]"), + njs_str("XβXγX,5") }, + { njs_str("var r = 'αβγ'.replace('β', 'X'); [r, r.length]"), njs_str("αXγ,3") }, + { njs_str("var r = 'αβγ'.replaceAll('β', 'X'); [r, r.length]"), + njs_str("αXγ,3") }, + { njs_str("var r = 'αβγ'.replace('γ', 'X'); [r, r.length]"), njs_str("αβX,3") }, + { njs_str("var r = 'αβγγ'.replaceAll('γ', 'X'); [r, r.length]"), + njs_str("αβXX,4") }, + { njs_str("var r = 'αβγ'.replace('', 'X'); [r, r.length]"), njs_str("Xαβγ,4") }, + { njs_str("var r = 'αβγ'.replaceAll('', 'X'); [r, r.length]"), + njs_str("XαXβXγX,7") }, + { njs_str("var s = 'αz'.toUTF8();" "var r = s.replace('z', 'β');" "r.length"), njs_str("4") }, + { njs_str("var s = 'αzz'.toUTF8();" + "var r = s.replaceAll('z', 'β');" + "r.length"), + njs_str("3") }, + { njs_str("'abc'.replace('b', (m, o, s) => `|${s}|${o}|${m}|`)"), njs_str("a|abc|1|b|c") }, + { njs_str("'abcb'.replaceAll('b', (m, o, s) => `|${s}|${o}|${m}|`)"), + njs_str("a|abcb|1|b|c|abcb|3|b|") }, + { njs_str("'abcdbe'.replace('b', '|$`X$\\'|')"), njs_str("a|aXcdbe|cdbe") }, + { njs_str("'abcdbe'.replaceAll('b', '|$`X$\\'|')"), + njs_str("a|aXcdbe|cd|abcdXe|e") }, + + { njs_str("'ABC'.replace('B', '$')"), njs_str("A$C") }, + { njs_str("'ABCB'.replaceAll('B', '$')"), + njs_str("A$C$") }, + { njs_str("'ABC'.replace('B', '$23')"), njs_str("A$23C") }, + { njs_str("'ABBC'.replaceAll('B', '$23')"), + njs_str("A$23$23C") }, + { njs_str("'undefined'.replace(void 0, 'x')"), njs_str("x") }, + { njs_str("'undefinedundefined'.replaceAll(void 0, 'x')"), + njs_str("xx") }, + { njs_str("'12345'.replace(3, () => 0)"), njs_str("12045") }, + { njs_str("'123435'.replaceAll(3, () => 0)"), + njs_str("120405") }, + { njs_str("var r = new String('undefined').replace(x, Function('return arguments[1]+42;')); var x; r"), njs_str("42") }, + { njs_str("var r = new String('undefined undefined').replaceAll(x, Function('return arguments[1]+42;')); var x; r"), + njs_str("42 52") }, + { njs_str("'123'.replace(3, function() { return {toString: ()=>({})}; })"), njs_str("TypeError: Cannot convert object to primitive value") }, + { njs_str("'123'.replaceAll(3, function() { return {toString: ()=>({})}; })"), + njs_str("TypeError: Cannot convert object to primitive value") }, + { njs_str("'12345'.replace(3, () => ({toString: () => 'aaaa'}))"), njs_str("12aaaa45") }, + { njs_str("'123435'.replaceAll(3, () => ({toString: () => 'aaaa'}))"), + njs_str("12aaaa4aaaa5") }, + { njs_str("'ABC'.replace('B', () => {throw 'OOps'})"), njs_str("OOps") }, + { njs_str("'ABCB'.replaceAll('B', () => {throw 'OOps'})"), + njs_str("OOps") }, + { njs_str("'abc'.replace(/a/, 'X')"), njs_str("Xbc") }, + { njs_str("'abc'.replaceAll(/a/, 'X')"), + njs_str("TypeError: String.prototype.replaceAll called with a non-global RegExp argument") }, + + { njs_str("'abac'.replaceAll(/a/g, 'X')"), + njs_str("XbXc") }, + { njs_str("'abccd'.replace(/c/, 'X')"), njs_str("abXcd") }, + { njs_str("'abccd'.replaceAll(/c/g, 'X')"), + njs_str("abXXd") }, + { njs_str("'abc'.replace(/c/, 'X')"), njs_str("abX") }, + { njs_str("'abc'.replaceAll(/c/g, 'X')"), + njs_str("abX") }, + { njs_str("'abccd'.replace(/c+/, 'X')"), njs_str("abXd") }, + { njs_str("'acccbccd'.replaceAll(/c+/g, 'X')"), + njs_str("aXbXd") }, + { njs_str("'abc'.replace(/f/, 'X')"), njs_str("abc") }, + { njs_str("'abc'.replaceAll(/f/g, 'X')"), + njs_str("abc") }, + { njs_str("'AB=C==='.replace(/=*$/, '')"), njs_str("AB=C") }, + { njs_str("'AB=C==='.replaceAll(/=*$/g, '')"), + njs_str("AB=C") }, + { njs_str("('a'.repeat(33) + 'bb').replace(/bb/, 'CC').slice(31)"), njs_str("aaCC") }, + { njs_str("('a'.repeat(33) + 'bb').replaceAll(/b/g, 'C').slice(31)"), + njs_str("aaCC") }, + { njs_str("'abccd'.replace(/c/g, 'X')"), njs_str("abXXd") }, + { njs_str("'abccd'.replaceAll(/c/g, 'X')"), + njs_str("abXXd") }, + { njs_str("('a'.repeat(33) + 'bb').replace(/bb/g, 'CC').slice(31)"), njs_str("aaCC") }, + { njs_str("('a'.repeat(33) + 'bb').replaceAll(/bb/g, 'CC').slice(31)"), + njs_str("aaCC") }, + { njs_str("'abccd'.replace(/[ac]/g, 'X')"), njs_str("XbXXd") }, + { njs_str("'abccd'.replaceAll(/[ac]/g, 'X')"), + njs_str("XbXXd") }, + { njs_str("'ab'.replace(/q*/g, 'X')"), njs_str("XaXbX") }, + { njs_str("'ab'.replaceAll(/q*/g, 'X')"), + njs_str("XaXbX") }, + { njs_str("'αβ'.replace(/q*/g, 'X')"), njs_str("XαXβX") }, + { njs_str("'αβ'.replaceAll(/q*/g, 'X')"), + njs_str("XαXβX") }, + { njs_str("'αβ'.replace(/(q)*/g, 'X')"), njs_str("XαXβX") }, + { njs_str("'αβ'.replaceAll(/(q)*/g, 'X')"), + njs_str("XαXβX") }, + { njs_str("'αβ'.replace(/q*/g, 'γ')"), njs_str("γαγβγ") }, + { njs_str("'αβ'.replaceAll(/q*/g, 'γ')"), + njs_str("γαγβγ") }, + { njs_str("':α:β:γ:'.replace(/:/g, '')"), njs_str("αβγ") }, + { njs_str("':α:β:γ:'.replaceAll(/:/g, '')"), + njs_str("αβγ") }, + { njs_str("':α:β:γ:'.replace(/[αβγ]/g, '')"), njs_str("::::") }, + { njs_str("':α:β:γ:'.replaceAll(/[αβγ]/g, '')"), + njs_str("::::") }, + { njs_str("'aabbccaa'.replace(/a*/g, '')"), njs_str("bbcc") }, + { njs_str("'aabbccaa'.replaceAll(/a*/g, '')"), + njs_str("bbcc") }, + { njs_str("'aabbccaab'.replace(/z*/g, '')"), njs_str("aabbccaab") }, + { njs_str("'aabbccaab'.replaceAll(/z*/g, '')"), + njs_str("aabbccaab") }, + { njs_str("''.replace(/a*/g, '')"), njs_str("") }, + { njs_str("''.replaceAll(/a*/g, '')"), + njs_str("") }, + { njs_str("'abcde'.replace(/d/, (m, o, s) => `|${s}|${o}|${m}|`)"), njs_str("abc|abcde|3|d|e") }, + { njs_str("'abcdde'.replaceAll(/d/g, (m, o, s) => `|${s}|${o}|${m}|`)"), + njs_str("abc|abcdde|3|d||abcdde|4|d|e") }, + { njs_str("'abcde'.replace(/(d)/, (m, p, o, s) => `|${s}|${o}|${m}|${p}|`)"), njs_str("abc|abcde|3|d|d|e") }, + { njs_str("'abcded'.replaceAll(/(d)/g, (m, p, o, s) => `|${s}|${o}|${m}|${p}|`)"), + njs_str("abc|abcded|3|d|d|e|abcded|5|d|d|") }, + { njs_str("'abc'.replace(/b/, () => 1)"), njs_str("a1c") }, + { njs_str("'abcb'.replaceAll(/b/g, () => 1)"), + njs_str("a1c1") }, + { njs_str("var n = 0; 'abbbc'.replace(/b/g, () => ++n)"), njs_str("a123c") }, + { njs_str("var n = 0; 'abbbc'.replaceAll(/b/g, () => ++n)"), + njs_str("a123c") }, + { njs_str("'abc'.replace(/x/, (m, o, s) => `|${s}|${o}|${m}|`)"), njs_str("abc") }, + { njs_str("'abc'.replaceAll(/x/g, (m, o, s) => `|${s}|${o}|${m}|`)"), + njs_str("abc") }, + { njs_str("'abc12345#$*%'.replace(/([^\\d]*)(\\d*)([^\\w]*)/," " (_, p1, p2, p3) => [p1, p2, p3].join('-'))"), njs_str("abc-12345-#$*%") }, + { njs_str("'abc12345#$*%'.replaceAll(/([^\\d]*)(\\d*)([^\\w]*)/g," + " (_, p1, p2, p3) => [p1, p2, p3].join('-'))"), + njs_str("abc-12345-#$*%--") }, + { njs_str("'abc'.replace(/(?b)/, (m, p, o, s, gr) => `|${gr.named}|`)"), njs_str("a|b|c") }, + { njs_str("'abcb'.replaceAll(/(?b)/g, (m, p, o, s, gr) => `|${gr.named}|`)"), + njs_str("a|b|c|b|") }, + { njs_str("'ABC'.replace(/[A-Z]/g, m => '-' + m.toLowerCase())"), njs_str("-a-b-c") }, + { njs_str("'ABC'.replaceAll(/[A-Z]/g, m => '-' + m.toLowerCase())"), + njs_str("-a-b-c") }, + { njs_str("'abc'.replace(/(b)c/g, '|$01|')"), njs_str("a|b|") }, + { njs_str("'abc'.replaceAll(/(b)c/g, '|$01|')"), + njs_str("a|b|") }, + { njs_str("'abc'.replace(/(b)c/g, '@$0|$01|$00@')"), njs_str("a@$0|b|$00@") }, + { njs_str("'abc'.replaceAll(/(b)c/g, '@$0|$01|$00@')"), + njs_str("a@$0|b|$00@") }, + { njs_str("'abcdeFGHIJ'.replace(/(a)(b)(c)(d)(e)(F)(G)(H)(I)(J)/, '$9|$10|$11|$01')"), njs_str("I|J|a1|a") }, + { njs_str("'abcdeFGHIJ abcdeFGHIJ'.replaceAll(/(a)(b)(c)(d)(e)(F)(G)(H)(I)(J)/g, '$9|$10|$11|$01')"), + njs_str("I|J|a1|a I|J|a1|a") }, + { njs_str("'abcdbe'.replace(/(b)/g, '$2$23')"), njs_str("a$2$23cd$2$23e") }, + { njs_str("'abcdbe'.replaceAll(/(b)/g, '$2$23')"), + njs_str("a$2$23cd$2$23e") }, + { njs_str("'abcdbe'.replace(/(b)/g, '$2$23X$$Y')"), njs_str("a$2$23X$Ycd$2$23X$Ye") }, + { njs_str("'abcdbe'.replaceAll(/(b)/g, '$2$23X$$Y')"), + njs_str("a$2$23X$Ycd$2$23X$Ye") }, + { njs_str("'abcdbe'.replace(/b/, '|$`X$\\'|')"), njs_str("a|aXcdbe|cdbe") }, + { njs_str("'abcdbe'.replaceAll(/b/g, '|$`X$\\'|')"), + njs_str("a|aXcdbe|cd|abcdXe|e") }, + { njs_str("'abcdbefbgh'.replace(/b/g, '|$`X$\\'|')"), njs_str("a|aXcdbefbgh|cd|abcdXefbgh|ef|abcdbefXgh|gh") }, + { njs_str("'abcdbefbgh'.replaceAll(/b/g, '|$`X$\\'|')"), + njs_str("a|aXcdbefbgh|cd|abcdXefbgh|ef|abcdbefXgh|gh") }, + { njs_str("'abc12345#$*%'.replace(/([^\\d]*)(\\d*)([^\\w]*)/, '$1-$2-$3')"), njs_str("abc-12345-#$*%") }, + { njs_str("'abc12345#$*%'.replaceAll(/([^\\d]*)(\\d*)([^\\w]*)/g, '$1-$2-$3')"), + njs_str("abc-12345-#$*%--") }, + { njs_str("'$1,$2'.replace(/(\\$(\\d))/g, '$$1-$1$2')"), njs_str("$1-$11,$1-$22") }, + { njs_str("'$1,$2'.replaceAll(/(\\$(\\d))/g, '$$1-$1$2')"), + njs_str("$1-$11,$1-$22") }, + { njs_str("'ABC'.replace(/(h*)(z*)(g*)/g, '$1@$2α$3')"), njs_str("@αA@αB@αC@α") }, + { njs_str("'ABC'.replaceAll(/(h*)(z*)(g*)/g, '$1@$2α$3')"), + njs_str("@αA@αB@αC@α") }, + { njs_str("'abc'.replace(/(h*)(z*)/g, '$1@$2#$3:')"), njs_str("@#$3:a@#$3:b@#$3:c@#$3:") }, + { njs_str("'abc'.replaceAll(/(h*)(z*)/g, '$1@$2#$3:')"), + njs_str("@#$3:a@#$3:b@#$3:c@#$3:") }, + { njs_str("/b(c)(z)?(.)/[Symbol.replace]('abcde', '[$1$2$3]')"), njs_str("a[cd]e") }, @@ -8963,60 +9162,117 @@ static njs_unit_test_t njs_test[] = { njs_str("'α'.replace(/(h*)/g, '$1βγ')"), njs_str("βγαβγ") }, + { njs_str("'α'.replaceAll(/(h*)/g, '$1βγ')"), + njs_str("βγαβγ") }, + { njs_str("'αg'.replace(/(h*)/g, '$1βγ')"), njs_str("βγαβγgβγ") }, + { njs_str("'αg'.replaceAll(/(h*)/g, '$1βγ')"), + njs_str("βγαβγgβγ") }, + { njs_str("'αg'.replace(/(α*)/g, '$1βγ')"), njs_str("αβγβγgβγ") }, + { njs_str("'αg'.replaceAll(/(α*)/g, '$1βγ')"), + njs_str("αβγβγgβγ") }, + { njs_str("'αg'.replace(/(h*)/g, 'fg$1βγ')"), njs_str("fgβγαfgβγgfgβγ") }, + { njs_str("'αg'.replaceAll(/(h*)/g, 'fg$1βγ')"), + njs_str("fgβγαfgβγgfgβγ") }, + { njs_str("'αgβfγ'.replace(/(gβ)/g, 'n$1i')"), njs_str("αngβifγ") }, + { njs_str("'αgβfγ'.replaceAll(/(gβ)/g, 'n$1i')"), + njs_str("αngβifγ") }, + { njs_str("'abc'.replace(/b/g, '|$&|')"), njs_str("a|b|c") }, + { njs_str("'abc'.replaceAll(/b/g, '|$&|')"), + njs_str("a|b|c") }, + { njs_str("'ABC'.replace(/((A)B)/g, '($1|$&|$2)')"), njs_str("(AB|AB|A)C") }, + { njs_str("'ABC'.replaceAll(/((A)B)/g, '($1|$&|$2)')"), + njs_str("(AB|AB|A)C") }, + { njs_str("'abc'.replace(/b/g, '$0')"), njs_str("a$0c") }, + { njs_str("'abc'.replaceAll(/b/g, '$0')"), + njs_str("a$0c") }, + { njs_str("'abc'.replace(/^/g, '|$&|')"), njs_str("||abc") }, + { njs_str("'abc'.replaceAll(/^/g, '|$&|')"), + njs_str("||abc") }, + { njs_str("var uri ='/u/v1/Aa/bB?type=m3u8&mt=42';" "uri.replace(/^\\/u\\/v1\\/[^/]*\\/([^\?]*)\\?.*(mt=[^&]*).*$/, '$1|$2')"), njs_str("bB|mt=42") }, + { njs_str("var uri ='/u/v1/Aa/bB?type=m3u8&mt=42 /u/v1/Aa/bB?type=m3u8&mt=43';" + "uri.replaceAll(/^\\/u\\/v1\\/[^/]*\\/([^\?]*)\\?.*(mt=[^&]*).*$/g, '$1|$2')"), + njs_str("bB|mt=43") }, + { njs_str("'ABC'.replace(/B/, '$')"), njs_str("A$C") }, + { njs_str("'ABBC'.replaceAll(/B/g, '$')"), + njs_str("A$$C") }, + { njs_str("'ABC'.replace(/(?B)/, '|$|@$@')"), njs_str("A|B|@@C") }, + { njs_str("'ABBC'.replaceAll(/(?B)/g, '|$|@$@')"), + njs_str("A|B|@@|B|@@C") }, + { njs_str("'ABC'.replace(/(?B)/, '|$B)/g, '|$B)/, '|$@')"), njs_str("A|@C") }, + { njs_str("'ABCB'.replaceAll(/(?B)/g, '|$@')"), + njs_str("A|@C|@") }, + { njs_str("('β' + 'α'.repeat(33)+'β').replace(/(α+)(β+)/, (m, p1) => p1[32])"), njs_str("βα") }, + { njs_str("('β' + 'α'.repeat(33)+'β').replaceAll(/(α+)(β+)/g, (m, p1) => p1[32])"), + njs_str("βα") }, + { njs_str("'abc'.replace(/(z*)/g, () => '@')"), njs_str("@a at b@c@") }, + { njs_str("'abc'.replaceAll(/(z*)/g, () => '@')"), + njs_str("@a at b@c@") }, + { njs_str("'abc'.replace(/(a*)/g, () => '@')"), njs_str("@@b at c@") }, + { njs_str("'abc'.replaceAll(/(a*)/g, () => '@')"), + njs_str("@@b at c@") }, + { njs_str("var O = RegExp.prototype[Symbol.replace];" "RegExp.prototype[Symbol.replace] = function (s, rep) { return O.call(this, s, `|${rep}|`); };" "'ABC'.replace(/B/, '+')"), njs_str("A|+|C") }, + { njs_str("var O = RegExp.prototype[Symbol.replace];" + "RegExp.prototype[Symbol.replace] = function (s, rep) { return O.call(this, s, `|${rep}|`); };" + "'ABC'.replaceAll(/B/g, '+')"), + njs_str("A|+|C") }, + { njs_str("var O = RegExp.prototype.exec;" "function mangled(s) { var r = O.call(this, s);" " Object.defineProperty(r, '0', {enumerable:false}); " @@ -9027,6 +9283,14 @@ static njs_unit_test_t njs_test[] = { njs_str("var O = RegExp.prototype.exec;" "function mangled(s) { var r = O.call(this, s);" + " Object.defineProperty(r, '0', {enumerable:false}); " + " return r; };" + "RegExp.prototype.exec = mangled;" + "'ABC'.replaceAll(/(B)/g, (m, p1, off, s) => `@${m}|${p1}|${off}|${s}@`)"), + njs_str("TypeError: Object.defineProperty is called on non-object") }, + + { njs_str("var O = RegExp.prototype.exec;" + "function mangled(s) { var r = O.call(this, s);" " Object.defineProperty(r, 'groups', {value: {g:1}}); " " return r; };" "RegExp.prototype.exec = mangled;" @@ -9035,12 +9299,28 @@ static njs_unit_test_t njs_test[] = { njs_str("var O = RegExp.prototype.exec;" "function mangled(s) { var r = O.call(this, s);" + " Object.defineProperty(r, 'groups', {value: {g:1}}); " + " return r; };" + "RegExp.prototype.exec = mangled;" + "'ABC'.replaceAll(/(B)/g, '$')"), + njs_str("TypeError: Object.defineProperty is called on non-object") }, + + { njs_str("var O = RegExp.prototype.exec;" + "function mangled(s) { var r = O.call(this, s);" " Object.defineProperty(r, 'groups', {value: {get g() {throw 'OOps'}}}); " " return r; };" "RegExp.prototype.exec = mangled;" "'ABC'.replace(/(B)/, '$')"), njs_str("OOps") }, + { njs_str("var O = RegExp.prototype.exec;" + "function mangled(s) { var r = O.call(this, s);" + " Object.defineProperty(r, 'groups', {value: {get g() {throw 'OOps'}}}); " + " return r; };" + "RegExp.prototype.exec = mangled;" + "'ABC'.replaceAll(/(B)/g, '$')"), + njs_str("TypeError: Object.defineProperty is called on non-object") }, + { njs_str("var name = /a/g[Symbol.replace].name; [name, typeof name]"), njs_str("[Symbol.replace],string") }, @@ -9077,6 +9357,17 @@ static njs_unit_test_t njs_test[] = "'any_string'.replace(re)"), njs_str("undefinedg") }, + { njs_str("var cnt = 0;" + "var a = [];" + "a[2] = '';" + "var re = /any_regexp/g;" + "re.exec = function () {" + " if (cnt++ > 1) return null;" + " return a;" + "};" + "'any_string'.replaceAll(re)"), + njs_str("undefinedg") }, + { njs_str("var a = [];" "a[2] = {toString() {a[2**20] = 1; return 'X';}}; " "a[4] = 'Y';" @@ -9100,6 +9391,18 @@ static njs_unit_test_t njs_test[] = "'abc'.replace(re, '@$1|$2|$3|$4|$99|$100|@')"), njs_str("@|X||Y|Z|0|@") }, + { njs_str("var cnt = 0;" + "var a = [];" + "a[2] = {toString() {a[2**20] = 1; return 'X';}}; " + "a[4] = 'Y';" + "a[99] = 'Z';" + "a[100] = '*';" + "a[200] = '!';" + "var re = /b/g;" + "re.exec = () => {if (cnt++ > 1) return null; return a};" + "'abc'.replaceAll(re, '@$1|$2|$3|$4|$99|$100|@')"), + njs_str("@|X||Y|Z|0|@") }, + { njs_str("var a = [];" "Object.defineProperty(a, 32768, {});" "var re = /any_regexp/;" @@ -9120,6 +9423,22 @@ static njs_unit_test_t njs_test[] = "'any_string'.replace(re)"), njs_str("undefinedg") }, + { njs_str("var cnt = 0;" + "var a = [];" + "Object.defineProperty(a, 32768, {});" + "var re = /any_regexp/g;" + "re.exec = function () {" + " if (cnt++ > 1) return null;" + " return a;" + "};" + "'any_string'.replace(re)"), + njs_str("undefinedg") }, + + { njs_str("var r = /h/g;" + "Object.defineProperty(r,'flags',{value: ''});" + "''.replaceAll(r,'');"), + njs_str("TypeError: String.prototype.replaceAll called with a non-global RegExp argument") }, + { njs_str("/=/"), njs_str("/=/") }, From jordanc.carter at outlook.com Mon Feb 6 03:01:44 2023 From: jordanc.carter at outlook.com (J Carter) Date: Mon, 6 Feb 2023 03:01:44 +0000 Subject: [PATCH 5 of 6] Upstream: allow any worker to resolve upstream servers In-Reply-To: References: Message-ID: Hi Aleksei, Why not permanently assign the task of resolving a given upstream server group (all servers/peers within it) to a single worker? It seems that this approach would resolve the SRV issues, and remove the need for the shared queue of tasks. The load would still be spread evenly for the most realistic scenarios - which is where there are many upstream server groups of few servers, as opposed to few upstream server groups of many servers. On 01/02/2023 01:37, Aleksei Bavshin wrote: > # HG changeset patch > # User Aleksei Bavshin > # Date 1670883784 28800 > # Mon Dec 12 14:23:04 2022 -0800 > # Node ID f8eb6b94d8f46008eb5f2f1dbc747750d5755506 > # Parent cfae397f1ea87a35c41febab6162fe5142aa767b > Upstream: allow any worker to resolve upstream servers. > > This change addresses one of the limitations of the current re-resolve > code, dependency on the worker 0. Now, each worker is able to pick a > resolve task from a shared priority queue. > > The single worker implementation relies on the fact that each peer is > assigned to a specific worker and no other process may access its data. > Thus, it's safe to keep `peer->host.event` in the shared zone and modify > as necessary. That assumption becomes invalid once we allow any free > worker to update the peer. Now, all the workers have to know when the > previous resolution result expires and maintain their own timers. A > single shared event structure is no longer sufficient. > > The obvious solution is to make timer events local to a worker by moving > them up to the nearest object in a local memory, upstream. From the > upstream timer event handler we can walk the list of the peers and pick > these that are expired and not already owned by another process. > > To reduce the time spent under a lock we can keep a priority queue of > pending tasks, sorted by expiration time. Each worker is able to get an > expired server from the head of the queue, perform the name resolution > and put the peer back with a new expiration time. > Per-upstream or per-zone rbtree was considered as a store for pending > tasks, but there won't be any performance benefit until a certain large > number of servers in the upstream. Per-zone queues also require more > intricate locking. > > The benefits of the change are obvious: we're no longer tied to a > lifetime of the first worker process and the distribution of the tasks > is more even. There are certain disadvantages though: > > - SRV record may resolve into multiple A/AAAA records with different TTL > kept in a worker-local cache of a resolver. The next query in the > same worker will reuse all the cache entries that are still valid. > With the task distribution implemented, any worker may schedule the > next update of a peer and thus we lose the benefit of a local cache. > > - The change introduces an additional short lock on the list of peers > and allows to acquire existing long locks from different processes. > For example, it's possible that different workers will resolve large > SRV records from the same upstream and attempt to update the list of > peers at the same time. > > diff --git a/src/http/modules/ngx_http_upstream_zone_module.c b/src/http/modules/ngx_http_upstream_zone_module.c > --- a/src/http/modules/ngx_http_upstream_zone_module.c > +++ b/src/http/modules/ngx_http_upstream_zone_module.c > @@ -10,6 +10,9 @@ > #include > > > +#define NGX_UPSTREAM_RESOLVE_NO_WORKER (ngx_uint_t) -1 > + > + > static char *ngx_http_upstream_zone(ngx_conf_t *cf, ngx_command_t *cmd, > void *conf); > static ngx_int_t ngx_http_upstream_init_zone(ngx_shm_zone_t *shm_zone, > @@ -40,6 +43,13 @@ static ngx_command_t ngx_http_upstream_ > static ngx_int_t ngx_http_upstream_zone_init_worker(ngx_cycle_t *cycle); > static void ngx_http_upstream_zone_resolve_timer(ngx_event_t *event); > static void ngx_http_upstream_zone_resolve_handler(ngx_resolver_ctx_t *ctx); > +static void ngx_http_upstream_zone_resolve_queue_insert(ngx_queue_t *queue, > + ngx_http_upstream_host_t *host); > +static void ngx_http_upstream_zone_start_resolve( > + ngx_http_upstream_srv_conf_t *uscf, ngx_http_upstream_host_t *host); > +static void ngx_http_upstream_zone_schedule_resolve( > + ngx_http_upstream_srv_conf_t *uscf, ngx_http_upstream_host_t *host, > + ngx_msec_t timer); > > > static ngx_http_module_t ngx_http_upstream_zone_module_ctx = { > @@ -231,6 +241,8 @@ ngx_http_upstream_zone_copy_peers(ngx_sl > peers->shpool = shpool; > peers->config = config; > > + ngx_queue_init(&peers->resolve_queue); > + > for (peerp = &peers->peer; *peerp; peerp = &peer->next) { > /* pool is unlocked */ > peer = ngx_http_upstream_zone_copy_peer(peers, *peerp); > @@ -248,6 +260,9 @@ ngx_http_upstream_zone_copy_peers(ngx_sl > return NULL; > } > > + ngx_http_upstream_rr_peer_ref(peers, peer); > + ngx_queue_insert_tail(&peers->resolve_queue, &peer->host->queue); > + > *peerp = peer; > peer->id = (*peers->config)++; > } > @@ -268,6 +283,8 @@ ngx_http_upstream_zone_copy_peers(ngx_sl > backup->shpool = shpool; > backup->config = config; > > + ngx_queue_init(&backup->resolve_queue); > + > for (peerp = &backup->peer; *peerp; peerp = &peer->next) { > /* pool is unlocked */ > peer = ngx_http_upstream_zone_copy_peer(backup, *peerp); > @@ -285,6 +302,9 @@ ngx_http_upstream_zone_copy_peers(ngx_sl > return NULL; > } > > + ngx_http_upstream_rr_peer_ref(backup, peer); > + ngx_queue_insert_tail(&backup->resolve_queue, &peer->host->queue); > + > *peerp = peer; > peer->id = (*backup->config)++; > } > @@ -357,6 +377,8 @@ ngx_http_upstream_zone_copy_peer(ngx_htt > > dst->host->peers = peers; > dst->host->peer = dst; > + dst->host->expires = ngx_current_msec; > + dst->host->worker = NGX_UPSTREAM_RESOLVE_NO_WORKER; > > dst->host->name.len = src->host->name.len; > ngx_memcpy(dst->host->name.data, src->host->name.data, > @@ -438,13 +460,124 @@ ngx_http_upstream_zone_remove_peer_locke > } > > > +static void > +ngx_http_upstream_zone_resolve_queue_insert(ngx_queue_t *queue, > + ngx_http_upstream_host_t *host) > +{ > + ngx_queue_t *q; > + ngx_http_upstream_host_t *item; > + > + q = ngx_queue_last(queue); > + > + while (q != ngx_queue_sentinel(queue)) { > + item = ngx_queue_data(q, ngx_http_upstream_host_t, queue); > + > + if ((ngx_msec_int_t) (item->expires - host->expires) <= 0) { > + break; > + } > + > + q = ngx_queue_prev(q); > + } > + > + ngx_queue_insert_after(q, &host->queue); > +} > + > + > +static void > +ngx_http_upstream_zone_schedule_resolve(ngx_http_upstream_srv_conf_t *uscf, > + ngx_http_upstream_host_t *host, > + ngx_msec_t timer) > +{ > + ngx_msec_t now; > + ngx_event_t *event; > + ngx_http_upstream_host_t *head; > + ngx_http_upstream_rr_peers_t *peers; > + > + now = ngx_current_msec; > + event = &uscf->event; > + peers = host->peers; > + > + ngx_http_upstream_rr_peers_wlock(peers); > + > + host->expires = now + timer; > + host->worker = NGX_UPSTREAM_RESOLVE_NO_WORKER; > + ngx_http_upstream_zone_resolve_queue_insert(&peers->resolve_queue, host); > + > + head = ngx_queue_data(ngx_queue_head(&peers->resolve_queue), > + ngx_http_upstream_host_t, queue); > + if ((ngx_msec_int_t) (head->expires - host->expires) < 0) { > + timer = ngx_max((ngx_msec_int_t) (head->expires - now), 1); > + } > + > + ngx_http_upstream_rr_peers_unlock(peers); > + > + if (!event->timer_set > + || (ngx_msec_int_t) (now + timer - event->timer.key) < 0) > + { > + ngx_add_timer(event, timer); > + } > +} > + > + > +static void > +ngx_http_upstream_zone_resolve_timer(ngx_event_t *event) > +{ > + ngx_msec_t now, timer; > + ngx_msec_int_t expires; > + ngx_http_upstream_host_t *host; > + ngx_http_upstream_srv_conf_t *uscf; > + ngx_http_upstream_rr_peers_t *peers; > + > + uscf = event->data; > + peers = uscf->peer.data; > + now = ngx_current_msec; > + > + timer = (ngx_msec_t) 1000 * > + (uscf->resolver->valid ? uscf->resolver->valid : 10); > + > + do { > + for ( ;; ) { > + ngx_http_upstream_rr_peers_wlock(peers); > + > + if (ngx_queue_empty(&peers->resolve_queue)) { > + ngx_http_upstream_rr_peers_unlock(peers); > + break; > + } > + > + host = ngx_queue_data(ngx_queue_head(&peers->resolve_queue), > + ngx_http_upstream_host_t, queue); > + expires = host->expires - now; > + > + if (expires > 0) { > + ngx_http_upstream_rr_peers_unlock(peers); > + timer = ngx_min(timer, (ngx_msec_t) expires); > + break; > + } > + > + ngx_queue_remove(&host->queue); > + host->worker = ngx_worker; > + > + ngx_http_upstream_rr_peers_unlock(peers); > + ngx_http_upstream_zone_start_resolve(uscf, host); > + } > + > + peers = peers->next; > + > + } while (peers); > + > + if (!event->timer_set > + || ((ngx_msec_int_t) (now + timer - event->timer.key)) < 0) > + { > + ngx_add_timer(event, timer); > + } > +} > + > + > static ngx_int_t > ngx_http_upstream_zone_init_worker(ngx_cycle_t *cycle) > { > ngx_uint_t i; > ngx_event_t *event; > - ngx_http_upstream_rr_peer_t *peer; > - ngx_http_upstream_rr_peers_t *peers; > ngx_http_upstream_srv_conf_t *uscf, **uscfp; > ngx_http_upstream_main_conf_t *umcf; > > @@ -470,34 +603,13 @@ ngx_http_upstream_zone_init_worker(ngx_c > continue; > } > > - peers = uscf->peer.data; > - > - do { > - ngx_http_upstream_rr_peers_wlock(peers); > - > - for (peer = peers->resolve; peer; peer = peer->next) { > - > - if (peer->host->worker != ngx_worker) { > - continue; > - } > - > - event = &peer->host->event; > - ngx_memzero(event, sizeof(ngx_event_t)); > + event = &uscf->event; > + event->data = uscf; > + event->handler = ngx_http_upstream_zone_resolve_timer; > + event->log = cycle->log; > + event->cancelable = 1; > > - event->data = uscf; > - event->handler = ngx_http_upstream_zone_resolve_timer; > - event->log = cycle->log; > - event->cancelable = 1; > - > - ngx_http_upstream_rr_peer_ref(peers, peer); > - ngx_add_timer(event, 1); > - } > - > - ngx_http_upstream_rr_peers_unlock(peers); > - > - peers = peers->next; > - > - } while (peers); > + ngx_add_timer(event, 1); > } > > return NGX_OK; > @@ -505,16 +617,13 @@ ngx_http_upstream_zone_init_worker(ngx_c > > > static void > -ngx_http_upstream_zone_resolve_timer(ngx_event_t *event) > +ngx_http_upstream_zone_start_resolve(ngx_http_upstream_srv_conf_t *uscf, > + ngx_http_upstream_host_t *host) > { > ngx_resolver_ctx_t *ctx; > - ngx_http_upstream_host_t *host; > ngx_http_upstream_rr_peer_t *template; > ngx_http_upstream_rr_peers_t *peers; > - ngx_http_upstream_srv_conf_t *uscf; > > - host = (ngx_http_upstream_host_t *) event; > - uscf = event->data; > peers = host->peers; > template = host->peer; > > @@ -540,11 +649,13 @@ ngx_http_upstream_zone_resolve_timer(ngx > } > > if (ctx == NGX_NO_RESOLVER) { > - ngx_log_error(NGX_LOG_ERR, event->log, 0, > + ngx_log_error(NGX_LOG_ERR, uscf->event.log, 0, > "no resolver defined to resolve %V", &host->name); > return; > } > > + host->upstream = uscf; > + > ctx->name = host->name; > ctx->handler = ngx_http_upstream_zone_resolve_handler; > ctx->data = host; > @@ -558,7 +669,8 @@ ngx_http_upstream_zone_resolve_timer(ngx > > retry: > > - ngx_add_timer(event, ngx_max(uscf->resolver_timeout, 1000)); > + ngx_http_upstream_zone_schedule_resolve(uscf, host, > + ngx_max(uscf->resolver_timeout, 1000)); > } > > > @@ -590,8 +702,8 @@ ngx_http_upstream_zone_resolve_handler(n > ngx_http_upstream_srv_conf_t *uscf; > > host = ctx->data; > - event = &host->event; > - uscf = event->data; > + uscf = host->upstream; > + event = &uscf->event; > peers = host->peers; > template = host->peer; > > @@ -651,7 +763,8 @@ ngx_http_upstream_zone_resolve_handler(n > > ngx_resolve_name_done(ctx); > > - ngx_add_timer(event, ngx_max(uscf->resolver_timeout, 1000)); > + ngx_http_upstream_zone_schedule_resolve(uscf, host, > + ngx_max(uscf->resolver_timeout, 1000)); > return; > } > > @@ -851,5 +964,5 @@ done: > > ngx_resolve_name_done(ctx); > > - ngx_add_timer(event, timer); > + ngx_http_upstream_zone_schedule_resolve(uscf, host, timer); > } > diff --git a/src/http/ngx_http_upstream.h b/src/http/ngx_http_upstream.h > --- a/src/http/ngx_http_upstream.h > +++ b/src/http/ngx_http_upstream.h > @@ -138,6 +138,7 @@ struct ngx_http_upstream_srv_conf_s { > ngx_uint_t no_port; /* unsigned no_port:1 */ > > #if (NGX_HTTP_UPSTREAM_ZONE) > + ngx_event_t event; > ngx_shm_zone_t *shm_zone; > ngx_resolver_t *resolver; > ngx_msec_t resolver_timeout; > diff --git a/src/http/ngx_http_upstream_round_robin.h b/src/http/ngx_http_upstream_round_robin.h > --- a/src/http/ngx_http_upstream_round_robin.h > +++ b/src/http/ngx_http_upstream_round_robin.h > @@ -21,12 +21,14 @@ typedef struct ngx_http_upstream_rr_peer > #if (NGX_HTTP_UPSTREAM_ZONE) > > typedef struct { > - ngx_event_t event; /* must be first */ > + ngx_queue_t queue; > ngx_uint_t worker; > + ngx_msec_t expires; > ngx_str_t name; > ngx_str_t service; > ngx_http_upstream_rr_peers_t *peers; > ngx_http_upstream_rr_peer_t *peer; > + ngx_http_upstream_srv_conf_t *upstream; /* local */ > } ngx_http_upstream_host_t; > > #endif > @@ -101,6 +103,7 @@ struct ngx_http_upstream_rr_peers_s { > #if (NGX_HTTP_UPSTREAM_ZONE) > ngx_uint_t *config; > ngx_http_upstream_rr_peer_t *resolve; > + ngx_queue_t resolve_queue; > ngx_uint_t zombies; > #endif > }; > diff --git a/src/stream/ngx_stream_upstream.h b/src/stream/ngx_stream_upstream.h > --- a/src/stream/ngx_stream_upstream.h > +++ b/src/stream/ngx_stream_upstream.h > @@ -85,6 +85,7 @@ struct ngx_stream_upstream_srv_conf_s { > ngx_uint_t no_port; /* unsigned no_port:1 */ > > #if (NGX_STREAM_UPSTREAM_ZONE) > + ngx_event_t event; > ngx_shm_zone_t *shm_zone; > ngx_resolver_t *resolver; > ngx_msec_t resolver_timeout; > diff --git a/src/stream/ngx_stream_upstream_round_robin.h b/src/stream/ngx_stream_upstream_round_robin.h > --- a/src/stream/ngx_stream_upstream_round_robin.h > +++ b/src/stream/ngx_stream_upstream_round_robin.h > @@ -21,12 +21,14 @@ typedef struct ngx_stream_upstream_rr_pe > #if (NGX_STREAM_UPSTREAM_ZONE) > > typedef struct { > - ngx_event_t event; /* must be first */ > + ngx_queue_t queue; > ngx_uint_t worker; > + ngx_msec_t expires; > ngx_str_t name; > ngx_str_t service; > ngx_stream_upstream_rr_peers_t *peers; > ngx_stream_upstream_rr_peer_t *peer; > + ngx_stream_upstream_srv_conf_t *upstream; /* local */ > } ngx_stream_upstream_host_t; > > #endif > @@ -99,6 +101,7 @@ struct ngx_stream_upstream_rr_peers_s { > #if (NGX_STREAM_UPSTREAM_ZONE) > ngx_uint_t *config; > ngx_stream_upstream_rr_peer_t *resolve; > + ngx_queue_t resolve_queue; > ngx_uint_t zombies; > #endif > }; > diff --git a/src/stream/ngx_stream_upstream_zone_module.c b/src/stream/ngx_stream_upstream_zone_module.c > --- a/src/stream/ngx_stream_upstream_zone_module.c > +++ b/src/stream/ngx_stream_upstream_zone_module.c > @@ -10,6 +10,9 @@ > #include > > > +#define NGX_UPSTREAM_RESOLVE_NO_WORKER (ngx_uint_t) -1 > + > + > static char *ngx_stream_upstream_zone(ngx_conf_t *cf, ngx_command_t *cmd, > void *conf); > static ngx_int_t ngx_stream_upstream_init_zone(ngx_shm_zone_t *shm_zone, > @@ -40,6 +43,13 @@ static ngx_command_t ngx_stream_upstrea > static ngx_int_t ngx_stream_upstream_zone_init_worker(ngx_cycle_t *cycle); > static void ngx_stream_upstream_zone_resolve_timer(ngx_event_t *event); > static void ngx_stream_upstream_zone_resolve_handler(ngx_resolver_ctx_t *ctx); > +static void ngx_stream_upstream_zone_resolve_queue_insert(ngx_queue_t *queue, > + ngx_stream_upstream_host_t *host); > +static void ngx_stream_upstream_zone_start_resolve( > + ngx_stream_upstream_srv_conf_t *uscf, ngx_stream_upstream_host_t *host); > +static void ngx_stream_upstream_zone_schedule_resolve( > + ngx_stream_upstream_srv_conf_t *uscf, ngx_stream_upstream_host_t *host, > + ngx_msec_t timer); > > > static ngx_stream_module_t ngx_stream_upstream_zone_module_ctx = { > @@ -228,6 +238,8 @@ ngx_stream_upstream_zone_copy_peers(ngx_ > peers->shpool = shpool; > peers->config = config; > > + ngx_queue_init(&peers->resolve_queue); > + > for (peerp = &peers->peer; *peerp; peerp = &peer->next) { > /* pool is unlocked */ > peer = ngx_stream_upstream_zone_copy_peer(peers, *peerp); > @@ -245,6 +257,9 @@ ngx_stream_upstream_zone_copy_peers(ngx_ > return NULL; > } > > + ngx_stream_upstream_rr_peer_ref(peers, peer); > + ngx_queue_insert_tail(&peers->resolve_queue, &peer->host->queue); > + > *peerp = peer; > peer->id = (*peers->config)++; > } > @@ -265,6 +280,8 @@ ngx_stream_upstream_zone_copy_peers(ngx_ > backup->shpool = shpool; > backup->config = config; > > + ngx_queue_init(&backup->resolve_queue); > + > for (peerp = &backup->peer; *peerp; peerp = &peer->next) { > /* pool is unlocked */ > peer = ngx_stream_upstream_zone_copy_peer(backup, *peerp); > @@ -282,6 +299,9 @@ ngx_stream_upstream_zone_copy_peers(ngx_ > return NULL; > } > > + ngx_stream_upstream_rr_peer_ref(backup, peer); > + ngx_queue_insert_tail(&backup->resolve_queue, &peer->host->queue); > + > *peerp = peer; > peer->id = (*backup->config)++; > } > @@ -354,6 +374,8 @@ ngx_stream_upstream_zone_copy_peer(ngx_s > > dst->host->peers = peers; > dst->host->peer = dst; > + dst->host->expires = ngx_current_msec; > + dst->host->worker = NGX_UPSTREAM_RESOLVE_NO_WORKER; > > dst->host->name.len = src->host->name.len; > ngx_memcpy(dst->host->name.data, src->host->name.data, > @@ -435,13 +457,124 @@ ngx_stream_upstream_zone_remove_peer_loc > } > > > +static void > +ngx_stream_upstream_zone_resolve_queue_insert(ngx_queue_t *queue, > + ngx_stream_upstream_host_t *host) > +{ > + ngx_queue_t *q; > + ngx_stream_upstream_host_t *item; > + > + q = ngx_queue_last(queue); > + > + while (q != ngx_queue_sentinel(queue)) { > + item = ngx_queue_data(q, ngx_stream_upstream_host_t, queue); > + > + if ((ngx_msec_int_t) (item->expires - host->expires) <= 0) { > + break; > + } > + > + q = ngx_queue_prev(q); > + } > + > + ngx_queue_insert_after(q, &host->queue); > +} > + > + > +static void > +ngx_stream_upstream_zone_schedule_resolve(ngx_stream_upstream_srv_conf_t *uscf, > + ngx_stream_upstream_host_t *host, > + ngx_msec_t timer) > +{ > + ngx_msec_t now; > + ngx_event_t *event; > + ngx_stream_upstream_host_t *head; > + ngx_stream_upstream_rr_peers_t *peers; > + > + now = ngx_current_msec; > + event = &uscf->event; > + peers = host->peers; > + > + ngx_stream_upstream_rr_peers_wlock(peers); > + > + host->expires = now + timer; > + host->worker = NGX_UPSTREAM_RESOLVE_NO_WORKER; > + ngx_stream_upstream_zone_resolve_queue_insert(&peers->resolve_queue, host); > + > + head = ngx_queue_data(ngx_queue_head(&peers->resolve_queue), > + ngx_stream_upstream_host_t, queue); > + if ((ngx_msec_int_t) (head->expires - host->expires) < 0) { > + timer = ngx_max((ngx_msec_int_t) (head->expires - now), 1); > + } > + > + ngx_stream_upstream_rr_peers_unlock(peers); > + > + if (!event->timer_set > + || (ngx_msec_int_t) (now + timer - event->timer.key) < 0) > + { > + ngx_add_timer(event, timer); > + } > +} > + > + > +static void > +ngx_stream_upstream_zone_resolve_timer(ngx_event_t *event) > +{ > + ngx_msec_t now, timer; > + ngx_msec_int_t expires; > + ngx_stream_upstream_host_t *host; > + ngx_stream_upstream_srv_conf_t *uscf; > + ngx_stream_upstream_rr_peers_t *peers; > + > + uscf = event->data; > + peers = uscf->peer.data; > + now = ngx_current_msec; > + > + timer = (ngx_msec_t) 1000 * > + (uscf->resolver->valid ? uscf->resolver->valid : 10); > + > + do { > + for ( ;; ) { > + ngx_stream_upstream_rr_peers_wlock(peers); > + > + if (ngx_queue_empty(&peers->resolve_queue)) { > + ngx_stream_upstream_rr_peers_unlock(peers); > + break; > + } > + > + host = ngx_queue_data(ngx_queue_head(&peers->resolve_queue), > + ngx_stream_upstream_host_t, queue); > + expires = host->expires - now; > + > + if (expires > 0) { > + ngx_stream_upstream_rr_peers_unlock(peers); > + timer = ngx_min(timer, (ngx_msec_t) expires); > + break; > + } > + > + ngx_queue_remove(&host->queue); > + host->worker = ngx_worker; > + > + ngx_stream_upstream_rr_peers_unlock(peers); > + ngx_stream_upstream_zone_start_resolve(uscf, host); > + } > + > + peers = peers->next; > + > + } while (peers); > + > + if (!event->timer_set > + || (ngx_msec_int_t) (now + timer - event->timer.key) < 0) > + { > + ngx_add_timer(event, timer); > + } > +} > + > + > static ngx_int_t > ngx_stream_upstream_zone_init_worker(ngx_cycle_t *cycle) > { > ngx_uint_t i; > ngx_event_t *event; > - ngx_stream_upstream_rr_peer_t *peer; > - ngx_stream_upstream_rr_peers_t *peers; > ngx_stream_upstream_srv_conf_t *uscf, **uscfp; > ngx_stream_upstream_main_conf_t *umcf; > > @@ -468,34 +601,13 @@ ngx_stream_upstream_zone_init_worker(ngx > continue; > } > > - peers = uscf->peer.data; > - > - do { > - ngx_stream_upstream_rr_peers_wlock(peers); > - > - for (peer = peers->resolve; peer; peer = peer->next) { > - > - if (peer->host->worker != ngx_worker) { > - continue; > - } > - > - event = &peer->host->event; > - ngx_memzero(event, sizeof(ngx_event_t)); > + event = &uscf->event; > + event->data = uscf; > + event->handler = ngx_stream_upstream_zone_resolve_timer; > + event->log = cycle->log; > + event->cancelable = 1; > > - event->data = uscf; > - event->handler = ngx_stream_upstream_zone_resolve_timer; > - event->log = cycle->log; > - event->cancelable = 1; > - > - ngx_stream_upstream_rr_peer_ref(peers, peer); > - ngx_add_timer(event, 1); > - } > - > - ngx_stream_upstream_rr_peers_unlock(peers); > - > - peers = peers->next; > - > - } while (peers); > + ngx_add_timer(event, 1); > } > > return NGX_OK; > @@ -503,16 +615,13 @@ ngx_stream_upstream_zone_init_worker(ngx > > > static void > -ngx_stream_upstream_zone_resolve_timer(ngx_event_t *event) > +ngx_stream_upstream_zone_start_resolve(ngx_stream_upstream_srv_conf_t *uscf, > + ngx_stream_upstream_host_t *host) > { > ngx_resolver_ctx_t *ctx; > - ngx_stream_upstream_host_t *host; > ngx_stream_upstream_rr_peer_t *template; > ngx_stream_upstream_rr_peers_t *peers; > - ngx_stream_upstream_srv_conf_t *uscf; > > - host = (ngx_stream_upstream_host_t *) event; > - uscf = event->data; > peers = host->peers; > template = host->peer; > > @@ -538,11 +647,13 @@ ngx_stream_upstream_zone_resolve_timer(n > } > > if (ctx == NGX_NO_RESOLVER) { > - ngx_log_error(NGX_LOG_ERR, event->log, 0, > + ngx_log_error(NGX_LOG_ERR, uscf->event.log, 0, > "no resolver defined to resolve %V", &host->name); > return; > } > > + host->upstream = uscf; > + > ctx->name = host->name; > ctx->handler = ngx_stream_upstream_zone_resolve_handler; > ctx->data = host; > @@ -556,7 +667,8 @@ ngx_stream_upstream_zone_resolve_timer(n > > retry: > > - ngx_add_timer(event, ngx_max(uscf->resolver_timeout, 1000)); > + ngx_stream_upstream_zone_schedule_resolve(uscf, host, > + ngx_max(uscf->resolver_timeout, 1000)); > } > > > @@ -588,8 +700,8 @@ ngx_stream_upstream_zone_resolve_handler > ngx_stream_upstream_srv_conf_t *uscf; > > host = ctx->data; > - event = &host->event; > - uscf = event->data; > + uscf = host->upstream; > + event = &uscf->event; > peers = host->peers; > template = host->peer; > > @@ -649,7 +761,8 @@ ngx_stream_upstream_zone_resolve_handler > > ngx_resolve_name_done(ctx); > > - ngx_add_timer(event, ngx_max(uscf->resolver_timeout, 1000)); > + ngx_stream_upstream_zone_schedule_resolve(uscf, host, > + ngx_max(uscf->resolver_timeout, 1000)); > return; > } > > @@ -849,5 +962,5 @@ done: > > ngx_resolve_name_done(ctx); > > - ngx_add_timer(event, timer); > + ngx_stream_upstream_zone_schedule_resolve(uscf, host, timer); > } > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > https://mailman.nginx.org/mailman/listinfo/nginx-devel From arut at nginx.com Mon Feb 6 14:27:01 2023 From: arut at nginx.com (Roman Arutyunyan) Date: Mon, 6 Feb 2023 18:27:01 +0400 Subject: [PATCH] QUIC: OpenSSL compatibility layer In-Reply-To: <20230202183522.o4w5xbjvegbchts4@Y9MQ9X2QVV> References: <64a365dcb52503e91d91.1671606452@arut-laptop> <20230109111316.yfqj24unoubibihz@N00W24XTQX> <20230202183522.o4w5xbjvegbchts4@Y9MQ9X2QVV> Message-ID: <20230206142701.pxzaq3irckqads7y@N00W24XTQX> Hi, On Thu, Feb 02, 2023 at 10:35:22PM +0400, Sergey Kandaurov wrote: > On Mon, Jan 09, 2023 at 03:13:16PM +0400, Roman Arutyunyan wrote: > > # HG changeset patch > > # User Roman Arutyunyan > > # Date 1673262402 -14400 > > # Mon Jan 09 15:06:42 2023 +0400 > > # Branch quic > > # Node ID 4e5dfe13c84fe50bec639f1b7dcc81604378a42b > > # Parent aaa2a3831eefe4315dfb8a9be7178c79ff67f163 > > QUIC: OpenSSL compatibility layer. > > > > The change allows to compile QUIC with OpenSSL which lacks BoringSSL QUIC API. > > > > This implementation does not support 0-RTT. > > > > diff --git a/README b/README > > --- a/README > > +++ b/README > > @@ -53,7 +53,7 @@ 1. Introduction > > > > 2. Installing > > > > - A library that provides QUIC support is required to build nginx, there > > + A library that provides QUIC support is recommended to build nginx, there > > are several of those available on the market: > > + BoringSSL [4] > > + LibreSSL [5] > > @@ -85,6 +85,10 @@ 2. Installing > > --with-cc-opt="-I../libressl/build/include" \ > > --with-ld-opt="-L../libressl/build/lib" > > > > + Alternatively, nginx can be configured with OpenSSL compatibility > > + layer, which emulates BoringSSL QUIC API for OpenSSL. This mode is > > + enabled by default if native QUIC support is not detected. > > + > > When configuring nginx, it's possible to enable QUIC and HTTP/3 > > using the following new configuration options: > > > > diff --git a/auto/lib/openssl/conf b/auto/lib/openssl/conf > > --- a/auto/lib/openssl/conf > > +++ b/auto/lib/openssl/conf > > @@ -10,6 +10,7 @@ if [ $OPENSSL != NONE ]; then > > > > if [ $USE_OPENSSL_QUIC = YES ]; then > > have=NGX_QUIC . auto/have > > + have=NGX_QUIC_OPENSSL_COMPAT . auto/have > > This won't build with QuicTLS sources specified in --with-openssl, > due to type/function redefinitions. The patch to address this: > > diff --git a/src/event/quic/ngx_event_quic_openssl_compat.h b/src/event/quic/ngx_event_quic_openssl_compat.h > --- a/src/event/quic/ngx_event_quic_openssl_compat.h > +++ b/src/event/quic/ngx_event_quic_openssl_compat.h > @@ -7,6 +7,10 @@ > #ifndef _NGX_EVENT_QUIC_OPENSSL_COMPAT_H_INCLUDED_ > #define _NGX_EVENT_QUIC_OPENSSL_COMPAT_H_INCLUDED_ > > +#ifdef TLSEXT_TYPE_quic_transport_parameters > +#undef NGX_QUIC_OPENSSL_COMPAT > +#else Good point, thanks. > + > > #include > #include > @@ -48,4 +52,6 @@ void SSL_get_peer_quic_transport_params( > const uint8_t **out_params, size_t *out_params_len); > > > +#endif /* TLSEXT_TYPE_quic_transport_parameters */ > + > #endif /* _NGX_EVENT_QUIC_OPENSSL_COMPAT_H_INCLUDED_ */ > > > > fi > > > > case "$CC" in > > @@ -124,6 +125,45 @@ else > > CORE_INCS="$CORE_INCS $ngx_feature_path" > > CORE_LIBS="$CORE_LIBS $ngx_feature_libs" > > OPENSSL=YES > > + > > + if [ $USE_OPENSSL_QUIC = YES ]; then > > + > > + ngx_feature="OpenSSL QUIC support" > > + ngx_feature_name="NGX_OPENSSL_QUIC" > > + ngx_feature_run=no > > + ngx_feature_incs="#include " > > + ngx_feature_test="SSL_set_quic_method(NULL, NULL)" > > + . auto/feature > > + > > + if [ $ngx_found = no ]; then > > bad indent OK, thanks. > > + > > + ngx_feature="OpenSSL QUIC compatibility" > > + ngx_feature_name="NGX_QUIC_OPENSSL_COMPAT" > > + ngx_feature_run=no > > + ngx_feature_incs="#include " > > + ngx_feature_test=" > > + SSL_set_max_early_data(NULL, TLS1_3_VERSION); > > While practicaly useful to check for TLSv1.3-aware library, > this abuses SSL_set_max_early_data() API. > > > + SSL_CTX_set_msg_callback(NULL, NULL); > > + SSL_CTX_set_keylog_callback(NULL, NULL); > > + SSL_CTX_add_custom_ext(NULL, 0, 0, NULL, NULL, > > + NULL, NULL, NULL)" > > Just using SSL_CTX_add_custom_ext() seems to be sufficient to ensure > this is OpenSSL version 1.1.1+. OK, left only SSL_CTX_add_custom_ext() and TLS1_3_VERSION. > > + . auto/feature > > + fi > > + > > + if [ $ngx_found = no ]; then > > +cat << END > > + > > +$0: error: certain modules require OpenSSL QUIC support. > > +You can either do not enable the modules, or install the OpenSSL library with > > +QUIC support into the system, or build the OpenSSL library with QUIC support > > +statically from the source with nginx by using --with-openssl= option. > > + > > +END > > + exit 1 > > + fi > > + > > + have=NGX_QUIC . auto/have > > + fi > > fi > > fi > > > > @@ -139,29 +179,4 @@ with nginx by using --with-openssl= > END > > exit 1 > > fi > > - > > - if [ $USE_OPENSSL_QUIC = YES ]; then > > - > > - ngx_feature="OpenSSL QUIC support" > > - ngx_feature_name="NGX_QUIC" > > - ngx_feature_run=no > > - ngx_feature_incs="#include " > > - ngx_feature_path= > > - ngx_feature_libs="-lssl -lcrypto $NGX_LIBDL $NGX_LIBPTHREAD" > > - ngx_feature_test="SSL_set_quic_method(NULL, NULL)" > > - . auto/feature > > - > > - if [ $ngx_found = no ]; then > > - > > -cat << END > > - > > -$0: error: certain modules require OpenSSL QUIC support. > > -You can either do not enable the modules, or install the OpenSSL library with > > -QUIC support into the system, or build the OpenSSL library with QUIC support > > -statically from the source with nginx by using --with-openssl= option. > > - > > -END > > - exit 1 > > - fi > > - fi > > fi > > diff --git a/auto/modules b/auto/modules > > --- a/auto/modules > > +++ b/auto/modules > > @@ -1342,7 +1342,8 @@ if [ $USE_OPENSSL_QUIC = YES ]; then > > src/event/quic/ngx_event_quic_tokens.h \ > > src/event/quic/ngx_event_quic_ack.h \ > > src/event/quic/ngx_event_quic_output.h \ > > - src/event/quic/ngx_event_quic_socket.h" > > + src/event/quic/ngx_event_quic_socket.h \ > > + src/event/quic/ngx_event_quic_openssl_compat.h" > > ngx_module_srcs="src/event/quic/ngx_event_quic.c \ > > src/event/quic/ngx_event_quic_udp.c \ > > src/event/quic/ngx_event_quic_transport.c \ > > @@ -1355,7 +1356,8 @@ if [ $USE_OPENSSL_QUIC = YES ]; then > > src/event/quic/ngx_event_quic_tokens.c \ > > src/event/quic/ngx_event_quic_ack.c \ > > src/event/quic/ngx_event_quic_output.c \ > > - src/event/quic/ngx_event_quic_socket.c" > > + src/event/quic/ngx_event_quic_socket.c \ > > + src/event/quic/ngx_event_quic_openssl_compat.c" > > > > ngx_module_libs= > > ngx_module_link=YES > > diff --git a/src/event/ngx_event_openssl.c b/src/event/ngx_event_openssl.c > > --- a/src/event/ngx_event_openssl.c > > +++ b/src/event/ngx_event_openssl.c > > @@ -9,6 +9,10 @@ > > #include > > #include > > > > +#if (NGX_QUIC_OPENSSL_COMPAT) > > +#include > > +#endif > > + > > > > #define NGX_SSL_PASSWORD_BUFFER_SIZE 4096 > > > > @@ -392,6 +396,10 @@ ngx_ssl_create(ngx_ssl_t *ssl, ngx_uint_ > > > > SSL_CTX_set_info_callback(ssl->ctx, ngx_ssl_info_callback); > > > > +#if (NGX_QUIC_OPENSSL_COMPAT) > > + ngx_quic_compat_init(ssl->ctx); > > +#endif > > + > > This enables compatibility unconditionally for all TLS versions and > consumers, including such modules as the mail module and connections > to upstream. While SSL_CTX_add_custom_ext() to enable the transport > parameters extension in TLS < 1.3 seems to be harmless, and enabling > the keylog callback just leads to redundant function calls, I propose > to move this close to consumers, e.g.: > > diff --git a/src/http/modules/ngx_http_ssl_module.c b/src/http/modules/ngx_http_ssl_module.c > --- a/src/http/modules/ngx_http_ssl_module.c > +++ b/src/http/modules/ngx_http_ssl_module.c > @@ -9,6 +9,10 @@ > #include > #include > > +#if (NGX_QUIC_OPENSSL_COMPAT) > +#include > +#endif > + > > typedef ngx_int_t (*ngx_ssl_variable_handler_pt)(ngx_connection_t *c, > ngx_pool_t *pool, ngx_str_t *s); > @@ -1317,16 +1321,19 @@ ngx_http_ssl_init(ngx_conf_t *cf) > continue; > } > > + cscf = addr[a].default_server; > + sscf = cscf->ctx->srv_conf[ngx_http_ssl_module.ctx_index]; > + > if (addr[a].opt.http3) { > name = "http3"; > +#if (NGX_QUIC_OPENSSL_COMPAT) > + ngx_quic_compat_init(sscf->ssl.ctx); This can be called several times per SSL_CTX, which will end up with an error in SSL_CTX_add_custom_ext(). Added a check for existing transport parameter in ngx_quic_compat_init() and logging. Also, added a check for error from ngx_quic_compat_init() here. > +#endif > > } else { > name = "ssl"; > } > > - cscf = addr[a].default_server; > - sscf = cscf->ctx->srv_conf[ngx_http_ssl_module.ctx_index]; > - > if (sscf->certificates) { > > if (addr[a].opt.http3 && !(sscf->protocols & NGX_SSL_TLSv1_3)) { > > (with similar change in the stream module) Yes, added. > So we could remove SOCK_DGRAM checks. Not sure about this. A single SSL_CTX can be shared between TCP and QUIC. > > return NGX_OK; > > } > > > > diff --git a/src/event/quic/ngx_event_quic_connection.h b/src/event/quic/ngx_event_quic_connection.h > > --- a/src/event/quic/ngx_event_quic_connection.h > > +++ b/src/event/quic/ngx_event_quic_connection.h > > @@ -24,6 +24,9 @@ typedef struct ngx_quic_send_ctx_s ng > > typedef struct ngx_quic_socket_s ngx_quic_socket_t; > > typedef struct ngx_quic_path_s ngx_quic_path_t; > > typedef struct ngx_quic_keys_s ngx_quic_keys_t; > > +#if (NGX_QUIC_OPENSSL_COMPAT) > > +typedef struct ngx_quic_compat_s ngx_quic_compat_t; > > +#endif > > > > #include > > #include > > @@ -36,6 +39,9 @@ typedef struct ngx_quic_keys_s ng > > #include > > #include > > #include > > +#if (NGX_QUIC_OPENSSL_COMPAT) > > +#include > > +#endif > > > > > > /* RFC 9002, 6.2.2. Handshakes and New Paths: kInitialRtt */ > > @@ -236,6 +242,10 @@ struct ngx_quic_connection_s { > > ngx_uint_t nshadowbufs; > > #endif > > > > +#if (NGX_QUIC_OPENSSL_COMPAT) > > + ngx_quic_compat_t *compat; > > +#endif > > + > > ngx_quic_streams_t streams; > > ngx_quic_congestion_t congestion; > > > > diff --git a/src/event/quic/ngx_event_quic_openssl_compat.c b/src/event/quic/ngx_event_quic_openssl_compat.c > > new file mode 100644 > > --- /dev/null > > +++ b/src/event/quic/ngx_event_quic_openssl_compat.c > > @@ -0,0 +1,656 @@ > > + > > +/* > > + * Copyright (C) Nginx, Inc. > > + */ > > + > > + > > +#include > > +#include > > +#include > > +#include > > + > > + > > +#if (NGX_QUIC_OPENSSL_COMPAT) > > + > > +#define NGX_QUIC_COMPAT_RECORD_SIZE 1024 > > + > > +#define NGX_QUIC_COMPAT_SSL_TP_EXT 0x39 > > For the record: > SSL library sources have the TLSEXT_TYPE_quic_transport_parameters > extension value written in the decimal form, so could we. > An exception is QuicTLS, which seems to derive hex form from draft days. In RFC 8446 TLS 1.3, extensions are specified as decimal numbers. In RFC 9001 QUIC TLS, transport parameter extension is specified as 0x39. The code follows those definitions literally :) > > +#define NGX_QUIC_COMPAT_CLIENT_EARLY "CLIENT_EARLY_TRAFFIC_SECRET" > > +#define NGX_QUIC_COMPAT_CLIENT_HANDSHAKE "CLIENT_HANDSHAKE_TRAFFIC_SECRET" > > +#define NGX_QUIC_COMPAT_SERVER_HANDSHAKE "SERVER_HANDSHAKE_TRAFFIC_SECRET" > > +#define NGX_QUIC_COMPAT_CLIENT_APPLICATION "CLIENT_TRAFFIC_SECRET_0" > > +#define NGX_QUIC_COMPAT_SERVER_APPLICATION "SERVER_TRAFFIC_SECRET_0" > > + > > + > > +typedef struct { > > + ngx_quic_secret_t secret; > > + ngx_uint_t cipher; > > +} ngx_quic_compat_keys_t; > > + > > + > > +typedef struct { > > + ngx_log_t *log; > > + > > + u_char type; > > + ngx_str_t payload; > > + uint64_t number; > > + ngx_quic_compat_keys_t *keys; > > + > > + enum ssl_encryption_level_t level; > > +} ngx_quic_compat_record_t; > > + > > + > > +struct ngx_quic_compat_s { > > + const SSL_QUIC_METHOD *method; > > + > > + enum ssl_encryption_level_t write_level; > > + enum ssl_encryption_level_t read_level; > > + > > + uint64_t read_record; > > + ngx_quic_compat_keys_t keys; > > + > > + ngx_str_t tp; > > + ngx_str_t ctp; > > +}; > > + > > + > > +static void ngx_quic_compat_keylog_callback(const SSL *ssl, const char *line); > > +static ngx_int_t ngx_quic_compat_set_encryption_secret(ngx_log_t *log, > > + ngx_quic_compat_keys_t *keys, enum ssl_encryption_level_t level, > > + const SSL_CIPHER *cipher, const uint8_t *secret, size_t secret_len); > > +static int ngx_quic_compat_add_transport_params_callback(SSL *ssl, > > + unsigned int ext_type, unsigned int context, const unsigned char **out, > > + size_t *outlen, X509 *x, size_t chainidx, int *al, void *add_arg); > > +static int ngx_quic_compat_parse_transport_params_callback(SSL *ssl, > > + unsigned int ext_type, unsigned int context, const unsigned char *in, > > + size_t inlen, X509 *x, size_t chainidx, int *al, void *parse_arg); > > +static void ngx_quic_compat_message_callback(int write_p, int version, > > + int content_type, const void *buf, size_t len, SSL *ssl, void *arg); > > +static size_t ngx_quic_compat_create_header(ngx_quic_compat_record_t *rec, > > + u_char *out, ngx_uint_t plain); > > +static ngx_int_t ngx_quic_compat_create_record(ngx_quic_compat_record_t *rec, > > + ngx_str_t *res); > > + > > + > > +ngx_int_t > > +ngx_quic_compat_init(SSL_CTX *ctx) > > +{ > > + SSL_CTX_set_keylog_callback(ctx, ngx_quic_compat_keylog_callback); > > + > > + if (SSL_CTX_add_custom_ext(ctx, NGX_QUIC_COMPAT_SSL_TP_EXT, > > + SSL_EXT_CLIENT_HELLO > > + |SSL_EXT_TLS1_3_ENCRYPTED_EXTENSIONS, > > + ngx_quic_compat_add_transport_params_callback, > > + NULL, > > + NULL, > > + ngx_quic_compat_parse_transport_params_callback, > > + NULL) > > + == 0) > > + { > > + return NGX_ERROR; > > + } > > + > > + return NGX_OK; > > +} > > + > > + > > +static void > > +ngx_quic_compat_keylog_callback(const SSL *ssl, const char *line) > > +{ > > + u_char ch, *p, *start, value; > > + size_t n; > > + ngx_uint_t write; > > + const SSL_CIPHER *cipher; > > + ngx_quic_compat_t *com; > > + ngx_connection_t *c; > > + ngx_quic_connection_t *qc; > > + enum ssl_encryption_level_t level; > > + u_char secret[EVP_MAX_MD_SIZE]; > > + > > + c = ngx_ssl_get_connection(ssl); > > + if (c->type != SOCK_DGRAM) { > > + return; > > + } > > + > > + p = (u_char *) line; > > + > > + for (start = p; *p && *p != ' '; p++); > > + > > + n = p - start; > > + > > + ngx_log_debug2(NGX_LOG_DEBUG_EVENT, c->log, 0, > > + "quic compat secret %*s", n, start); > > + > > + if (n == sizeof(NGX_QUIC_COMPAT_CLIENT_EARLY) - 1 > > + && ngx_strncmp(start, NGX_QUIC_COMPAT_CLIENT_EARLY, n) == 0) > > For the record, with the current approach, this condition will never match: Thanks, removed this one from this patch. > > + { > > + level = ssl_encryption_early_data; > > + write = 0; > > + > > + } else if (n == sizeof(NGX_QUIC_COMPAT_CLIENT_HANDSHAKE) - 1 > > + && ngx_strncmp(start, NGX_QUIC_COMPAT_CLIENT_HANDSHAKE, n) == 0) > > + { > > + level = ssl_encryption_handshake; > > + write = 0; > > + > > + } else if (n == sizeof(NGX_QUIC_COMPAT_SERVER_HANDSHAKE) - 1 > > + && ngx_strncmp(start, NGX_QUIC_COMPAT_SERVER_HANDSHAKE, n) == 0) > > + { > > + level = ssl_encryption_handshake; > > + write = 1; > > + > > + } else if (n == sizeof(NGX_QUIC_COMPAT_CLIENT_APPLICATION) - 1 > > + && ngx_strncmp(start, NGX_QUIC_COMPAT_CLIENT_APPLICATION, n) > > + == 0) > > + { > > + level = ssl_encryption_application; > > + write = 0; > > + > > + } else if (n == sizeof(NGX_QUIC_COMPAT_SERVER_APPLICATION) - 1 > > + && ngx_strncmp(start, NGX_QUIC_COMPAT_SERVER_APPLICATION, n) > > + == 0) > > + { > > + level = ssl_encryption_application; > > + write = 1; > > + > > + } else { > > + return; > > + } > > + > > + if (*p++ == '\0') { > > + return; > > + } > > + > > + for ( /* void */ ; *p && *p != ' '; p++); > > + > > + if (*p++ == '\0') { > > + return; > > + } > > + > > + for (n = 0, start = p; *p; p++) { > > + ch = *p; > > + > > + if (ch >= '0' && ch <= '9') { > > + value = ch - '0'; > > + goto next; > > + } > > + > > + ch = (u_char) (ch | 0x20); > > + > > + if (ch >= 'a' && ch <= 'f') { > > + value = ch - 'a' + 10; > > + goto next; > > + } > > + > > + ngx_log_error(NGX_LOG_EMERG, c->log, 0, > > + "invalid OpenSSL QUIC secret format"); > > + > > + return; > > + > > + next: > > + > > + if ((p - start) % 2) { > > + secret[n++] += value; > > + > > + } else { > > + if (n >= EVP_MAX_MD_SIZE) { > > + ngx_log_error(NGX_LOG_EMERG, c->log, 0, > > + "too big OpenSSL QUIC secret"); > > + return; > > + } > > + > > + secret[n] = (value << 4); > > + } > > + } > > + > > + qc = ngx_quic_get_connection(c); > > + com = qc->compat; > > + cipher = SSL_get_current_cipher(ssl); > > + > > + if (write) { > > + com->method->set_write_secret((SSL *) ssl, level, cipher, secret, n); > > + com->write_level = level; > > + > > + } else { > > + com->method->set_read_secret((SSL *) ssl, level, cipher, secret, n); > > + com->read_level = level; > > + com->read_record = 0; > > + > > + (void) ngx_quic_compat_set_encryption_secret(c->log, &com->keys, level, > > + cipher, secret, n); > > + } > > +} > > + > > + > > +static ngx_int_t > > +ngx_quic_compat_set_encryption_secret(ngx_log_t *log, > > + ngx_quic_compat_keys_t *keys, enum ssl_encryption_level_t level, > > + const SSL_CIPHER *cipher, const uint8_t *secret, size_t secret_len) > > +{ > > + ngx_int_t key_len; > > + ngx_str_t secret_str; > > + ngx_uint_t i; > > + ngx_quic_hkdf_t seq[2]; > > + ngx_quic_secret_t *peer_secret; > > + ngx_quic_ciphers_t ciphers; > > + > > + peer_secret = &keys->secret; > > + > > + keys->cipher = SSL_CIPHER_get_id(cipher); > > + > > + key_len = ngx_quic_ciphers(keys->cipher, &ciphers, level); > > + > > + if (key_len == NGX_ERROR) { > > + ngx_ssl_error(NGX_LOG_INFO, log, 0, "unexpected cipher"); > > + return NGX_ERROR; > > + } > > + > > + if (sizeof(peer_secret->secret.data) < secret_len) { > > + ngx_log_error(NGX_LOG_ALERT, log, 0, > > + "unexpected secret len: %uz", secret_len); > > + return NGX_ERROR; > > + } > > + > > + peer_secret->secret.len = secret_len; > > + ngx_memcpy(peer_secret->secret.data, secret, secret_len); > > + > > + peer_secret->key.len = key_len; > > + peer_secret->iv.len = NGX_QUIC_IV_LEN; > > + > > + secret_str.len = secret_len; > > + secret_str.data = (u_char *) secret; > > + > > + ngx_quic_hkdf_set(&seq[0], "tls13 key", &peer_secret->key, &secret_str); > > + ngx_quic_hkdf_set(&seq[1], "tls13 iv", &peer_secret->iv, &secret_str); > > + > > + for (i = 0; i < (sizeof(seq) / sizeof(seq[0])); i++) { > > + if (ngx_quic_hkdf_expand(&seq[i], ciphers.d, log) != NGX_OK) { > > + return NGX_ERROR; > > + } > > + } > > + > > + return NGX_OK; > > +} > > + > > + > > +static int > > +ngx_quic_compat_add_transport_params_callback(SSL *ssl, unsigned int ext_type, > > + unsigned int context, const unsigned char **out, size_t *outlen, X509 *x, > > + size_t chainidx, int *al, void *add_arg) > > +{ > > + ngx_connection_t *c; > > + ngx_quic_compat_t *com; > > + ngx_quic_connection_t *qc; > > + > > + c = ngx_ssl_get_connection(ssl); > > + if (c->type != SOCK_DGRAM) { > > + return 0; > > + } > > + > > + ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0, > > + "quic compat add transport params"); > > + > > + qc = ngx_quic_get_connection(c); > > + com = qc->compat; > > + > > + *out = com->tp.data; > > + *outlen = com->tp.len; > > + > > + return 1; > > +} > > + > > + > > +static int > > +ngx_quic_compat_parse_transport_params_callback(SSL *ssl, unsigned int ext_type, > > + unsigned int context, const unsigned char *in, size_t inlen, X509 *x, > > + size_t chainidx, int *al, void *parse_arg) > > +{ > > + u_char *p; > > + ngx_connection_t *c; > > + ngx_quic_compat_t *com; > > + ngx_quic_connection_t *qc; > > + > > + c = ngx_ssl_get_connection(ssl); > > + if (c->type != SOCK_DGRAM) { > > + return 0; > > + } > > + > > + ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0, > > + "quic compat parse transport params"); > > + > > + qc = ngx_quic_get_connection(c); > > + com = qc->compat; > > + > > + p = ngx_pnalloc(c->pool, inlen); > > + if (p == NULL) { > > + return 0; > > + } > > + > > + ngx_memcpy(p, in, inlen); > > + > > + com->ctp.data = p; > > + com->ctp.len = inlen; > > + > > + return 1; > > +} > > + > > + > > +int > > +SSL_set_quic_method(SSL *ssl, const SSL_QUIC_METHOD *quic_method) > > +{ > > + BIO *rbio, *wbio; > > + ngx_connection_t *c; > > + ngx_quic_compat_t *com; > > + ngx_quic_connection_t *qc; > > + > > + c = ngx_ssl_get_connection(ssl); > > + > > + ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0, "quic compat set method"); > > + > > + qc = ngx_quic_get_connection(c); > > + > > + qc->compat = ngx_pcalloc(c->pool, sizeof(ngx_quic_compat_t)); > > + if (qc->compat == NULL) { > > + return 0; > > + } > > + > > + com = qc->compat; > > + com->method = quic_method; > > + > > + rbio = BIO_new(BIO_s_mem()); > > + if (rbio == NULL) { > > + return 0; > > + } > > + > > + wbio = BIO_new(BIO_s_null()); > > + if (wbio == NULL) { > > + return 0; > > + } > > + > > + SSL_set_bio(ssl, rbio, wbio); > > + > > + SSL_set_msg_callback(ssl, ngx_quic_compat_message_callback); > > + > > + /* early data is not supported */ > > + SSL_set_max_early_data(ssl, 0); > > + > > + return 1; > > +} > > + > > + > > +static void > > +ngx_quic_compat_message_callback(int write_p, int version, int content_type, > > + const void *buf, size_t len, SSL *ssl, void *arg) > > +{ > > + ngx_uint_t alert; > > + ngx_connection_t *c; > > + ngx_quic_compat_t *com; > > + ngx_quic_connection_t *qc; > > + enum ssl_encryption_level_t level; > > + > > + if (!write_p) { > > + return; > > + } > > + > > + c = ngx_ssl_get_connection(ssl); > > + qc = ngx_quic_get_connection(c); > > + > > + if (qc == NULL) { > > + /* closing */ > > + return; > > + } > > + > > + com = qc->compat; > > + level = com->write_level; > > + > > + switch (content_type) { > > + > > + case SSL3_RT_HANDSHAKE: > > + ngx_log_debug2(NGX_LOG_DEBUG_EVENT, c->log, 0, > > + "quic compat tx %s len:%uz ", > > + ngx_quic_level_name(level), len); > > + > > + (void) com->method->add_handshake_data(ssl, level, buf, len); > > + > > + break; > > + > > + case SSL3_RT_ALERT: > > + if (len >= 2) { > > + alert = ((u_char *) buf)[1]; > > + > > + ngx_log_debug3(NGX_LOG_DEBUG_EVENT, c->log, 0, > > + "quic compat %s alert:%ui len:%uz ", > > + ngx_quic_level_name(level), alert, len); > > + > > + (void) com->method->send_alert(ssl, level, alert); > > + > > + break; > > + } > > + > > + /* fall through */ > > + > > + default: > > + ngx_log_debug3(NGX_LOG_DEBUG_EVENT, c->log, 0, > > + "quic compat %s ignore msg:%d len:%uz ", > > + ngx_quic_level_name(level), content_type, len); > > Other content_type values are typically SSL3_RT_HEADER with its short header, > and SSL3_RT_INNER_CONTENT_TYPE pseudo content type used to preface the inner > TLSv1.3 content type. > As such, I think such log should be removed for brevity as typically useless. OK, removed. > > + > > + break; > > + } > > +} > > + > > + > > +int > > +SSL_provide_quic_data(SSL *ssl, enum ssl_encryption_level_t level, > > + const uint8_t *data, size_t len) > > +{ > > + BIO *rbio; > > + size_t n; > > + u_char *p; > > + ngx_str_t res; > > + ngx_connection_t *c; > > + ngx_quic_compat_t *com; > > + ngx_quic_connection_t *qc; > > + ngx_quic_compat_record_t rec; > > + u_char in[NGX_QUIC_COMPAT_RECORD_SIZE + 1]; > > + u_char out[NGX_QUIC_COMPAT_RECORD_SIZE + 1 > > + + SSL3_RT_HEADER_LENGTH > > + + EVP_GCM_TLS_TAG_LEN]; > > + > > + c = ngx_ssl_get_connection(ssl); > > + > > + ngx_log_debug2(NGX_LOG_DEBUG_EVENT, c->log, 0, "quic compat rx %s len:%uz", > > + ngx_quic_level_name(level), len); > > + > > + qc = ngx_quic_get_connection(c); > > + com = qc->compat; > > + rbio = SSL_get_rbio(ssl); > > + > > + while (len) { > > + ngx_memzero(&rec, sizeof(ngx_quic_compat_record_t)); > > + > > + rec.type = SSL3_RT_HANDSHAKE; > > + rec.log = c->log; > > + rec.number = com->read_record++; > > + rec.keys = &com->keys; > > + > > + if (level == ssl_encryption_initial) { > > + n = ngx_min(len, 65535); > > + > > + rec.payload.len = n; > > + rec.payload.data = (u_char *) data; > > + > > + ngx_quic_compat_create_header(&rec, out, 1); > > + > > + BIO_write(rbio, out, SSL3_RT_HEADER_LENGTH); > > + BIO_write(rbio, data, n); > > + > > +#if defined(NGX_QUIC_DEBUG_CRYPTO) && defined(NGX_QUIC_DEBUG_PACKETS) > > + ngx_log_debug5(NGX_LOG_DEBUG_EVENT, c->log, 0, > > + "quic compat record len:%uz %*xs%*xs", > > + n + SSL3_RT_HEADER_LENGTH, > > + (size_t) SSL3_RT_HEADER_LENGTH, out, n, data); > > +#endif > > + > > + } else { > > + n = ngx_min(len, NGX_QUIC_COMPAT_RECORD_SIZE); > > + > > + p = ngx_cpymem(in, data, n); > > + *p++ = SSL3_RT_HANDSHAKE; > > + > > + rec.payload.len = p - in; > > + rec.payload.data = in; > > + > > + res.data = out; > > + > > + if (ngx_quic_compat_create_record(&rec, &res) != NGX_OK) { > > + return 0; > > + } > > + > > +#if defined(NGX_QUIC_DEBUG_CRYPTO) && defined(NGX_QUIC_DEBUG_PACKETS) > > + ngx_log_debug2(NGX_LOG_DEBUG_EVENT, c->log, 0, > > + "quic compat record len:%uz %xV", res.len, &res); > > +#endif > > + > > + BIO_write(rbio, res.data, res.len); > > + } > > + > > + data += n; > > + len -= n; > > + } > > + > > + return 1; > > +} > > + > > + > > +static size_t > > +ngx_quic_compat_create_header(ngx_quic_compat_record_t *rec, u_char *out, > > + ngx_uint_t plain) > > +{ > > + u_char type; > > + size_t len; > > + > > + len = rec->payload.len; > > + > > + if (plain) { > > + type = rec->type; > > + > > + } else { > > + type = SSL3_RT_APPLICATION_DATA; > > + len += EVP_GCM_TLS_TAG_LEN; > > + } > > + > > + out[0] = type; > > + out[1] = 0x03; > > + out[2] = 0x03; > > + out[3] = (len >> 8); > > + out[4] = len; > > + > > + return 5; > > +} > > + > > + > > +static ngx_int_t > > +ngx_quic_compat_create_record(ngx_quic_compat_record_t *rec, ngx_str_t *res) > > +{ > > + ngx_str_t ad, out; > > + ngx_quic_secret_t *secret; > > + ngx_quic_ciphers_t ciphers; > > + u_char nonce[NGX_QUIC_IV_LEN]; > > + > > + ad.data = res->data; > > + ad.len = ngx_quic_compat_create_header(rec, ad.data, 0); > > + > > + out.len = rec->payload.len + EVP_GCM_TLS_TAG_LEN; > > + out.data = res->data + ad.len; > > + > > +#ifdef NGX_QUIC_DEBUG_CRYPTO > > + ngx_log_debug2(NGX_LOG_DEBUG_EVENT, rec->log, 0, > > + "quic compat ad len:%uz %xV", ad.len, &ad); > > +#endif > > + > > + if (ngx_quic_ciphers(rec->keys->cipher, &ciphers, rec->level) == NGX_ERROR) > > + { > > + return NGX_ERROR; > > + } > > + > > + secret = &rec->keys->secret; > > + > > + ngx_memcpy(nonce, secret->iv.data, secret->iv.len); > > + ngx_quic_compute_nonce(nonce, sizeof(nonce), rec->number); > > + > > + if (ngx_quic_tls_seal(ciphers.c, secret, &out, > > + nonce, &rec->payload, &ad, rec->log) > > + != NGX_OK) > > + { > > + return NGX_ERROR; > > + } > > + > > + res->len = ad.len + out.len; > > + > > + return NGX_OK; > > +} > > + > > + > > +enum ssl_encryption_level_t > > +SSL_quic_read_level(const SSL *ssl) > > +{ > > + ngx_connection_t *c; > > + ngx_quic_connection_t *qc; > > + > > + c = ngx_ssl_get_connection(ssl); > > + qc = ngx_quic_get_connection(c); > > + > > + return qc->compat->read_level; > > +} > > + > > + > > +enum ssl_encryption_level_t > > +SSL_quic_write_level(const SSL *ssl) > > +{ > > + ngx_connection_t *c; > > + ngx_quic_connection_t *qc; > > + > > + c = ngx_ssl_get_connection(ssl); > > + qc = ngx_quic_get_connection(c); > > + > > + return qc->compat->write_level; > > +} > > + > > + > > +int > > +SSL_set_quic_transport_params(SSL *ssl, const uint8_t *params, > > + size_t params_len) > > +{ > > + ngx_connection_t *c; > > + ngx_quic_compat_t *com; > > + ngx_quic_connection_t *qc; > > + > > + c = ngx_ssl_get_connection(ssl); > > + qc = ngx_quic_get_connection(c); > > + com = qc->compat; > > + > > + com->tp.len = params_len; > > + com->tp.data = (u_char *) params; > > + > > + return 1; > > +} > > + > > + > > +void > > +SSL_get_peer_quic_transport_params(const SSL *ssl, const uint8_t **out_params, > > + size_t *out_params_len) > > +{ > > + ngx_connection_t *c; > > + ngx_quic_compat_t *com; > > + ngx_quic_connection_t *qc; > > + > > + c = ngx_ssl_get_connection(ssl); > > + qc = ngx_quic_get_connection(c); > > + com = qc->compat; > > + > > + *out_params = com->ctp.data; > > + *out_params_len = com->ctp.len; > > +} > > + > > +#endif /* NGX_QUIC_OPENSSL_COMPAT */ > > diff --git a/src/event/quic/ngx_event_quic_openssl_compat.h b/src/event/quic/ngx_event_quic_openssl_compat.h > > new file mode 100644 > > --- /dev/null > > +++ b/src/event/quic/ngx_event_quic_openssl_compat.h > > @@ -0,0 +1,51 @@ > > + > > +/* > > + * Copyright (C) Nginx, Inc. > > + */ > > + > > + > > +#ifndef _NGX_EVENT_QUIC_OPENSSL_COMPAT_H_INCLUDED_ > > +#define _NGX_EVENT_QUIC_OPENSSL_COMPAT_H_INCLUDED_ > > + > > + > > +#include > > +#include > > + > > + > > +enum ssl_encryption_level_t { > > + ssl_encryption_initial = 0, > > + ssl_encryption_early_data, > > + ssl_encryption_handshake, > > + ssl_encryption_application > > +}; > > + > > + > > +typedef struct ssl_quic_method_st { > > + int (*set_read_secret)(SSL *ssl, enum ssl_encryption_level_t level, > > + const SSL_CIPHER *cipher, > > + const uint8_t *rsecret, size_t secret_len); > > + int (*set_write_secret)(SSL *ssl, enum ssl_encryption_level_t level, > > + const SSL_CIPHER *cipher, > > + const uint8_t *wsecret, size_t secret_len); > > bad indent Yes, thanks. > > + int (*add_handshake_data)(SSL *ssl, enum ssl_encryption_level_t level, > > + const uint8_t *data, size_t len); > > + int (*flush_flight)(SSL *ssl); > > + int (*send_alert)(SSL *ssl, enum ssl_encryption_level_t level, > > + uint8_t alert); > > +} SSL_QUIC_METHOD; > > + > > + > > +ngx_int_t ngx_quic_compat_init(SSL_CTX *ctx); > > + > > +int SSL_set_quic_method(SSL *ssl, const SSL_QUIC_METHOD *quic_method); > > +int SSL_provide_quic_data(SSL *ssl, enum ssl_encryption_level_t level, > > + const uint8_t *data, size_t len); > > +enum ssl_encryption_level_t SSL_quic_read_level(const SSL *ssl); > > +enum ssl_encryption_level_t SSL_quic_write_level(const SSL *ssl); > > +int SSL_set_quic_transport_params(SSL *ssl, const uint8_t *params, > > + size_t params_len); > > +void SSL_get_peer_quic_transport_params(const SSL *ssl, > > + const uint8_t **out_params, size_t *out_params_len); > > + > > + > > +#endif /* _NGX_EVENT_QUIC_OPENSSL_COMPAT_H_INCLUDED_ */ > > diff --git a/src/event/quic/ngx_event_quic_protection.c b/src/event/quic/ngx_event_quic_protection.c > > --- a/src/event/quic/ngx_event_quic_protection.c > > +++ b/src/event/quic/ngx_event_quic_protection.c > > @@ -23,37 +23,6 @@ > > #endif > > > > > > -#ifdef OPENSSL_IS_BORINGSSL > > -#define ngx_quic_cipher_t EVP_AEAD > > -#else > > -#define ngx_quic_cipher_t EVP_CIPHER > > -#endif > > - > > - > > -typedef struct { > > - const ngx_quic_cipher_t *c; > > - const EVP_CIPHER *hp; > > - const EVP_MD *d; > > -} ngx_quic_ciphers_t; > > - > > - > > -typedef struct { > > - size_t out_len; > > - u_char *out; > > - > > - size_t prk_len; > > - const uint8_t *prk; > > - > > - size_t label_len; > > - const u_char *label; > > -} ngx_quic_hkdf_t; > > - > > -#define ngx_quic_hkdf_set(seq, _label, _out, _prk) \ > > - (seq)->out_len = (_out)->len; (seq)->out = (_out)->data; \ > > - (seq)->prk_len = (_prk)->len, (seq)->prk = (_prk)->data, \ > > - (seq)->label_len = (sizeof(_label) - 1); (seq)->label = (u_char *)(_label); > > - > > - > > static ngx_int_t ngx_hkdf_expand(u_char *out_key, size_t out_len, > > const EVP_MD *digest, const u_char *prk, size_t prk_len, > > const u_char *info, size_t info_len); > > @@ -63,20 +32,12 @@ static ngx_int_t ngx_hkdf_extract(u_char > > > > static uint64_t ngx_quic_parse_pn(u_char **pos, ngx_int_t len, u_char *mask, > > uint64_t *largest_pn); > > -static void ngx_quic_compute_nonce(u_char *nonce, size_t len, uint64_t pn); > > -static ngx_int_t ngx_quic_ciphers(ngx_uint_t id, > > - ngx_quic_ciphers_t *ciphers, enum ssl_encryption_level_t level); > > > > static ngx_int_t ngx_quic_tls_open(const ngx_quic_cipher_t *cipher, > > ngx_quic_secret_t *s, ngx_str_t *out, u_char *nonce, ngx_str_t *in, > > ngx_str_t *ad, ngx_log_t *log); > > -static ngx_int_t ngx_quic_tls_seal(const ngx_quic_cipher_t *cipher, > > - ngx_quic_secret_t *s, ngx_str_t *out, u_char *nonce, ngx_str_t *in, > > - ngx_str_t *ad, ngx_log_t *log); > > static ngx_int_t ngx_quic_tls_hp(ngx_log_t *log, const EVP_CIPHER *cipher, > > ngx_quic_secret_t *s, u_char *out, u_char *in); > > -static ngx_int_t ngx_quic_hkdf_expand(ngx_quic_hkdf_t *hkdf, > > - const EVP_MD *digest, ngx_log_t *log); > > > > static ngx_int_t ngx_quic_create_packet(ngx_quic_header_t *pkt, > > ngx_str_t *res); > > @@ -84,7 +45,7 @@ static ngx_int_t ngx_quic_create_retry_p > > ngx_str_t *res); > > > > > > -static ngx_int_t > > +ngx_int_t > > ngx_quic_ciphers(ngx_uint_t id, ngx_quic_ciphers_t *ciphers, > > enum ssl_encryption_level_t level) > > { > > @@ -221,7 +182,7 @@ ngx_quic_keys_set_initial_secret(ngx_qui > > } > > > > > > -static ngx_int_t > > +ngx_int_t > > ngx_quic_hkdf_expand(ngx_quic_hkdf_t *h, const EVP_MD *digest, ngx_log_t *log) > > { > > size_t info_len; > > @@ -480,7 +441,7 @@ ngx_quic_tls_open(const ngx_quic_cipher_ > > } > > > > > > -static ngx_int_t > > +ngx_int_t > > ngx_quic_tls_seal(const ngx_quic_cipher_t *cipher, ngx_quic_secret_t *s, > > ngx_str_t *out, u_char *nonce, ngx_str_t *in, ngx_str_t *ad, ngx_log_t *log) > > { > > @@ -961,7 +922,7 @@ ngx_quic_parse_pn(u_char **pos, ngx_int_ > > } > > > > > > -static void > > +void > > ngx_quic_compute_nonce(u_char *nonce, size_t len, uint64_t pn) > > { > > nonce[len - 8] ^= (pn >> 56) & 0x3f; > > diff --git a/src/event/quic/ngx_event_quic_protection.h b/src/event/quic/ngx_event_quic_protection.h > > --- a/src/event/quic/ngx_event_quic_protection.h > > +++ b/src/event/quic/ngx_event_quic_protection.h > > @@ -23,6 +23,13 @@ > > #define NGX_QUIC_MAX_MD_SIZE 48 > > > > > > +#ifdef OPENSSL_IS_BORINGSSL > > +#define ngx_quic_cipher_t EVP_AEAD > > +#else > > +#define ngx_quic_cipher_t EVP_CIPHER > > +#endif > > + > > + > > typedef struct { > > size_t len; > > u_char data[NGX_QUIC_MAX_MD_SIZE]; > > @@ -56,6 +63,30 @@ struct ngx_quic_keys_s { > > }; > > > > > > +typedef struct { > > + const ngx_quic_cipher_t *c; > > + const EVP_CIPHER *hp; > > + const EVP_MD *d; > > +} ngx_quic_ciphers_t; > > + > > + > > +typedef struct { > > + size_t out_len; > > + u_char *out; > > + > > + size_t prk_len; > > + const uint8_t *prk; > > + > > + size_t label_len; > > + const u_char *label; > > +} ngx_quic_hkdf_t; > > + > > +#define ngx_quic_hkdf_set(seq, _label, _out, _prk) \ > > + (seq)->out_len = (_out)->len; (seq)->out = (_out)->data; \ > > + (seq)->prk_len = (_prk)->len, (seq)->prk = (_prk)->data, \ > > + (seq)->label_len = (sizeof(_label) - 1); (seq)->label = (u_char *)(_label); > > + > > + > > ngx_int_t ngx_quic_keys_set_initial_secret(ngx_quic_keys_t *keys, > > ngx_str_t *secret, ngx_log_t *log); > > ngx_int_t ngx_quic_keys_set_encryption_secret(ngx_log_t *log, > > @@ -70,6 +101,14 @@ void ngx_quic_keys_switch(ngx_connection > > ngx_int_t ngx_quic_keys_update(ngx_connection_t *c, ngx_quic_keys_t *keys); > > ngx_int_t ngx_quic_encrypt(ngx_quic_header_t *pkt, ngx_str_t *res); > > ngx_int_t ngx_quic_decrypt(ngx_quic_header_t *pkt, uint64_t *largest_pn); > > +void ngx_quic_compute_nonce(u_char *nonce, size_t len, uint64_t pn); > > +ngx_int_t ngx_quic_ciphers(ngx_uint_t id, ngx_quic_ciphers_t *ciphers, > > + enum ssl_encryption_level_t level); > > +ngx_int_t ngx_quic_tls_seal(const ngx_quic_cipher_t *cipher, > > + ngx_quic_secret_t *s, ngx_str_t *out, u_char *nonce, ngx_str_t *in, > > + ngx_str_t *ad, ngx_log_t *log); > > +ngx_int_t ngx_quic_hkdf_expand(ngx_quic_hkdf_t *hkdf, const EVP_MD *digest, > > + ngx_log_t *log); > > > > > > #endif /* _NGX_EVENT_QUIC_PROTECTION_H_INCLUDED_ */ > > diff --git a/src/event/quic/ngx_event_quic_ssl.c b/src/event/quic/ngx_event_quic_ssl.c > > --- a/src/event/quic/ngx_event_quic_ssl.c > > +++ b/src/event/quic/ngx_event_quic_ssl.c > > @@ -10,6 +10,13 @@ > > #include > > > > > > +#if defined OPENSSL_IS_BORINGSSL \ > > + || defined LIBRESSL_VERSION_NUMBER \ > > + || NGX_QUIC_OPENSSL_COMPAT > > +#define NGX_QUIC_BORINGSSL_API 1 > > +#endif > > + > > + > > /* > > * RFC 9000, 7.5. Cryptographic Message Buffering > > * > > @@ -18,7 +25,7 @@ > > #define NGX_QUIC_MAX_BUFFERED 65535 > > > > > > -#if defined OPENSSL_IS_BORINGSSL || defined LIBRESSL_VERSION_NUMBER > > +#if (NGX_QUIC_BORINGSSL_API) > > static int ngx_quic_set_read_secret(ngx_ssl_conn_t *ssl_conn, > > enum ssl_encryption_level_t level, const SSL_CIPHER *cipher, > > const uint8_t *secret, size_t secret_len); > > @@ -39,7 +46,7 @@ static int ngx_quic_send_alert(ngx_ssl_c > > static ngx_int_t ngx_quic_crypto_input(ngx_connection_t *c, ngx_chain_t *data); > > > > > > -#if defined OPENSSL_IS_BORINGSSL || defined LIBRESSL_VERSION_NUMBER > > +#if (NGX_QUIC_BORINGSSL_API) > > > > static int > > ngx_quic_set_read_secret(ngx_ssl_conn_t *ssl_conn, > > @@ -67,12 +74,6 @@ ngx_quic_set_read_secret(ngx_ssl_conn_t > > return 0; > > } > > > > - if (level == ssl_encryption_early_data) { > > - if (ngx_quic_init_streams(c) != NGX_OK) { > > - return 0; > > - } > > - } > > - > > return 1; > > } > > > > @@ -138,10 +139,6 @@ ngx_quic_set_encryption_secrets(ngx_ssl_ > > } > > > > if (level == ssl_encryption_early_data) { > > - if (ngx_quic_init_streams(c) != NGX_OK) { > > - return 0; > > - } > > - > > return 1; > > } > > > > @@ -456,6 +453,12 @@ ngx_quic_crypto_input(ngx_connection_t * > > return NGX_ERROR; > > } > > > > + if (ngx_quic_keys_available(qc->keys, ssl_encryption_early_data)) { > > + if (ngx_quic_init_streams(c) != NGX_OK) { > > + return NGX_ERROR; > > + } > > + } > > + > > return NGX_OK; > > } > > > > For the record, these chunks were merged. > > > @@ -540,7 +543,7 @@ ngx_quic_init_connection(ngx_connection_ > > ssl_conn = c->ssl->connection; > > > > if (!quic_method.send_alert) { > > -#if defined OPENSSL_IS_BORINGSSL || defined LIBRESSL_VERSION_NUMBER > > +#if (NGX_QUIC_BORINGSSL_API) > > quic_method.set_read_secret = ngx_quic_set_read_secret; > > quic_method.set_write_secret = ngx_quic_set_write_secret; > > #else > > diff --git a/src/event/quic/ngx_event_quic_transport.h b/src/event/quic/ngx_event_quic_transport.h > > --- a/src/event/quic/ngx_event_quic_transport.h > > +++ b/src/event/quic/ngx_event_quic_transport.h > > @@ -11,6 +11,10 @@ > > #include > > #include > > > > +#if (NGX_QUIC_OPENSSL_COMPAT) > > +#include > > +#endif > > + > > > > /* > > * RFC 9000, 17.2. Long Header Packets > > I think it make sense to adjust the compat header inclusion in > ngx_event_quic_connection.h, in order to make this part unnecessary, > and for more consistency: > > diff --git a/src/event/quic/ngx_event_quic_connection.h b/src/event/quic/ngx_eve > nt_quic_connection.h > --- a/src/event/quic/ngx_event_quic_connection.h > +++ b/src/event/quic/ngx_event_quic_connection.h > @@ -28,6 +28,9 @@ typedef struct ngx_quic_keys_s ng > typedef struct ngx_quic_compat_s ngx_quic_compat_t; Considering the following, moved ngx_quic_compat_t definition to ngx_event_quic_openssl_compat.h. If NGX_QUIC_OPENSSL_COMPAT is undefined, the definition does not make sense. > #endif > > +#if (NGX_QUIC_OPENSSL_COMPAT) > +#include > +#endif > #include > #include > #include > @@ -39,9 +42,6 @@ typedef struct ngx_quic_compat_s ng > #include > #include > #include > -#if (NGX_QUIC_OPENSSL_COMPAT) > -#include > -#endif OK. > /* RFC 9002, 6.2.2. Handshakes and New Paths: kInitialRtt */ > diff --git a/src/event/quic/ngx_event_quic_transport.h b/src/event/quic/ngx_event_quic_transport.h > --- a/src/event/quic/ngx_event_quic_transport.h > +++ b/src/event/quic/ngx_event_quic_transport.h > @@ -11,10 +11,6 @@ > #include > #include > > -#if (NGX_QUIC_OPENSSL_COMPAT) > -#include > -#endif > - > > /* > * RFC 9000, 17.2. Long Header Packets > > > In other news, I looked how to obtain 0-RTT keys without interacting > with SSL_read_early_data(), which enters the OpenSSL handshake state > machine into a special mode and causes TLSv1.3/QUIC handshake message > mismatches. This approach uses the SSL_SESSION_get_master_key() API. > In TLSv1.3, it returns PSK, which accoring to TLSv1.3 key schedule > is used as the input secret to extract Early Secret, then to derive > client_early_traffic_secret with ClientHello message transcript digest, > which we also control, so obtaining 0-RTT keys is relatively easy. > Still, OpenSSL knows nothing about we accept 0-RTT, and as such > it won't set the required early_data extension in ServerHello, > which means we rejected it, even if 0-RTT was read and acknowledged. > So the only way to switch the state seems to call SSL_read_early_data() > with all its hard-ways. > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > https://mailman.nginx.org/mailman/listinfo/nginx-devel -- Roman Arutyunyan -------------- next part -------------- # HG changeset patch # User Roman Arutyunyan # Date 1675427689 -14400 # Fri Feb 03 16:34:49 2023 +0400 # Branch quic # Node ID 9cf1fc42260e7e0e19fe5707f1b054d6499a4157 # Parent def8e398d7c50131f8dac844814fff729da5c86c QUIC: OpenSSL compatibility layer. The change allows to compile QUIC with OpenSSL which lacks BoringSSL QUIC API. This implementation does not support 0-RTT. diff --git a/README b/README --- a/README +++ b/README @@ -53,7 +53,7 @@ 1. Introduction 2. Installing - A library that provides QUIC support is required to build nginx, there + A library that provides QUIC support is recommended to build nginx, there are several of those available on the market: + BoringSSL [4] + LibreSSL [5] @@ -85,6 +85,10 @@ 2. Installing --with-cc-opt="-I../libressl/build/include" \ --with-ld-opt="-L../libressl/build/lib" + Alternatively, nginx can be configured with OpenSSL compatibility + layer, which emulates BoringSSL QUIC API for OpenSSL. This mode is + enabled by default if native QUIC support is not detected. + When configuring nginx, it's possible to enable QUIC and HTTP/3 using the following new configuration options: diff --git a/auto/lib/openssl/conf b/auto/lib/openssl/conf --- a/auto/lib/openssl/conf +++ b/auto/lib/openssl/conf @@ -10,6 +10,7 @@ if [ $OPENSSL != NONE ]; then if [ $USE_OPENSSL_QUIC = YES ]; then have=NGX_QUIC . auto/have + have=NGX_QUIC_OPENSSL_COMPAT . auto/have fi case "$CC" in @@ -124,6 +125,43 @@ else CORE_INCS="$CORE_INCS $ngx_feature_path" CORE_LIBS="$CORE_LIBS $ngx_feature_libs" OPENSSL=YES + + if [ $USE_OPENSSL_QUIC = YES ]; then + + ngx_feature="OpenSSL QUIC support" + ngx_feature_name="NGX_OPENSSL_QUIC" + ngx_feature_run=no + ngx_feature_incs="#include " + ngx_feature_test="SSL_set_quic_method(NULL, NULL)" + . auto/feature + + if [ $ngx_found = no ]; then + + ngx_feature="OpenSSL QUIC compatibility" + ngx_feature_name="NGX_QUIC_OPENSSL_COMPAT" + ngx_feature_run=no + ngx_feature_incs="#include " + ngx_feature_test=" + (void) TLS1_3_VERSION; + SSL_CTX_add_custom_ext(NULL, 0, 0, NULL, NULL, + NULL, NULL, NULL)" + . auto/feature + fi + + if [ $ngx_found = no ]; then +cat << END + +$0: error: certain modules require OpenSSL QUIC support. +You can either do not enable the modules, or install the OpenSSL library with +QUIC support into the system, or build the OpenSSL library with QUIC support +statically from the source with nginx by using --with-openssl= option. + +END + exit 1 + fi + + have=NGX_QUIC . auto/have + fi fi fi @@ -139,29 +177,4 @@ with nginx by using --with-openssl= option. - -END - exit 1 - fi - fi fi diff --git a/auto/modules b/auto/modules --- a/auto/modules +++ b/auto/modules @@ -1342,7 +1342,8 @@ if [ $USE_OPENSSL_QUIC = YES ]; then src/event/quic/ngx_event_quic_tokens.h \ src/event/quic/ngx_event_quic_ack.h \ src/event/quic/ngx_event_quic_output.h \ - src/event/quic/ngx_event_quic_socket.h" + src/event/quic/ngx_event_quic_socket.h \ + src/event/quic/ngx_event_quic_openssl_compat.h" ngx_module_srcs="src/event/quic/ngx_event_quic.c \ src/event/quic/ngx_event_quic_udp.c \ src/event/quic/ngx_event_quic_transport.c \ @@ -1355,7 +1356,8 @@ if [ $USE_OPENSSL_QUIC = YES ]; then src/event/quic/ngx_event_quic_tokens.c \ src/event/quic/ngx_event_quic_ack.c \ src/event/quic/ngx_event_quic_output.c \ - src/event/quic/ngx_event_quic_socket.c" + src/event/quic/ngx_event_quic_socket.c \ + src/event/quic/ngx_event_quic_openssl_compat.c" ngx_module_libs= ngx_module_link=YES diff --git a/src/event/ngx_event_openssl.c b/src/event/ngx_event_openssl.c --- a/src/event/ngx_event_openssl.c +++ b/src/event/ngx_event_openssl.c @@ -9,6 +9,10 @@ #include #include +#if (NGX_QUIC_OPENSSL_COMPAT) +#include +#endif + #define NGX_SSL_PASSWORD_BUFFER_SIZE 4096 diff --git a/src/event/quic/ngx_event_quic_connection.h b/src/event/quic/ngx_event_quic_connection.h --- a/src/event/quic/ngx_event_quic_connection.h +++ b/src/event/quic/ngx_event_quic_connection.h @@ -25,6 +25,9 @@ typedef struct ngx_quic_socket_s ng typedef struct ngx_quic_path_s ngx_quic_path_t; typedef struct ngx_quic_keys_s ngx_quic_keys_t; +#if (NGX_QUIC_OPENSSL_COMPAT) +#include +#endif #include #include #include @@ -236,6 +239,10 @@ struct ngx_quic_connection_s { ngx_uint_t nshadowbufs; #endif +#if (NGX_QUIC_OPENSSL_COMPAT) + ngx_quic_compat_t *compat; +#endif + ngx_quic_streams_t streams; ngx_quic_congestion_t congestion; diff --git a/src/event/quic/ngx_event_quic_openssl_compat.c b/src/event/quic/ngx_event_quic_openssl_compat.c new file mode 100644 --- /dev/null +++ b/src/event/quic/ngx_event_quic_openssl_compat.c @@ -0,0 +1,646 @@ + +/* + * Copyright (C) Nginx, Inc. + */ + + +#include +#include +#include +#include + + +#if (NGX_QUIC_OPENSSL_COMPAT) + +#define NGX_QUIC_COMPAT_RECORD_SIZE 1024 + +#define NGX_QUIC_COMPAT_SSL_TP_EXT 0x39 + +#define NGX_QUIC_COMPAT_CLIENT_HANDSHAKE "CLIENT_HANDSHAKE_TRAFFIC_SECRET" +#define NGX_QUIC_COMPAT_SERVER_HANDSHAKE "SERVER_HANDSHAKE_TRAFFIC_SECRET" +#define NGX_QUIC_COMPAT_CLIENT_APPLICATION "CLIENT_TRAFFIC_SECRET_0" +#define NGX_QUIC_COMPAT_SERVER_APPLICATION "SERVER_TRAFFIC_SECRET_0" + + +typedef struct { + ngx_quic_secret_t secret; + ngx_uint_t cipher; +} ngx_quic_compat_keys_t; + + +typedef struct { + ngx_log_t *log; + + u_char type; + ngx_str_t payload; + uint64_t number; + ngx_quic_compat_keys_t *keys; + + enum ssl_encryption_level_t level; +} ngx_quic_compat_record_t; + + +struct ngx_quic_compat_s { + const SSL_QUIC_METHOD *method; + + enum ssl_encryption_level_t write_level; + enum ssl_encryption_level_t read_level; + + uint64_t read_record; + ngx_quic_compat_keys_t keys; + + ngx_str_t tp; + ngx_str_t ctp; +}; + + +static void ngx_quic_compat_keylog_callback(const SSL *ssl, const char *line); +static ngx_int_t ngx_quic_compat_set_encryption_secret(ngx_log_t *log, + ngx_quic_compat_keys_t *keys, enum ssl_encryption_level_t level, + const SSL_CIPHER *cipher, const uint8_t *secret, size_t secret_len); +static int ngx_quic_compat_add_transport_params_callback(SSL *ssl, + unsigned int ext_type, unsigned int context, const unsigned char **out, + size_t *outlen, X509 *x, size_t chainidx, int *al, void *add_arg); +static int ngx_quic_compat_parse_transport_params_callback(SSL *ssl, + unsigned int ext_type, unsigned int context, const unsigned char *in, + size_t inlen, X509 *x, size_t chainidx, int *al, void *parse_arg); +static void ngx_quic_compat_message_callback(int write_p, int version, + int content_type, const void *buf, size_t len, SSL *ssl, void *arg); +static size_t ngx_quic_compat_create_header(ngx_quic_compat_record_t *rec, + u_char *out, ngx_uint_t plain); +static ngx_int_t ngx_quic_compat_create_record(ngx_quic_compat_record_t *rec, + ngx_str_t *res); + + +ngx_int_t +ngx_quic_compat_init(ngx_conf_t *cf, SSL_CTX *ctx) +{ + SSL_CTX_set_keylog_callback(ctx, ngx_quic_compat_keylog_callback); + + if (SSL_CTX_has_client_custom_ext(ctx, NGX_QUIC_COMPAT_SSL_TP_EXT)) { + return NGX_OK; + } + + if (SSL_CTX_add_custom_ext(ctx, NGX_QUIC_COMPAT_SSL_TP_EXT, + SSL_EXT_CLIENT_HELLO + |SSL_EXT_TLS1_3_ENCRYPTED_EXTENSIONS, + ngx_quic_compat_add_transport_params_callback, + NULL, + NULL, + ngx_quic_compat_parse_transport_params_callback, + NULL) + == 0) + { + ngx_log_error(NGX_LOG_EMERG, cf->log, 0, + "SSL_CTX_add_custom_ext() failed"); + return NGX_ERROR; + } + + return NGX_OK; +} + + +static void +ngx_quic_compat_keylog_callback(const SSL *ssl, const char *line) +{ + u_char ch, *p, *start, value; + size_t n; + ngx_uint_t write; + const SSL_CIPHER *cipher; + ngx_quic_compat_t *com; + ngx_connection_t *c; + ngx_quic_connection_t *qc; + enum ssl_encryption_level_t level; + u_char secret[EVP_MAX_MD_SIZE]; + + c = ngx_ssl_get_connection(ssl); + if (c->type != SOCK_DGRAM) { + return; + } + + p = (u_char *) line; + + for (start = p; *p && *p != ' '; p++); + + n = p - start; + + ngx_log_debug2(NGX_LOG_DEBUG_EVENT, c->log, 0, + "quic compat secret %*s", n, start); + + if (n == sizeof(NGX_QUIC_COMPAT_CLIENT_HANDSHAKE) - 1 + && ngx_strncmp(start, NGX_QUIC_COMPAT_CLIENT_HANDSHAKE, n) == 0) + { + level = ssl_encryption_handshake; + write = 0; + + } else if (n == sizeof(NGX_QUIC_COMPAT_SERVER_HANDSHAKE) - 1 + && ngx_strncmp(start, NGX_QUIC_COMPAT_SERVER_HANDSHAKE, n) == 0) + { + level = ssl_encryption_handshake; + write = 1; + + } else if (n == sizeof(NGX_QUIC_COMPAT_CLIENT_APPLICATION) - 1 + && ngx_strncmp(start, NGX_QUIC_COMPAT_CLIENT_APPLICATION, n) + == 0) + { + level = ssl_encryption_application; + write = 0; + + } else if (n == sizeof(NGX_QUIC_COMPAT_SERVER_APPLICATION) - 1 + && ngx_strncmp(start, NGX_QUIC_COMPAT_SERVER_APPLICATION, n) + == 0) + { + level = ssl_encryption_application; + write = 1; + + } else { + return; + } + + if (*p++ == '\0') { + return; + } + + for ( /* void */ ; *p && *p != ' '; p++); + + if (*p++ == '\0') { + return; + } + + for (n = 0, start = p; *p; p++) { + ch = *p; + + if (ch >= '0' && ch <= '9') { + value = ch - '0'; + goto next; + } + + ch = (u_char) (ch | 0x20); + + if (ch >= 'a' && ch <= 'f') { + value = ch - 'a' + 10; + goto next; + } + + ngx_log_error(NGX_LOG_EMERG, c->log, 0, + "invalid OpenSSL QUIC secret format"); + + return; + + next: + + if ((p - start) % 2) { + secret[n++] += value; + + } else { + if (n >= EVP_MAX_MD_SIZE) { + ngx_log_error(NGX_LOG_EMERG, c->log, 0, + "too big OpenSSL QUIC secret"); + return; + } + + secret[n] = (value << 4); + } + } + + qc = ngx_quic_get_connection(c); + com = qc->compat; + cipher = SSL_get_current_cipher(ssl); + + if (write) { + com->method->set_write_secret((SSL *) ssl, level, cipher, secret, n); + com->write_level = level; + + } else { + com->method->set_read_secret((SSL *) ssl, level, cipher, secret, n); + com->read_level = level; + com->read_record = 0; + + (void) ngx_quic_compat_set_encryption_secret(c->log, &com->keys, level, + cipher, secret, n); + } +} + + +static ngx_int_t +ngx_quic_compat_set_encryption_secret(ngx_log_t *log, + ngx_quic_compat_keys_t *keys, enum ssl_encryption_level_t level, + const SSL_CIPHER *cipher, const uint8_t *secret, size_t secret_len) +{ + ngx_int_t key_len; + ngx_str_t secret_str; + ngx_uint_t i; + ngx_quic_hkdf_t seq[2]; + ngx_quic_secret_t *peer_secret; + ngx_quic_ciphers_t ciphers; + + peer_secret = &keys->secret; + + keys->cipher = SSL_CIPHER_get_id(cipher); + + key_len = ngx_quic_ciphers(keys->cipher, &ciphers, level); + + if (key_len == NGX_ERROR) { + ngx_ssl_error(NGX_LOG_INFO, log, 0, "unexpected cipher"); + return NGX_ERROR; + } + + if (sizeof(peer_secret->secret.data) < secret_len) { + ngx_log_error(NGX_LOG_ALERT, log, 0, + "unexpected secret len: %uz", secret_len); + return NGX_ERROR; + } + + peer_secret->secret.len = secret_len; + ngx_memcpy(peer_secret->secret.data, secret, secret_len); + + peer_secret->key.len = key_len; + peer_secret->iv.len = NGX_QUIC_IV_LEN; + + secret_str.len = secret_len; + secret_str.data = (u_char *) secret; + + ngx_quic_hkdf_set(&seq[0], "tls13 key", &peer_secret->key, &secret_str); + ngx_quic_hkdf_set(&seq[1], "tls13 iv", &peer_secret->iv, &secret_str); + + for (i = 0; i < (sizeof(seq) / sizeof(seq[0])); i++) { + if (ngx_quic_hkdf_expand(&seq[i], ciphers.d, log) != NGX_OK) { + return NGX_ERROR; + } + } + + return NGX_OK; +} + + +static int +ngx_quic_compat_add_transport_params_callback(SSL *ssl, unsigned int ext_type, + unsigned int context, const unsigned char **out, size_t *outlen, X509 *x, + size_t chainidx, int *al, void *add_arg) +{ + ngx_connection_t *c; + ngx_quic_compat_t *com; + ngx_quic_connection_t *qc; + + c = ngx_ssl_get_connection(ssl); + if (c->type != SOCK_DGRAM) { + return 0; + } + + ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0, + "quic compat add transport params"); + + qc = ngx_quic_get_connection(c); + com = qc->compat; + + *out = com->tp.data; + *outlen = com->tp.len; + + return 1; +} + + +static int +ngx_quic_compat_parse_transport_params_callback(SSL *ssl, unsigned int ext_type, + unsigned int context, const unsigned char *in, size_t inlen, X509 *x, + size_t chainidx, int *al, void *parse_arg) +{ + u_char *p; + ngx_connection_t *c; + ngx_quic_compat_t *com; + ngx_quic_connection_t *qc; + + c = ngx_ssl_get_connection(ssl); + if (c->type != SOCK_DGRAM) { + return 0; + } + + ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0, + "quic compat parse transport params"); + + qc = ngx_quic_get_connection(c); + com = qc->compat; + + p = ngx_pnalloc(c->pool, inlen); + if (p == NULL) { + return 0; + } + + ngx_memcpy(p, in, inlen); + + com->ctp.data = p; + com->ctp.len = inlen; + + return 1; +} + + +int +SSL_set_quic_method(SSL *ssl, const SSL_QUIC_METHOD *quic_method) +{ + BIO *rbio, *wbio; + ngx_connection_t *c; + ngx_quic_compat_t *com; + ngx_quic_connection_t *qc; + + c = ngx_ssl_get_connection(ssl); + + ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0, "quic compat set method"); + + qc = ngx_quic_get_connection(c); + + qc->compat = ngx_pcalloc(c->pool, sizeof(ngx_quic_compat_t)); + if (qc->compat == NULL) { + return 0; + } + + com = qc->compat; + com->method = quic_method; + + rbio = BIO_new(BIO_s_mem()); + if (rbio == NULL) { + return 0; + } + + wbio = BIO_new(BIO_s_null()); + if (wbio == NULL) { + return 0; + } + + SSL_set_bio(ssl, rbio, wbio); + + SSL_set_msg_callback(ssl, ngx_quic_compat_message_callback); + + /* early data is not supported */ + SSL_set_max_early_data(ssl, 0); + + return 1; +} + + +static void +ngx_quic_compat_message_callback(int write_p, int version, int content_type, + const void *buf, size_t len, SSL *ssl, void *arg) +{ + ngx_uint_t alert; + ngx_connection_t *c; + ngx_quic_compat_t *com; + ngx_quic_connection_t *qc; + enum ssl_encryption_level_t level; + + if (!write_p) { + return; + } + + c = ngx_ssl_get_connection(ssl); + qc = ngx_quic_get_connection(c); + + if (qc == NULL) { + /* closing */ + return; + } + + com = qc->compat; + level = com->write_level; + + switch (content_type) { + + case SSL3_RT_HANDSHAKE: + ngx_log_debug2(NGX_LOG_DEBUG_EVENT, c->log, 0, + "quic compat tx %s len:%uz ", + ngx_quic_level_name(level), len); + + (void) com->method->add_handshake_data(ssl, level, buf, len); + + break; + + case SSL3_RT_ALERT: + if (len >= 2) { + alert = ((u_char *) buf)[1]; + + ngx_log_debug3(NGX_LOG_DEBUG_EVENT, c->log, 0, + "quic compat %s alert:%ui len:%uz ", + ngx_quic_level_name(level), alert, len); + + (void) com->method->send_alert(ssl, level, alert); + } + + break; + } +} + + +int +SSL_provide_quic_data(SSL *ssl, enum ssl_encryption_level_t level, + const uint8_t *data, size_t len) +{ + BIO *rbio; + size_t n; + u_char *p; + ngx_str_t res; + ngx_connection_t *c; + ngx_quic_compat_t *com; + ngx_quic_connection_t *qc; + ngx_quic_compat_record_t rec; + u_char in[NGX_QUIC_COMPAT_RECORD_SIZE + 1]; + u_char out[NGX_QUIC_COMPAT_RECORD_SIZE + 1 + + SSL3_RT_HEADER_LENGTH + + EVP_GCM_TLS_TAG_LEN]; + + c = ngx_ssl_get_connection(ssl); + + ngx_log_debug2(NGX_LOG_DEBUG_EVENT, c->log, 0, "quic compat rx %s len:%uz", + ngx_quic_level_name(level), len); + + qc = ngx_quic_get_connection(c); + com = qc->compat; + rbio = SSL_get_rbio(ssl); + + while (len) { + ngx_memzero(&rec, sizeof(ngx_quic_compat_record_t)); + + rec.type = SSL3_RT_HANDSHAKE; + rec.log = c->log; + rec.number = com->read_record++; + rec.keys = &com->keys; + + if (level == ssl_encryption_initial) { + n = ngx_min(len, 65535); + + rec.payload.len = n; + rec.payload.data = (u_char *) data; + + ngx_quic_compat_create_header(&rec, out, 1); + + BIO_write(rbio, out, SSL3_RT_HEADER_LENGTH); + BIO_write(rbio, data, n); + +#if defined(NGX_QUIC_DEBUG_CRYPTO) && defined(NGX_QUIC_DEBUG_PACKETS) + ngx_log_debug5(NGX_LOG_DEBUG_EVENT, c->log, 0, + "quic compat record len:%uz %*xs%*xs", + n + SSL3_RT_HEADER_LENGTH, + (size_t) SSL3_RT_HEADER_LENGTH, out, n, data); +#endif + + } else { + n = ngx_min(len, NGX_QUIC_COMPAT_RECORD_SIZE); + + p = ngx_cpymem(in, data, n); + *p++ = SSL3_RT_HANDSHAKE; + + rec.payload.len = p - in; + rec.payload.data = in; + + res.data = out; + + if (ngx_quic_compat_create_record(&rec, &res) != NGX_OK) { + return 0; + } + +#if defined(NGX_QUIC_DEBUG_CRYPTO) && defined(NGX_QUIC_DEBUG_PACKETS) + ngx_log_debug2(NGX_LOG_DEBUG_EVENT, c->log, 0, + "quic compat record len:%uz %xV", res.len, &res); +#endif + + BIO_write(rbio, res.data, res.len); + } + + data += n; + len -= n; + } + + return 1; +} + + +static size_t +ngx_quic_compat_create_header(ngx_quic_compat_record_t *rec, u_char *out, + ngx_uint_t plain) +{ + u_char type; + size_t len; + + len = rec->payload.len; + + if (plain) { + type = rec->type; + + } else { + type = SSL3_RT_APPLICATION_DATA; + len += EVP_GCM_TLS_TAG_LEN; + } + + out[0] = type; + out[1] = 0x03; + out[2] = 0x03; + out[3] = (len >> 8); + out[4] = len; + + return 5; +} + + +static ngx_int_t +ngx_quic_compat_create_record(ngx_quic_compat_record_t *rec, ngx_str_t *res) +{ + ngx_str_t ad, out; + ngx_quic_secret_t *secret; + ngx_quic_ciphers_t ciphers; + u_char nonce[NGX_QUIC_IV_LEN]; + + ad.data = res->data; + ad.len = ngx_quic_compat_create_header(rec, ad.data, 0); + + out.len = rec->payload.len + EVP_GCM_TLS_TAG_LEN; + out.data = res->data + ad.len; + +#ifdef NGX_QUIC_DEBUG_CRYPTO + ngx_log_debug2(NGX_LOG_DEBUG_EVENT, rec->log, 0, + "quic compat ad len:%uz %xV", ad.len, &ad); +#endif + + if (ngx_quic_ciphers(rec->keys->cipher, &ciphers, rec->level) == NGX_ERROR) + { + return NGX_ERROR; + } + + secret = &rec->keys->secret; + + ngx_memcpy(nonce, secret->iv.data, secret->iv.len); + ngx_quic_compute_nonce(nonce, sizeof(nonce), rec->number); + + if (ngx_quic_tls_seal(ciphers.c, secret, &out, + nonce, &rec->payload, &ad, rec->log) + != NGX_OK) + { + return NGX_ERROR; + } + + res->len = ad.len + out.len; + + return NGX_OK; +} + + +enum ssl_encryption_level_t +SSL_quic_read_level(const SSL *ssl) +{ + ngx_connection_t *c; + ngx_quic_connection_t *qc; + + c = ngx_ssl_get_connection(ssl); + qc = ngx_quic_get_connection(c); + + return qc->compat->read_level; +} + + +enum ssl_encryption_level_t +SSL_quic_write_level(const SSL *ssl) +{ + ngx_connection_t *c; + ngx_quic_connection_t *qc; + + c = ngx_ssl_get_connection(ssl); + qc = ngx_quic_get_connection(c); + + return qc->compat->write_level; +} + + +int +SSL_set_quic_transport_params(SSL *ssl, const uint8_t *params, + size_t params_len) +{ + ngx_connection_t *c; + ngx_quic_compat_t *com; + ngx_quic_connection_t *qc; + + c = ngx_ssl_get_connection(ssl); + qc = ngx_quic_get_connection(c); + com = qc->compat; + + com->tp.len = params_len; + com->tp.data = (u_char *) params; + + return 1; +} + + +void +SSL_get_peer_quic_transport_params(const SSL *ssl, const uint8_t **out_params, + size_t *out_params_len) +{ + ngx_connection_t *c; + ngx_quic_compat_t *com; + ngx_quic_connection_t *qc; + + c = ngx_ssl_get_connection(ssl); + qc = ngx_quic_get_connection(c); + com = qc->compat; + + *out_params = com->ctp.data; + *out_params_len = com->ctp.len; +} + +#endif /* NGX_QUIC_OPENSSL_COMPAT */ diff --git a/src/event/quic/ngx_event_quic_openssl_compat.h b/src/event/quic/ngx_event_quic_openssl_compat.h new file mode 100644 --- /dev/null +++ b/src/event/quic/ngx_event_quic_openssl_compat.h @@ -0,0 +1,60 @@ + +/* + * Copyright (C) Nginx, Inc. + */ + + +#ifndef _NGX_EVENT_QUIC_OPENSSL_COMPAT_H_INCLUDED_ +#define _NGX_EVENT_QUIC_OPENSSL_COMPAT_H_INCLUDED_ + +#ifdef TLSEXT_TYPE_quic_transport_parameters +#undef NGX_QUIC_OPENSSL_COMPAT +#else + + +#include +#include + + +typedef struct ngx_quic_compat_s ngx_quic_compat_t; + + +enum ssl_encryption_level_t { + ssl_encryption_initial = 0, + ssl_encryption_early_data, + ssl_encryption_handshake, + ssl_encryption_application +}; + + +typedef struct ssl_quic_method_st { + int (*set_read_secret)(SSL *ssl, enum ssl_encryption_level_t level, + const SSL_CIPHER *cipher, + const uint8_t *rsecret, size_t secret_len); + int (*set_write_secret)(SSL *ssl, enum ssl_encryption_level_t level, + const SSL_CIPHER *cipher, + const uint8_t *wsecret, size_t secret_len); + int (*add_handshake_data)(SSL *ssl, enum ssl_encryption_level_t level, + const uint8_t *data, size_t len); + int (*flush_flight)(SSL *ssl); + int (*send_alert)(SSL *ssl, enum ssl_encryption_level_t level, + uint8_t alert); +} SSL_QUIC_METHOD; + + +ngx_int_t ngx_quic_compat_init(ngx_conf_t *cf, SSL_CTX *ctx); + +int SSL_set_quic_method(SSL *ssl, const SSL_QUIC_METHOD *quic_method); +int SSL_provide_quic_data(SSL *ssl, enum ssl_encryption_level_t level, + const uint8_t *data, size_t len); +enum ssl_encryption_level_t SSL_quic_read_level(const SSL *ssl); +enum ssl_encryption_level_t SSL_quic_write_level(const SSL *ssl); +int SSL_set_quic_transport_params(SSL *ssl, const uint8_t *params, + size_t params_len); +void SSL_get_peer_quic_transport_params(const SSL *ssl, + const uint8_t **out_params, size_t *out_params_len); + + +#endif /* TLSEXT_TYPE_quic_transport_parameters */ + +#endif /* _NGX_EVENT_QUIC_OPENSSL_COMPAT_H_INCLUDED_ */ diff --git a/src/event/quic/ngx_event_quic_protection.c b/src/event/quic/ngx_event_quic_protection.c --- a/src/event/quic/ngx_event_quic_protection.c +++ b/src/event/quic/ngx_event_quic_protection.c @@ -23,37 +23,6 @@ #endif -#ifdef OPENSSL_IS_BORINGSSL -#define ngx_quic_cipher_t EVP_AEAD -#else -#define ngx_quic_cipher_t EVP_CIPHER -#endif - - -typedef struct { - const ngx_quic_cipher_t *c; - const EVP_CIPHER *hp; - const EVP_MD *d; -} ngx_quic_ciphers_t; - - -typedef struct { - size_t out_len; - u_char *out; - - size_t prk_len; - const uint8_t *prk; - - size_t label_len; - const u_char *label; -} ngx_quic_hkdf_t; - -#define ngx_quic_hkdf_set(seq, _label, _out, _prk) \ - (seq)->out_len = (_out)->len; (seq)->out = (_out)->data; \ - (seq)->prk_len = (_prk)->len, (seq)->prk = (_prk)->data, \ - (seq)->label_len = (sizeof(_label) - 1); (seq)->label = (u_char *)(_label); - - static ngx_int_t ngx_hkdf_expand(u_char *out_key, size_t out_len, const EVP_MD *digest, const u_char *prk, size_t prk_len, const u_char *info, size_t info_len); @@ -63,20 +32,12 @@ static ngx_int_t ngx_hkdf_extract(u_char static uint64_t ngx_quic_parse_pn(u_char **pos, ngx_int_t len, u_char *mask, uint64_t *largest_pn); -static void ngx_quic_compute_nonce(u_char *nonce, size_t len, uint64_t pn); -static ngx_int_t ngx_quic_ciphers(ngx_uint_t id, - ngx_quic_ciphers_t *ciphers, enum ssl_encryption_level_t level); static ngx_int_t ngx_quic_tls_open(const ngx_quic_cipher_t *cipher, ngx_quic_secret_t *s, ngx_str_t *out, u_char *nonce, ngx_str_t *in, ngx_str_t *ad, ngx_log_t *log); -static ngx_int_t ngx_quic_tls_seal(const ngx_quic_cipher_t *cipher, - ngx_quic_secret_t *s, ngx_str_t *out, u_char *nonce, ngx_str_t *in, - ngx_str_t *ad, ngx_log_t *log); static ngx_int_t ngx_quic_tls_hp(ngx_log_t *log, const EVP_CIPHER *cipher, ngx_quic_secret_t *s, u_char *out, u_char *in); -static ngx_int_t ngx_quic_hkdf_expand(ngx_quic_hkdf_t *hkdf, - const EVP_MD *digest, ngx_log_t *log); static ngx_int_t ngx_quic_create_packet(ngx_quic_header_t *pkt, ngx_str_t *res); @@ -84,7 +45,7 @@ static ngx_int_t ngx_quic_create_retry_p ngx_str_t *res); -static ngx_int_t +ngx_int_t ngx_quic_ciphers(ngx_uint_t id, ngx_quic_ciphers_t *ciphers, enum ssl_encryption_level_t level) { @@ -221,7 +182,7 @@ ngx_quic_keys_set_initial_secret(ngx_qui } -static ngx_int_t +ngx_int_t ngx_quic_hkdf_expand(ngx_quic_hkdf_t *h, const EVP_MD *digest, ngx_log_t *log) { size_t info_len; @@ -480,7 +441,7 @@ ngx_quic_tls_open(const ngx_quic_cipher_ } -static ngx_int_t +ngx_int_t ngx_quic_tls_seal(const ngx_quic_cipher_t *cipher, ngx_quic_secret_t *s, ngx_str_t *out, u_char *nonce, ngx_str_t *in, ngx_str_t *ad, ngx_log_t *log) { @@ -961,7 +922,7 @@ ngx_quic_parse_pn(u_char **pos, ngx_int_ } -static void +void ngx_quic_compute_nonce(u_char *nonce, size_t len, uint64_t pn) { nonce[len - 8] ^= (pn >> 56) & 0x3f; diff --git a/src/event/quic/ngx_event_quic_protection.h b/src/event/quic/ngx_event_quic_protection.h --- a/src/event/quic/ngx_event_quic_protection.h +++ b/src/event/quic/ngx_event_quic_protection.h @@ -23,6 +23,13 @@ #define NGX_QUIC_MAX_MD_SIZE 48 +#ifdef OPENSSL_IS_BORINGSSL +#define ngx_quic_cipher_t EVP_AEAD +#else +#define ngx_quic_cipher_t EVP_CIPHER +#endif + + typedef struct { size_t len; u_char data[NGX_QUIC_MAX_MD_SIZE]; @@ -56,6 +63,30 @@ struct ngx_quic_keys_s { }; +typedef struct { + const ngx_quic_cipher_t *c; + const EVP_CIPHER *hp; + const EVP_MD *d; +} ngx_quic_ciphers_t; + + +typedef struct { + size_t out_len; + u_char *out; + + size_t prk_len; + const uint8_t *prk; + + size_t label_len; + const u_char *label; +} ngx_quic_hkdf_t; + +#define ngx_quic_hkdf_set(seq, _label, _out, _prk) \ + (seq)->out_len = (_out)->len; (seq)->out = (_out)->data; \ + (seq)->prk_len = (_prk)->len, (seq)->prk = (_prk)->data, \ + (seq)->label_len = (sizeof(_label) - 1); (seq)->label = (u_char *)(_label); + + ngx_int_t ngx_quic_keys_set_initial_secret(ngx_quic_keys_t *keys, ngx_str_t *secret, ngx_log_t *log); ngx_int_t ngx_quic_keys_set_encryption_secret(ngx_log_t *log, @@ -70,6 +101,14 @@ void ngx_quic_keys_switch(ngx_connection ngx_int_t ngx_quic_keys_update(ngx_connection_t *c, ngx_quic_keys_t *keys); ngx_int_t ngx_quic_encrypt(ngx_quic_header_t *pkt, ngx_str_t *res); ngx_int_t ngx_quic_decrypt(ngx_quic_header_t *pkt, uint64_t *largest_pn); +void ngx_quic_compute_nonce(u_char *nonce, size_t len, uint64_t pn); +ngx_int_t ngx_quic_ciphers(ngx_uint_t id, ngx_quic_ciphers_t *ciphers, + enum ssl_encryption_level_t level); +ngx_int_t ngx_quic_tls_seal(const ngx_quic_cipher_t *cipher, + ngx_quic_secret_t *s, ngx_str_t *out, u_char *nonce, ngx_str_t *in, + ngx_str_t *ad, ngx_log_t *log); +ngx_int_t ngx_quic_hkdf_expand(ngx_quic_hkdf_t *hkdf, const EVP_MD *digest, + ngx_log_t *log); #endif /* _NGX_EVENT_QUIC_PROTECTION_H_INCLUDED_ */ diff --git a/src/event/quic/ngx_event_quic_ssl.c b/src/event/quic/ngx_event_quic_ssl.c --- a/src/event/quic/ngx_event_quic_ssl.c +++ b/src/event/quic/ngx_event_quic_ssl.c @@ -10,6 +10,13 @@ #include +#if defined OPENSSL_IS_BORINGSSL \ + || defined LIBRESSL_VERSION_NUMBER \ + || NGX_QUIC_OPENSSL_COMPAT +#define NGX_QUIC_BORINGSSL_API 1 +#endif + + /* * RFC 9000, 7.5. Cryptographic Message Buffering * @@ -18,7 +25,7 @@ #define NGX_QUIC_MAX_BUFFERED 65535 -#if defined OPENSSL_IS_BORINGSSL || defined LIBRESSL_VERSION_NUMBER +#if (NGX_QUIC_BORINGSSL_API) static int ngx_quic_set_read_secret(ngx_ssl_conn_t *ssl_conn, enum ssl_encryption_level_t level, const SSL_CIPHER *cipher, const uint8_t *secret, size_t secret_len); @@ -39,7 +46,7 @@ static int ngx_quic_send_alert(ngx_ssl_c static ngx_int_t ngx_quic_crypto_input(ngx_connection_t *c, ngx_chain_t *data); -#if defined OPENSSL_IS_BORINGSSL || defined LIBRESSL_VERSION_NUMBER +#if (NGX_QUIC_BORINGSSL_API) static int ngx_quic_set_read_secret(ngx_ssl_conn_t *ssl_conn, @@ -517,7 +524,7 @@ ngx_quic_init_connection(ngx_connection_ ssl_conn = c->ssl->connection; if (!quic_method.send_alert) { -#if defined OPENSSL_IS_BORINGSSL || defined LIBRESSL_VERSION_NUMBER +#if (NGX_QUIC_BORINGSSL_API) quic_method.set_read_secret = ngx_quic_set_read_secret; quic_method.set_write_secret = ngx_quic_set_write_secret; #else diff --git a/src/http/modules/ngx_http_ssl_module.c b/src/http/modules/ngx_http_ssl_module.c --- a/src/http/modules/ngx_http_ssl_module.c +++ b/src/http/modules/ngx_http_ssl_module.c @@ -9,6 +9,10 @@ #include #include +#if (NGX_QUIC_OPENSSL_COMPAT) +#include +#endif + typedef ngx_int_t (*ngx_ssl_variable_handler_pt)(ngx_connection_t *c, ngx_pool_t *pool, ngx_str_t *s); @@ -1317,16 +1321,22 @@ ngx_http_ssl_init(ngx_conf_t *cf) continue; } + cscf = addr[a].default_server; + sscf = cscf->ctx->srv_conf[ngx_http_ssl_module.ctx_index]; + if (addr[a].opt.http3) { name = "http3"; +#if (NGX_QUIC_OPENSSL_COMPAT) + if (ngx_quic_compat_init(cf, sscf->ssl.ctx) != NGX_OK) { + return NGX_ERROR; + } +#endif + } else { name = "ssl"; } - cscf = addr[a].default_server; - sscf = cscf->ctx->srv_conf[ngx_http_ssl_module.ctx_index]; - if (sscf->certificates) { if (addr[a].opt.http3 && !(sscf->protocols & NGX_SSL_TLSv1_3)) { diff --git a/src/stream/ngx_stream_ssl_module.c b/src/stream/ngx_stream_ssl_module.c --- a/src/stream/ngx_stream_ssl_module.c +++ b/src/stream/ngx_stream_ssl_module.c @@ -9,6 +9,10 @@ #include #include +#if (NGX_QUIC_OPENSSL_COMPAT) +#include +#endif + typedef ngx_int_t (*ngx_ssl_variable_handler_pt)(ngx_connection_t *c, ngx_pool_t *pool, ngx_str_t *s); @@ -1218,6 +1222,12 @@ ngx_stream_ssl_init(ngx_conf_t *cf) scf = listen[i].ctx->srv_conf[ngx_stream_ssl_module.ctx_index]; +#if (NGX_QUIC_OPENSSL_COMPAT) + if (ngx_quic_compat_init(cf, scf->ssl.ctx) != NGX_OK) { + return NGX_ERROR; + } +#endif + if (scf->certificates && !(scf->protocols & NGX_SSL_TLSv1_3)) { ngx_log_error(NGX_LOG_EMERG, cf->log, 0, "\"ssl_protocols\" must enable TLSv1.3 for " From pluknet at nginx.com Mon Feb 6 15:27:22 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Mon, 6 Feb 2023 19:27:22 +0400 Subject: [PATCH 1 of 3] HTTP/3: "quic" parameter of "listen" directive In-Reply-To: <2fcc1c60be1c89aad546.1675260068@arut-laptop> References: <2fcc1c60be1c89aad546.1675260068@arut-laptop> Message-ID: <278A9993-4B29-46C3-8E20-B01412633615@nginx.com> > On 1 Feb 2023, at 18:01, Roman Arutyunyan wrote: > > # HG changeset patch > # User Roman Arutyunyan > # Date 1675254688 -14400 > # Wed Feb 01 16:31:28 2023 +0400 > # Branch quic > # Node ID 2fcc1c60be1c89aad5464bcc06f1189d1adc885a > # Parent def8e398d7c50131f8dac844814fff729da5c86c > HTTP/3: "quic" parameter of "listen" directive. > > Now "listen" directve has a new "quic" parameter which enables QUIC protocol > for the address. Further, to enable HTTP/3, a new directive "http3" is > introduced. The hq-interop protocol is enabled by "http3_hq" as before. > Now application protocol is chosen by ALPN. > > Previously used "http3" parameter of "listen" is deprecated. Looks good in general, minor comments below. > > diff --git a/README b/README > --- a/README > +++ b/README > @@ -93,13 +93,13 @@ 2. Installing > > 3. Configuration > > - The HTTP "listen" directive got a new option "http3" which enables > - HTTP/3 over QUIC on the specified port. > + The HTTP "listen" directive got a new option "quic" which enables > + QUIC as client transport protocol instead of TCP. > > The Stream "listen" directive got a new option "quic" which enables > QUIC as client transport protocol instead of TCP or plain UDP. > > - Along with "http3" or "quic", it's also possible to specify "reuseport" > + Along with "quic", it's also possible to specify "reuseport" > option [8] to make it work properly with multiple workers. > > To enable address validation: > @@ -133,12 +133,13 @@ 3. Configuration > > A number of directives were added that configure HTTP/3: > > + http3 > + http3_hq > http3_stream_buffer_size > http3_max_concurrent_pushes > http3_max_concurrent_streams > http3_push > http3_push_preload > - http3_hq (requires NGX_HTTP_V3_HQ macro) > > In http, an additional variable is available: $http3. > The value of $http3 is "h3" for HTTP/3 connections, > @@ -160,13 +161,15 @@ Example configuration: > server { > # for better compatibility it's recommended > # to use the same port for quic and https > - listen 8443 http3 reuseport; > + listen 8443 quic reuseport; > listen 8443 ssl; > > ssl_certificate certs/example.com.crt; > ssl_certificate_key certs/example.com.key; > ssl_protocols TLSv1.3; > > + http3 on; > + > location / { > # required for browsers to direct them into quic port > add_header Alt-Svc 'h3=":8443"; ma=86400'; > diff --git a/src/http/modules/ngx_http_ssl_module.c b/src/http/modules/ngx_http_ssl_module.c > --- a/src/http/modules/ngx_http_ssl_module.c > +++ b/src/http/modules/ngx_http_ssl_module.c > @@ -427,7 +427,7 @@ ngx_http_ssl_alpn_select(ngx_ssl_conn_t > #if (NGX_HTTP_V2 || NGX_HTTP_V3) > ngx_http_connection_t *hc; > #endif > -#if (NGX_HTTP_V3 && NGX_HTTP_V3_HQ) > +#if (NGX_HTTP_V3) > ngx_http_v3_srv_conf_t *h3scf; > #endif > #if (NGX_HTTP_V2 || NGX_HTTP_V3 || NGX_DEBUG) > @@ -455,19 +455,26 @@ ngx_http_ssl_alpn_select(ngx_ssl_conn_t > } else > #endif > #if (NGX_HTTP_V3) > - if (hc->addr_conf->http3) { > + if (hc->addr_conf->quic) { > > -#if (NGX_HTTP_V3_HQ) > h3scf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_v3_module); > > - if (h3scf->hq) { > + if (h3scf->enable && h3scf->enable_hq) { > + srv = (unsigned char *) NGX_HTTP_V3_ALPN_PROTO > + NGX_HTTP_V3_HQ_ALPN_PROTO; > + srvlen = sizeof(NGX_HTTP_V3_ALPN_PROTO NGX_HTTP_V3_HQ_ALPN_PROTO) > + - 1; > + > + } else if (h3scf->enable_hq) { > srv = (unsigned char *) NGX_HTTP_V3_HQ_ALPN_PROTO; > srvlen = sizeof(NGX_HTTP_V3_HQ_ALPN_PROTO) - 1; > - } else > -#endif > - { > + > + } else if (h3scf->enable || hc->addr_conf->http3) { > srv = (unsigned char *) NGX_HTTP_V3_ALPN_PROTO; > srvlen = sizeof(NGX_HTTP_V3_ALPN_PROTO) - 1; > + > + } else { > + return SSL_TLSEXT_ERR_ALERT_FATAL; > } > > } else > @@ -1313,12 +1320,12 @@ ngx_http_ssl_init(ngx_conf_t *cf) > addr = port[p].addrs.elts; > for (a = 0; a < port[p].addrs.nelts; a++) { > > - if (!addr[a].opt.ssl && !addr[a].opt.http3) { > + if (!addr[a].opt.ssl && !addr[a].opt.quic) { > continue; > } > > - if (addr[a].opt.http3) { > - name = "http3"; > + if (addr[a].opt.quic) { > + name = "quic"; > > } else { > name = "ssl"; > @@ -1329,7 +1336,7 @@ ngx_http_ssl_init(ngx_conf_t *cf) > > if (sscf->certificates) { > > - if (addr[a].opt.http3 && !(sscf->protocols & NGX_SSL_TLSv1_3)) { > + if (addr[a].opt.quic && !(sscf->protocols & NGX_SSL_TLSv1_3)) { > ngx_log_error(NGX_LOG_EMERG, cf->log, 0, > "\"ssl_protocols\" must enable TLSv1.3 for " > "the \"listen ... %s\" directive in %s:%ui", > diff --git a/src/http/ngx_http.c b/src/http/ngx_http.c > --- a/src/http/ngx_http.c > +++ b/src/http/ngx_http.c > @@ -1242,6 +1242,7 @@ ngx_http_add_addresses(ngx_conf_t *cf, n > #endif > #if (NGX_HTTP_V3) > ngx_uint_t http3; > + ngx_uint_t quic; > #endif > > /* > @@ -1280,6 +1281,7 @@ ngx_http_add_addresses(ngx_conf_t *cf, n > #endif > #if (NGX_HTTP_V3) > http3 = lsopt->http3 || addr[i].opt.http3; > + quic = lsopt->quic || addr[i].opt.quic; > #endif > > if (lsopt->set) { > @@ -1319,6 +1321,7 @@ ngx_http_add_addresses(ngx_conf_t *cf, n > #endif > #if (NGX_HTTP_V3) > addr[i].opt.http3 = http3; > + addr[i].opt.quic = quic; > #endif > > return NGX_OK; > @@ -1823,7 +1826,7 @@ ngx_http_add_listening(ngx_conf_t *cf, n > > #if (NGX_HTTP_V3) > > - ls->quic = addr->opt.http3; > + ls->quic = addr->opt.quic; > > if (ls->quic) { > ngx_rbtree_init(&ls->rbtree, &ls->sentinel, > @@ -1866,6 +1869,7 @@ ngx_http_add_addrs(ngx_conf_t *cf, ngx_h > #endif > #if (NGX_HTTP_V3) > addrs[i].conf.http3 = addr[i].opt.http3; > + addrs[i].conf.quic = addr[i].opt.quic; > #endif > addrs[i].conf.proxy_protocol = addr[i].opt.proxy_protocol; > > @@ -1934,6 +1938,7 @@ ngx_http_add_addrs6(ngx_conf_t *cf, ngx_ > #endif > #if (NGX_HTTP_V3) > addrs6[i].conf.http3 = addr[i].opt.http3; > + addrs6[i].conf.quic = addr[i].opt.quic; > #endif > addrs6[i].conf.proxy_protocol = addr[i].opt.proxy_protocol; > > diff --git a/src/http/ngx_http_core_module.c b/src/http/ngx_http_core_module.c > --- a/src/http/ngx_http_core_module.c > +++ b/src/http/ngx_http_core_module.c > @@ -4191,6 +4191,11 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx > > if (ngx_strcmp(value[n].data, "http3") == 0) { > #if (NGX_HTTP_V3) > + ngx_conf_log_error(NGX_LOG_WARN, cf, 0, > + "the \"listen ... http3\" directive " > + "is deprecated, use " > + "the \"http3\" directive instead"); > + lsopt.quic = 1; It can be unclear that we need the "quic" parameter. > lsopt.http3 = 1; > lsopt.type = SOCK_DGRAM; > continue; > @@ -4202,6 +4207,19 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx > #endif > } > > + if (ngx_strcmp(value[n].data, "quic") == 0) { > +#if (NGX_HTTP_V3) > + lsopt.quic = 1; > + lsopt.type = SOCK_DGRAM; > + continue; > +#else > + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, > + "the \"quic\" parameter requires " > + "ngx_http_v3_module"); > + return NGX_CONF_ERROR; > +#endif > + } > + > if (ngx_strncmp(value[n].data, "so_keepalive=", 13) == 0) { > > if (ngx_strcmp(&value[n].data[13], "on") == 0) { > @@ -4304,8 +4322,8 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx > } > > #if (NGX_HTTP_SSL && NGX_HTTP_V3) > - if (lsopt.ssl && lsopt.http3) { > - return "\"ssl\" parameter is incompatible with \"http3\""; > + if (lsopt.ssl && lsopt.quic) { > + return "\"ssl\" parameter is incompatible with \"quic\""; > } > #endif > > diff --git a/src/http/ngx_http_core_module.h b/src/http/ngx_http_core_module.h > --- a/src/http/ngx_http_core_module.h > +++ b/src/http/ngx_http_core_module.h > @@ -76,6 +76,7 @@ typedef struct { > unsigned ssl:1; > unsigned http2:1; > unsigned http3:1; > + unsigned quic:1; > #if (NGX_HAVE_INET6) > unsigned ipv6only:1; > #endif > @@ -240,6 +241,7 @@ struct ngx_http_addr_conf_s { > unsigned ssl:1; > unsigned http2:1; > unsigned http3:1; > + unsigned quic:1; > unsigned proxy_protocol:1; > }; > > diff --git a/src/http/ngx_http_request.c b/src/http/ngx_http_request.c > --- a/src/http/ngx_http_request.c > +++ b/src/http/ngx_http_request.c > @@ -325,7 +325,7 @@ ngx_http_init_connection(ngx_connection_ > #endif > > #if (NGX_HTTP_V3) > - if (hc->addr_conf->http3) { > + if (hc->addr_conf->quic) { > ngx_http_v3_init_stream(c); > return; > } > diff --git a/src/http/v3/ngx_http_v3.c b/src/http/v3/ngx_http_v3.c > --- a/src/http/v3/ngx_http_v3.c > +++ b/src/http/v3/ngx_http_v3.c > @@ -17,12 +17,9 @@ static void ngx_http_v3_cleanup_session( > ngx_int_t > ngx_http_v3_init_session(ngx_connection_t *c) > { > - ngx_pool_cleanup_t *cln; > - ngx_http_connection_t *hc; > - ngx_http_v3_session_t *h3c; > -#if (NGX_HTTP_V3_HQ) > - ngx_http_v3_srv_conf_t *h3scf; > -#endif > + ngx_pool_cleanup_t *cln; > + ngx_http_connection_t *hc; > + ngx_http_v3_session_t *h3c; > > hc = c->data; > > @@ -36,13 +33,6 @@ ngx_http_v3_init_session(ngx_connection_ > h3c->max_push_id = (uint64_t) -1; > h3c->goaway_push_id = (uint64_t) -1; > > -#if (NGX_HTTP_V3_HQ) > - h3scf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_v3_module); > - if (h3scf->hq) { > - h3c->hq = 1; > - } > -#endif > - > ngx_queue_init(&h3c->blocked); > ngx_queue_init(&h3c->pushing); > > diff --git a/src/http/v3/ngx_http_v3.h b/src/http/v3/ngx_http_v3.h > --- a/src/http/v3/ngx_http_v3.h > +++ b/src/http/v3/ngx_http_v3.h > @@ -21,6 +21,7 @@ > > #define NGX_HTTP_V3_ALPN_PROTO "\x02h3" > #define NGX_HTTP_V3_HQ_ALPN_PROTO "\x0Ahq-interop" > +#define NGX_HTTP_V3_HQ_PROTO "hq-interop" > > #define NGX_HTTP_V3_VARLEN_INT_LEN 4 > #define NGX_HTTP_V3_PREFIX_INT_LEN 11 > @@ -101,13 +102,12 @@ > > > typedef struct { > + ngx_flag_t enable; > + ngx_flag_t enable_hq; > size_t max_table_capacity; > ngx_uint_t max_blocked_streams; > ngx_uint_t max_concurrent_pushes; > ngx_uint_t max_concurrent_streams; > -#if (NGX_HTTP_V3_HQ) > - ngx_flag_t hq; > -#endif > ngx_quic_conf_t quic; > } ngx_http_v3_srv_conf_t; > > @@ -147,9 +147,7 @@ struct ngx_http_v3_session_s { > off_t payload_bytes; > > unsigned goaway:1; > -#if (NGX_HTTP_V3_HQ) > unsigned hq:1; > -#endif > > ngx_connection_t *known_streams[NGX_HTTP_V3_MAX_KNOWN_STREAM]; > }; > diff --git a/src/http/v3/ngx_http_v3_module.c b/src/http/v3/ngx_http_v3_module.c > --- a/src/http/v3/ngx_http_v3_module.c > +++ b/src/http/v3/ngx_http_v3_module.c > @@ -32,6 +32,20 @@ static ngx_conf_post_t ngx_http_quic_mt > > static ngx_command_t ngx_http_v3_commands[] = { > > + { ngx_string("http3"), > + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_CONF_FLAG, > + ngx_conf_set_flag_slot, > + NGX_HTTP_SRV_CONF_OFFSET, > + offsetof(ngx_http_v3_srv_conf_t, enable), > + NULL }, > + > + { ngx_string("http3_hq"), > + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_CONF_FLAG, > + ngx_conf_set_flag_slot, > + NGX_HTTP_SRV_CONF_OFFSET, > + offsetof(ngx_http_v3_srv_conf_t, enable_hq), > + NULL }, > + > { ngx_string("http3_max_concurrent_pushes"), > NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_CONF_TAKE1, > ngx_conf_set_num_slot, > @@ -46,15 +60,6 @@ static ngx_command_t ngx_http_v3_comman > offsetof(ngx_http_v3_srv_conf_t, max_concurrent_streams), > NULL }, > > -#if (NGX_HTTP_V3_HQ) > - { ngx_string("http3_hq"), > - NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_CONF_FLAG, > - ngx_conf_set_flag_slot, > - NGX_HTTP_SRV_CONF_OFFSET, > - offsetof(ngx_http_v3_srv_conf_t, hq), > - NULL }, > -#endif > - > { ngx_string("http3_push"), > NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1, > ngx_http_v3_push, > @@ -160,14 +165,12 @@ static ngx_int_t > ngx_http_v3_variable(ngx_http_request_t *r, > ngx_http_variable_value_t *v, uintptr_t data) > { > - if (r->connection->quic) { > -#if (NGX_HTTP_V3_HQ) > + ngx_http_v3_session_t *h3c; > > - ngx_http_v3_srv_conf_t *h3scf; > + if (r->connection->quic) { > + h3c = ngx_http_v3_get_session(r->connection); > > - h3scf = ngx_http_get_module_srv_conf(r, ngx_http_v3_module); > - > - if (h3scf->hq) { > + if (h3c->hq) { > v->len = sizeof("hq") - 1; > v->valid = 1; > v->no_cacheable = 0; > @@ -177,8 +180,6 @@ ngx_http_v3_variable(ngx_http_request_t > return NGX_OK; > } > > -#endif > - > v->len = sizeof("h3") - 1; > v->valid = 1; > v->no_cacheable = 0; > @@ -232,12 +233,12 @@ ngx_http_v3_create_srv_conf(ngx_conf_t * > * h3scf->quic.timeout = 0; > * h3scf->max_blocked_streams = 0; > */ > + > + h3scf->enable = NGX_CONF_UNSET; > + h3scf->enable_hq = NGX_CONF_UNSET; > h3scf->max_table_capacity = NGX_HTTP_V3_MAX_TABLE_CAPACITY; > h3scf->max_concurrent_pushes = NGX_CONF_UNSET_UINT; > h3scf->max_concurrent_streams = NGX_CONF_UNSET_UINT; > -#if (NGX_HTTP_V3_HQ) > - h3scf->hq = NGX_CONF_UNSET; > -#endif > > h3scf->quic.mtu = NGX_CONF_UNSET_SIZE; > h3scf->quic.stream_buffer_size = NGX_CONF_UNSET_SIZE; > @@ -264,6 +265,10 @@ ngx_http_v3_merge_srv_conf(ngx_conf_t *c > > ngx_http_ssl_srv_conf_t *sscf; > > + ngx_conf_merge_value(conf->enable, prev->enable, 0); > + > + ngx_conf_merge_value(conf->enable_hq, prev->enable_hq, 0); > + > ngx_conf_merge_uint_value(conf->max_concurrent_pushes, > prev->max_concurrent_pushes, 10); > > @@ -272,11 +277,6 @@ ngx_http_v3_merge_srv_conf(ngx_conf_t *c > > conf->max_blocked_streams = conf->max_concurrent_streams; > > -#if (NGX_HTTP_V3_HQ) > - ngx_conf_merge_value(conf->hq, prev->hq, 0); > -#endif > - > - > ngx_conf_merge_size_value(conf->quic.mtu, prev->quic.mtu, > NGX_QUIC_MAX_UDP_PAYLOAD_SIZE); > > diff --git a/src/http/v3/ngx_http_v3_request.c b/src/http/v3/ngx_http_v3_request.c > --- a/src/http/v3/ngx_http_v3_request.c > +++ b/src/http/v3/ngx_http_v3_request.c > @@ -110,7 +110,10 @@ ngx_http_v3_init_stream(ngx_connection_t > ngx_int_t > ngx_http_v3_init(ngx_connection_t *c) > { > + unsigned int len; > + const unsigned char *data; > ngx_http_v3_session_t *h3c; > + ngx_http_v3_srv_conf_t *h3scf; > ngx_http_core_loc_conf_t *clcf; > > ngx_log_debug0(NGX_LOG_DEBUG_HTTP, c->log, 0, "http3 init"); > @@ -119,11 +122,23 @@ ngx_http_v3_init(ngx_connection_t *c) > clcf = ngx_http_v3_get_module_loc_conf(c, ngx_http_core_module); > ngx_add_timer(&h3c->keepalive, clcf->keepalive_timeout); > > -#if (NGX_HTTP_V3_HQ) > - if (h3c->hq) { > - return NGX_OK; > + h3scf = ngx_http_v3_get_module_srv_conf(c, ngx_http_v3_module); > + > + if (h3scf->enable_hq) { > + if (!h3scf->enable) { > + h3c->hq = 1; > + return NGX_OK; > + } > + > + SSL_get0_alpn_selected(c->ssl->connection, &data, &len); > + > + if (len == sizeof(NGX_HTTP_V3_HQ_PROTO) - 1 > + && ngx_strncmp(data, NGX_HTTP_V3_HQ_PROTO, len) == 0) > + { > + h3c->hq = 1; > + return NGX_OK; > + } > } > -#endif > > return ngx_http_v3_send_settings(c); > } > @@ -147,10 +162,7 @@ ngx_http_v3_shutdown(ngx_connection_t *c > if (!h3c->goaway) { > h3c->goaway = 1; > > -#if (NGX_HTTP_V3_HQ) > - if (!h3c->hq) > -#endif > - { > + if (!h3c->hq) { > (void) ngx_http_v3_send_goaway(c, h3c->next_request_id); > } > > @@ -205,10 +217,7 @@ ngx_http_v3_init_request_stream(ngx_conn > { > h3c->goaway = 1; > > -#if (NGX_HTTP_V3_HQ) > - if (!h3c->hq) > -#endif > - { > + if (!h3c->hq) { > if (ngx_http_v3_send_goaway(c, h3c->next_request_id) != NGX_OK) { > ngx_http_close_connection(c); > return; > @@ -236,10 +245,7 @@ ngx_http_v3_init_request_stream(ngx_conn > > rev = c->read; > > -#if (NGX_HTTP_V3_HQ) > - if (!h3c->hq) > -#endif > - { > + if (!h3c->hq) { > rev->handler = ngx_http_v3_wait_request_handler; > c->write->handler = ngx_http_empty_handler; > } > @@ -398,14 +404,14 @@ ngx_http_v3_wait_request_handler(ngx_eve > void > ngx_http_v3_reset_stream(ngx_connection_t *c) > { > + ngx_http_v3_session_t *h3c; > ngx_http_v3_srv_conf_t *h3scf; > > h3scf = ngx_http_v3_get_module_srv_conf(c, ngx_http_v3_module); > > - if (h3scf->max_table_capacity > 0 && !c->read->eof > -#if (NGX_HTTP_V3_HQ) > - && !h3scf->hq > -#endif > + h3c = ngx_http_v3_get_session(c); > + > + if (h3scf->max_table_capacity > 0 && !c->read->eof && !h3c->hq > && (c->quic->id & NGX_QUIC_STREAM_UNIDIRECTIONAL) == 0) > { > (void) ngx_http_v3_send_cancel_stream(c, c->quic->id); > @@ -993,9 +999,11 @@ failed: > static ngx_int_t > ngx_http_v3_process_request_header(ngx_http_request_t *r) > { > - ssize_t n; > - ngx_buf_t *b; > - ngx_connection_t *c; > + ssize_t n; > + ngx_buf_t *b; > + ngx_connection_t *c; > + ngx_http_v3_session_t *h3c; > + ngx_http_v3_srv_conf_t *h3scf; > > c = r->connection; > > @@ -1003,6 +1011,19 @@ ngx_http_v3_process_request_header(ngx_h > return NGX_ERROR; > } > > + h3c = ngx_http_v3_get_session(c); > + h3scf = ngx_http_get_module_srv_conf(r, ngx_http_v3_module); > + > + if (!r->http_connection->addr_conf->http3) { > + if ((h3c->hq && !h3scf->enable_hq) || (!h3c->hq && !h3scf->enable)) { can be reduced, hq is always false in the right part: if ((h3c->hq && !h3scf->enable_hq) || !h3scf->enable) { > + ngx_log_error(NGX_LOG_INFO, c->log, 0, > + "client attempted to request the server name " > + "for which the negotitated protocol is unavailable"); typo: negotiated s/unavailable/disabled/ ? > + ngx_http_finalize_request(r, NGX_HTTP_MISDIRECTED_REQUEST); > + return NGX_ERROR; > + } > + } > + > if (ngx_http_v3_construct_cookie_header(r) != NGX_OK) { > return NGX_ERROR; > } > diff --git a/src/http/v3/ngx_http_v3_uni.c b/src/http/v3/ngx_http_v3_uni.c > --- a/src/http/v3/ngx_http_v3_uni.c > +++ b/src/http/v3/ngx_http_v3_uni.c > @@ -37,12 +37,9 @@ void > ngx_http_v3_init_uni_stream(ngx_connection_t *c) > { > uint64_t n; > -#if (NGX_HTTP_V3_HQ) > ngx_http_v3_session_t *h3c; > -#endif > ngx_http_v3_uni_stream_t *us; > > -#if (NGX_HTTP_V3_HQ) > h3c = ngx_http_v3_get_session(c); > if (h3c->hq) { > ngx_http_v3_finalize_connection(c, > @@ -52,7 +49,6 @@ ngx_http_v3_init_uni_stream(ngx_connecti > ngx_http_v3_close_uni_stream(c); > return; > } > -#endif > > ngx_log_debug0(NGX_LOG_DEBUG_HTTP, c->log, 0, "http3 init uni stream"); > -- Sergey Kandaurov From mdounin at mdounin.ru Tue Feb 7 03:12:01 2023 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 7 Feb 2023 06:12:01 +0300 Subject: [PATCH 1 of 3] HTTP/3: "quic" parameter of "listen" directive In-Reply-To: <2fcc1c60be1c89aad546.1675260068@arut-laptop> References: <2fcc1c60be1c89aad546.1675260068@arut-laptop> Message-ID: Hello! On Wed, Feb 01, 2023 at 06:01:08PM +0400, Roman Arutyunyan wrote: > # HG changeset patch > # User Roman Arutyunyan > # Date 1675254688 -14400 > # Wed Feb 01 16:31:28 2023 +0400 > # Branch quic > # Node ID 2fcc1c60be1c89aad5464bcc06f1189d1adc885a > # Parent def8e398d7c50131f8dac844814fff729da5c86c > HTTP/3: "quic" parameter of "listen" directive. > > Now "listen" directve has a new "quic" parameter which enables QUIC protocol > for the address. Further, to enable HTTP/3, a new directive "http3" is > introduced. The hq-interop protocol is enabled by "http3_hq" as before. > Now application protocol is chosen by ALPN. > > Previously used "http3" parameter of "listen" is deprecated. > > diff --git a/README b/README > --- a/README > +++ b/README > @@ -93,13 +93,13 @@ 2. Installing > > 3. Configuration > > - The HTTP "listen" directive got a new option "http3" which enables > - HTTP/3 over QUIC on the specified port. > + The HTTP "listen" directive got a new option "quic" which enables > + QUIC as client transport protocol instead of TCP. > > The Stream "listen" directive got a new option "quic" which enables > QUIC as client transport protocol instead of TCP or plain UDP. > > - Along with "http3" or "quic", it's also possible to specify "reuseport" > + Along with "quic", it's also possible to specify "reuseport" > option [8] to make it work properly with multiple workers. > > To enable address validation: > @@ -133,12 +133,13 @@ 3. Configuration > > A number of directives were added that configure HTTP/3: > > + http3 > + http3_hq > http3_stream_buffer_size > http3_max_concurrent_pushes > http3_max_concurrent_streams > http3_push > http3_push_preload > - http3_hq (requires NGX_HTTP_V3_HQ macro) > > In http, an additional variable is available: $http3. > The value of $http3 is "h3" for HTTP/3 connections, > @@ -160,13 +161,15 @@ Example configuration: > server { > # for better compatibility it's recommended > # to use the same port for quic and https > - listen 8443 http3 reuseport; > + listen 8443 quic reuseport; > listen 8443 ssl; > > ssl_certificate certs/example.com.crt; > ssl_certificate_key certs/example.com.key; > ssl_protocols TLSv1.3; > > + http3 on; > + Shouldn't "http3" be "on" by default, so it will be used for all QUIC listening sockets automatically (unless disabled), basically matching the "listen ... http3" behaviour? Not sure though, "off" by default seems to better match generic approach and prevents if from being enabled accidentally. On the other hand, "listen ... quic" has little to no uses without HTTP/3 being enabled, and requirement to explicitly enable it looks somewhat silly. [...] > diff --git a/src/http/ngx_http_core_module.c b/src/http/ngx_http_core_module.c > --- a/src/http/ngx_http_core_module.c > +++ b/src/http/ngx_http_core_module.c > @@ -4191,6 +4191,11 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx > > if (ngx_strcmp(value[n].data, "http3") == 0) { > #if (NGX_HTTP_V3) > + ngx_conf_log_error(NGX_LOG_WARN, cf, 0, > + "the \"listen ... http3\" directive " > + "is deprecated, use " > + "the \"http3\" directive instead"); Note that this doesn't provide enough information to rewrite the configuration. Something like "use the \"listen ... quic\" and \"http3\" directives instead" should be better (assuming "http3" is off by default). > + lsopt.quic = 1; > lsopt.http3 = 1; > lsopt.type = SOCK_DGRAM; > continue; [...] -- Maxim Dounin http://mdounin.ru/ From mdounin at mdounin.ru Tue Feb 7 03:12:50 2023 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 7 Feb 2023 06:12:50 +0300 Subject: [PATCH 2 of 3] HTTP/2: "http2" directive In-Reply-To: <139b348815b3f1975317.1675260069@arut-laptop> References: <139b348815b3f1975317.1675260069@arut-laptop> Message-ID: Hello! On Wed, Feb 01, 2023 at 06:01:09PM +0400, Roman Arutyunyan wrote: > # HG changeset patch > # User Roman Arutyunyan > # Date 1675255790 -14400 > # Wed Feb 01 16:49:50 2023 +0400 > # Branch quic > # Node ID 139b348815b3f19753176988f06b590a3e0af4c0 > # Parent 2fcc1c60be1c89aad5464bcc06f1189d1adc885a > HTTP/2: "http2" directive. > > The directive enables HTTP/2 in the current server. The previous way to > enable HTTP/2 via "listen ... http2" is now deprecated. It might be a good idea to describe here how it works, notably ALPN negotiation (and OpenSSL version where it works for non-default virtual servers) and preface-based protocol detection for non-SSL listening sockets. As well as benefits of this approach. > > diff --git a/src/http/modules/ngx_http_ssl_module.c b/src/http/modules/ngx_http_ssl_module.c > --- a/src/http/modules/ngx_http_ssl_module.c > +++ b/src/http/modules/ngx_http_ssl_module.c > @@ -427,6 +427,9 @@ ngx_http_ssl_alpn_select(ngx_ssl_conn_t > #if (NGX_HTTP_V2 || NGX_HTTP_V3) > ngx_http_connection_t *hc; > #endif > +#if (NGX_HTTP_V2) > + ngx_http_v2_srv_conf_t *h2scf; > +#endif > #if (NGX_HTTP_V3) > ngx_http_v3_srv_conf_t *h3scf; > #endif > @@ -448,12 +451,9 @@ ngx_http_ssl_alpn_select(ngx_ssl_conn_t > hc = c->data; > #endif > > -#if (NGX_HTTP_V2) > - if (hc->addr_conf->http2) { > - srv = (unsigned char *) NGX_HTTP_V2_ALPN_PROTO NGX_HTTP_ALPN_PROTOS; > - srvlen = sizeof(NGX_HTTP_V2_ALPN_PROTO NGX_HTTP_ALPN_PROTOS) - 1; > - } else > -#endif > + srv = (unsigned char *) NGX_HTTP_ALPN_PROTOS; > + srvlen = sizeof(NGX_HTTP_ALPN_PROTOS) - 1; > + > #if (NGX_HTTP_V3) > if (hc->addr_conf->quic) { > > @@ -479,10 +479,16 @@ ngx_http_ssl_alpn_select(ngx_ssl_conn_t > > } else > #endif > +#if (NGX_HTTP_V2) > { > - srv = (unsigned char *) NGX_HTTP_ALPN_PROTOS; > - srvlen = sizeof(NGX_HTTP_ALPN_PROTOS) - 1; > + h2scf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_v2_module); > + > + if (h2scf->enable || hc->addr_conf->http2) { > + srv = (unsigned char *) NGX_HTTP_V2_ALPN_PROTO NGX_HTTP_ALPN_PROTOS; > + srvlen = sizeof(NGX_HTTP_V2_ALPN_PROTO NGX_HTTP_ALPN_PROTOS) - 1; > + } > } > +#endif > > if (SSL_select_next_proto((unsigned char **) out, outlen, srv, srvlen, > in, inlen) > diff --git a/src/http/ngx_http_core_module.c b/src/http/ngx_http_core_module.c > --- a/src/http/ngx_http_core_module.c > +++ b/src/http/ngx_http_core_module.c > @@ -4179,6 +4179,11 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx > > if (ngx_strcmp(value[n].data, "http2") == 0) { > #if (NGX_HTTP_V2) > + ngx_conf_log_error(NGX_LOG_WARN, cf, 0, > + "the \"listen ... http2\" directive " > + "is deprecated, use " > + "the \"http2\" directive instead"); > + > lsopt.http2 = 1; > continue; > #else > diff --git a/src/http/ngx_http_request.c b/src/http/ngx_http_request.c > --- a/src/http/ngx_http_request.c > +++ b/src/http/ngx_http_request.c > @@ -318,12 +318,6 @@ ngx_http_init_connection(ngx_connection_ > rev->handler = ngx_http_wait_request_handler; > c->write->handler = ngx_http_empty_handler; > > -#if (NGX_HTTP_V2) > - if (hc->addr_conf->http2) { > - rev->handler = ngx_http_v2_init; > - } > -#endif > - > #if (NGX_HTTP_V3) > if (hc->addr_conf->quic) { > ngx_http_v3_init_stream(c); > @@ -383,6 +377,9 @@ ngx_http_wait_request_handler(ngx_event_ > ngx_buf_t *b; > ngx_connection_t *c; > ngx_http_connection_t *hc; > +#if (NGX_HTTP_V2) > + ngx_http_v2_srv_conf_t *h2scf; > +#endif > ngx_http_core_srv_conf_t *cscf; > > c = rev->data; > @@ -429,6 +426,8 @@ ngx_http_wait_request_handler(ngx_event_ > b->end = b->last + size; > } > > + size = b->end - b->last; > + > n = c->recv(c, b->last, size); > > if (n == NGX_AGAIN) { > @@ -443,12 +442,16 @@ ngx_http_wait_request_handler(ngx_event_ > return; > } > > - /* > - * We are trying to not hold c->buffer's memory for an idle connection. > - */ > - > - if (ngx_pfree(c->pool, b->start) == NGX_OK) { > - b->start = NULL; > + if (b->pos == b->last) { > + > + /* > + * We are trying to not hold c->buffer's memory for an > + * idle connection. > + */ > + > + if (ngx_pfree(c->pool, b->start) == NGX_OK) { > + b->start = NULL; > + } > } > > return; > @@ -489,6 +492,30 @@ ngx_http_wait_request_handler(ngx_event_ > } > } > > +#if (NGX_HTTP_V2) > + > + h2scf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_v2_module); > + > + if (!c->ssl && (h2scf->enable || hc->addr_conf->http2)) { > + > + size = ngx_min(sizeof(NGX_HTTP_V2_PREFACE) - 1, > + (size_t) (b->last - b->pos)); > + > + if (ngx_memcmp(b->pos, NGX_HTTP_V2_PREFACE, size) == 0) { > + > + if (size == sizeof(NGX_HTTP_V2_PREFACE) - 1) { > + ngx_http_v2_init(rev); > + return; > + } Note that ngx_http_wait_request_handler() is used for both new and keepalive connections. It does not look like a good idea to start an HTTP/2 connection after processing some HTTP/1.x requests. > + > + c->log->action = "waiting for request"; > + ngx_post_event(rev, &ngx_posted_events); > + return; > + } > + } > + > +#endif > + > c->log->action = "reading client request line"; > > ngx_reusable_connection(c, 0); > @@ -808,13 +835,16 @@ ngx_http_ssl_handshake_handler(ngx_conne > #if (NGX_HTTP_V2 \ > && defined TLSEXT_TYPE_application_layer_protocol_negotiation) > { > - unsigned int len; > - const unsigned char *data; > - ngx_http_connection_t *hc; > + unsigned int len; > + const unsigned char *data; > + ngx_http_connection_t *hc; > + ngx_http_v2_srv_conf_t *h2scf; > > hc = c->data; > > - if (hc->addr_conf->http2) { > + h2scf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_v2_module); > + > + if (h2scf->enable || hc->addr_conf->http2) { > > SSL_get0_alpn_selected(c->ssl->connection, &data, &len); > > diff --git a/src/http/v2/ngx_http_v2.c b/src/http/v2/ngx_http_v2.c > --- a/src/http/v2/ngx_http_v2.c > +++ b/src/http/v2/ngx_http_v2.c > @@ -232,6 +232,7 @@ static ngx_http_v2_parse_header_t ngx_h > void > ngx_http_v2_init(ngx_event_t *rev) > { > + u_char *p, *end; > ngx_connection_t *c; > ngx_pool_cleanup_t *cln; > ngx_http_connection_t *hc; > @@ -335,6 +336,29 @@ ngx_http_v2_init(ngx_event_t *rev) > c->idle = 1; > ngx_reusable_connection(c, 0); > > + if (c->buffer) { > + p = c->buffer->pos; > + end = c->buffer->last; > + > + do { > + p = h2c->state.handler(h2c, p, end); > + > + if (p == NULL) { > + return; > + } > + > + } while (p != end); > + > + h2c->total_bytes += p - c->buffer->pos; > + c->buffer->pos = p; > + > + if (h2c->total_bytes / 8 > h2c->payload_bytes + 1048576) { > + ngx_log_error(NGX_LOG_INFO, c->log, 0, "http2 flood detected"); > + ngx_http_v2_finalize_connection(h2c, NGX_HTTP_V2_NO_ERROR); > + return; > + } I think flood protection can be safely omitted here, given that client_header_buffer_size is expected to be small (1k by default), and flood protection is not going to kick in till at least 8M of total bytes from the client. > + } > + > ngx_http_v2_read_handler(rev); > } > > @@ -871,7 +895,7 @@ static u_char * > ngx_http_v2_state_preface(ngx_http_v2_connection_t *h2c, u_char *pos, > u_char *end) > { > - static const u_char preface[] = "PRI * HTTP/2.0\r\n"; > + static const u_char preface[] = NGX_HTTP_V2_PREFACE_START; > > if ((size_t) (end - pos) < sizeof(preface) - 1) { > return ngx_http_v2_state_save(h2c, pos, end, ngx_http_v2_state_preface); > @@ -892,7 +916,7 @@ static u_char * > ngx_http_v2_state_preface_end(ngx_http_v2_connection_t *h2c, u_char *pos, > u_char *end) > { > - static const u_char preface[] = "\r\nSM\r\n\r\n"; > + static const u_char preface[] = NGX_HTTP_V2_PREFACE_END; > > if ((size_t) (end - pos) < sizeof(preface) - 1) { > return ngx_http_v2_state_save(h2c, pos, end, > @@ -3946,10 +3970,22 @@ static void > ngx_http_v2_run_request(ngx_http_request_t *r) > { > ngx_connection_t *fc; > + ngx_http_v3_srv_conf_t *h2scf; > ngx_http_v2_connection_t *h2c; > > fc = r->connection; > > + h2scf = ngx_http_get_module_srv_conf(r, ngx_http_v2_module); > + > + if (!h2scf->enable && !r->http_connection->addr_conf->http2) { > + ngx_log_error(NGX_LOG_INFO, fc->log, 0, > + "client attempted to request the server name " > + "for which the negotitated protocol is unavailable"); Typo, should be "negotiated". > + > + ngx_http_finalize_request(r, NGX_HTTP_MISDIRECTED_REQUEST); > + goto failed; > + } > + > if (ngx_http_v2_construct_request_line(r) != NGX_OK) { > goto failed; > } > diff --git a/src/http/v2/ngx_http_v2.h b/src/http/v2/ngx_http_v2.h > --- a/src/http/v2/ngx_http_v2.h > +++ b/src/http/v2/ngx_http_v2.h > @@ -64,6 +64,16 @@ typedef u_char *(*ngx_http_v2_handler_pt > > > typedef struct { > + ngx_flag_t enable; > + size_t pool_size; > + ngx_uint_t concurrent_streams; > + ngx_uint_t concurrent_pushes; > + size_t preread_size; > + ngx_uint_t streams_index_mask; > +} ngx_http_v2_srv_conf_t; > + > + > +typedef struct { > ngx_str_t name; > ngx_str_t value; > } ngx_http_v2_header_t; > @@ -408,9 +418,17 @@ ngx_int_t ngx_http_v2_table_size(ngx_htt > #define NGX_HTTP_V2_USER_AGENT_INDEX 58 > #define NGX_HTTP_V2_VARY_INDEX 59 > > +#define NGX_HTTP_V2_PREFACE_START "PRI * HTTP/2.0\r\n" > +#define NGX_HTTP_V2_PREFACE_END "\r\nSM\r\n\r\n" > +#define NGX_HTTP_V2_PREFACE NGX_HTTP_V2_PREFACE_START \ > + NGX_HTTP_V2_PREFACE_END > + > > u_char *ngx_http_v2_string_encode(u_char *dst, u_char *src, size_t len, > u_char *tmp, ngx_uint_t lower); > > > +extern ngx_module_t ngx_http_v2_module; > + > + > #endif /* _NGX_HTTP_V2_H_INCLUDED_ */ > diff --git a/src/http/v2/ngx_http_v2_module.c b/src/http/v2/ngx_http_v2_module.c > --- a/src/http/v2/ngx_http_v2_module.c > +++ b/src/http/v2/ngx_http_v2_module.c > @@ -75,6 +75,13 @@ static ngx_conf_post_t ngx_http_v2_chun > > static ngx_command_t ngx_http_v2_commands[] = { > > + { ngx_string("http2"), > + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_CONF_FLAG, > + ngx_conf_set_flag_slot, > + NGX_HTTP_SRV_CONF_OFFSET, > + offsetof(ngx_http_v2_srv_conf_t, enable), > + NULL }, > + > { ngx_string("http2_recv_buffer_size"), > NGX_HTTP_MAIN_CONF|NGX_CONF_TAKE1, > ngx_conf_set_size_slot, > @@ -314,6 +321,8 @@ ngx_http_v2_create_srv_conf(ngx_conf_t * > return NULL; > } > > + h2scf->enable = NGX_CONF_UNSET; > + > h2scf->pool_size = NGX_CONF_UNSET_SIZE; > > h2scf->concurrent_streams = NGX_CONF_UNSET_UINT; > @@ -333,6 +342,8 @@ ngx_http_v2_merge_srv_conf(ngx_conf_t *c > ngx_http_v2_srv_conf_t *prev = parent; > ngx_http_v2_srv_conf_t *conf = child; > > + ngx_conf_merge_value(conf->enable, prev->enable, 0); > + > ngx_conf_merge_size_value(conf->pool_size, prev->pool_size, 4096); > > ngx_conf_merge_uint_value(conf->concurrent_streams, > diff --git a/src/http/v2/ngx_http_v2_module.h b/src/http/v2/ngx_http_v2_module.h > --- a/src/http/v2/ngx_http_v2_module.h > +++ b/src/http/v2/ngx_http_v2_module.h > @@ -21,15 +21,6 @@ typedef struct { > > > typedef struct { > - size_t pool_size; > - ngx_uint_t concurrent_streams; > - ngx_uint_t concurrent_pushes; > - size_t preread_size; > - ngx_uint_t streams_index_mask; > -} ngx_http_v2_srv_conf_t; > - > - > -typedef struct { > size_t chunk_size; > > ngx_flag_t push_preload; > @@ -39,7 +30,4 @@ typedef struct { > } ngx_http_v2_loc_conf_t; > > > -extern ngx_module_t ngx_http_v2_module; > - > - > #endif /* _NGX_HTTP_V2_MODULE_H_INCLUDED_ */ Otherwise looks good. -- Maxim Dounin http://mdounin.ru/ From xeioex at nginx.com Tue Feb 7 03:43:37 2023 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Tue, 07 Feb 2023 03:43:37 +0000 Subject: [njs] Fetch: fixed compatibility with nginx-1.22 and below. Message-ID: details: https://hg.nginx.org/njs/rev/51f737b21b48 branches: changeset: 2039:51f737b21b48 user: Dmitry Volyntsev date: Fri Feb 03 22:17:53 2023 -0800 description: Fetch: fixed compatibility with nginx-1.22 and below. Previously, ngx_table_elt_t structure was used directly in njs fetch code. This prevents from building with nginx-1.22 as the structure was changed after nginx-1.22. The fix is to use njs defined structure instead of ngx_table_elt_t. This fixes #610 issue on Github. diffstat: nginx/ngx_js_fetch.c | 34 ++++++++++++++++++++++------------ 1 files changed, 22 insertions(+), 12 deletions(-) diffs (128 lines): diff -r 69547623da14 -r 51f737b21b48 nginx/ngx_js_fetch.c --- a/nginx/ngx_js_fetch.c Fri Feb 03 06:41:01 2023 -0800 +++ b/nginx/ngx_js_fetch.c Fri Feb 03 22:17:53 2023 -0800 @@ -47,6 +47,16 @@ typedef struct { } ngx_js_http_chunk_parse_t; +typedef struct ngx_js_tb_elt_s ngx_js_tb_elt_t; + +struct ngx_js_tb_elt_s { + ngx_uint_t hash; + ngx_str_t key; + ngx_str_t value; + ngx_js_tb_elt_t *next; +}; + + typedef struct { enum { GUARD_NONE = 0, @@ -651,7 +661,7 @@ ngx_js_ext_fetch(njs_vm_t *vm, njs_value njs_value_t *init, *value; ngx_js_http_t *http; ngx_list_part_t *part; - ngx_table_elt_t *h; + ngx_js_tb_elt_t *h; ngx_js_request_t request; ngx_connection_t *c; ngx_resolver_ctx_t *ctx; @@ -850,7 +860,7 @@ ngx_js_ext_headers_constructor(njs_vm_t return NJS_ERROR; } - rc = ngx_list_init(&headers->header_list, pool, 4, sizeof(ngx_table_elt_t)); + rc = ngx_list_init(&headers->header_list, pool, 4, sizeof(ngx_js_tb_elt_t)); if (rc != NGX_OK) { njs_vm_memory_error(vm); return NJS_ERROR; @@ -933,7 +943,7 @@ ngx_js_ext_response_constructor(njs_vm_t pool = ngx_external_pool(vm, njs_vm_external_ptr(vm)); rc = ngx_list_init(&response->headers.header_list, pool, 4, - sizeof(ngx_table_elt_t)); + sizeof(ngx_js_tb_elt_t)); if (rc != NGX_OK) { njs_vm_memory_error(vm); return NJS_ERROR; @@ -1076,7 +1086,7 @@ ngx_js_headers_inherit(njs_vm_t *vm, ngx njs_int_t ret; ngx_uint_t i; ngx_list_part_t *part; - ngx_table_elt_t *h; + ngx_js_tb_elt_t *h; part = &orig->header_list.part; h = part->elts; @@ -1900,7 +1910,7 @@ ngx_js_request_constructor(njs_vm_t *vm, pool = ngx_external_pool(vm, external); rc = ngx_list_init(&request->headers.header_list, pool, 4, - sizeof(ngx_table_elt_t)); + sizeof(ngx_js_tb_elt_t)); if (rc != NGX_OK) { njs_vm_memory_error(vm); return NJS_ERROR; @@ -2031,7 +2041,7 @@ ngx_js_request_constructor(njs_vm_t *vm, */ rc = ngx_list_init(&request->headers.header_list, pool, 4, - sizeof(ngx_table_elt_t)); + sizeof(ngx_js_tb_elt_t)); if (rc != NGX_OK) { njs_vm_memory_error(vm); return NJS_ERROR; @@ -2161,7 +2171,7 @@ ngx_js_headers_append(njs_vm_t *vm, ngx_ { u_char *p, *end; ngx_uint_t i; - ngx_table_elt_t *h, **ph; + ngx_js_tb_elt_t *h, **ph; ngx_list_part_t *part; const njs_str_t *f; @@ -2360,7 +2370,7 @@ ngx_js_http_process_headers(ngx_js_http_ if (http->response.headers.header_list.size == 0) { rc = ngx_list_init(&http->response.headers.header_list, http->pool, 4, - sizeof(ngx_table_elt_t)); + sizeof(ngx_js_tb_elt_t)); if (rc != NGX_OK) { ngx_js_http_error(http, 0, "alloc failed"); return NGX_ERROR; @@ -3149,7 +3159,7 @@ ngx_headers_js_get(njs_vm_t *vm, njs_val size_t len; njs_int_t rc; ngx_uint_t i; - ngx_table_elt_t *h, *ph; + ngx_js_tb_elt_t *h, *ph; ngx_list_part_t *part; ngx_js_headers_t *headers; @@ -3296,7 +3306,7 @@ ngx_headers_js_ext_delete(njs_vm_t *vm, njs_str_t name; ngx_uint_t i; ngx_list_part_t *part; - ngx_table_elt_t *h; + ngx_js_tb_elt_t *h; ngx_js_headers_t *headers; headers = njs_vm_external(vm, ngx_http_js_fetch_headers_proto_id, @@ -3478,7 +3488,7 @@ ngx_headers_js_ext_keys(njs_vm_t *vm, nj ngx_uint_t i, k, length; njs_value_t *start; ngx_list_part_t *part; - ngx_table_elt_t *h; + ngx_js_tb_elt_t *h; ngx_js_headers_t *headers; headers = njs_vm_external(vm, ngx_http_js_fetch_headers_proto_id, value); @@ -3558,7 +3568,7 @@ ngx_headers_js_ext_set(njs_vm_t *vm, njs njs_str_t name, value; ngx_uint_t i; ngx_list_part_t *part; - ngx_table_elt_t *h, **ph, **pp; + ngx_js_tb_elt_t *h, **ph, **pp; ngx_js_headers_t *headers; headers = njs_vm_external(vm, ngx_http_js_fetch_headers_proto_id, From xeioex at nginx.com Tue Feb 7 03:43:39 2023 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Tue, 07 Feb 2023 03:43:39 +0000 Subject: [njs] Version 0.7.10. Message-ID: details: https://hg.nginx.org/njs/rev/3feba3459e7b branches: changeset: 2040:3feba3459e7b user: Dmitry Volyntsev date: Mon Feb 06 09:57:17 2023 -0800 description: Version 0.7.10. diffstat: CHANGES | 29 +++++++++++++++++++++++++++++ 1 files changed, 29 insertions(+), 0 deletions(-) diffs (36 lines): diff -r 51f737b21b48 -r 3feba3459e7b CHANGES --- a/CHANGES Fri Feb 03 22:17:53 2023 -0800 +++ b/CHANGES Mon Feb 06 09:57:17 2023 -0800 @@ -1,3 +1,32 @@ +Changes with njs 0.7.10 7 Feb 2023 + nginx modules: + + *) Feature: added Request, Response and Headers ctors in Fetch API. + + *) Bugfix: fixed nginx logger callback for calls in master process. + + Core: + + *) Feature: added signal support in CLI. + + *) Feature: added "xml" module for working with XML documents. + + *) Feature: extended support for symmetric and asymmetric keys + in WebCrypto. Most notably JWK format for importKey() was added. + + *) Feature: extended support for symmetric and asymmetric keys + in WebCrypto. Most notably JWK format for importKey() was added. + generateKey() and exportKey() were also implemented. + + *) Feature: added String.prototype.replaceAll(). + + *) Bugfix: fixed for(expr1; conditional syntax error handling. + + *) Bugfix: fixed Object.values() and Object.entries() with external + objects. + + *) Bugfix: fixed RegExp.prototype[@@replace](). + Changes with njs 0.7.9 17 Nov 2022 nginx modules: From xeioex at nginx.com Tue Feb 7 03:43:41 2023 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Tue, 07 Feb 2023 03:43:41 +0000 Subject: [njs] Added tag 0.7.10 for changeset 60a3ffe734b6 Message-ID: details: https://hg.nginx.org/njs/rev/d20fb5d9d28f branches: changeset: 2041:d20fb5d9d28f user: Dmitry Volyntsev date: Mon Feb 06 10:50:08 2023 -0800 description: Added tag 0.7.10 for changeset 60a3ffe734b6 diffstat: .hgtags | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diffs (8 lines): diff -r 3feba3459e7b -r d20fb5d9d28f .hgtags --- a/.hgtags Mon Feb 06 09:57:17 2023 -0800 +++ b/.hgtags Mon Feb 06 10:50:08 2023 -0800 @@ -55,3 +55,4 @@ 461dfb0bb60e531d361319f30993f29860c19f55 1592d46d9076aa832b2d37d50b90f5edfca67030 0.7.7 3308415d7de83c3c0c7c65405bec4836685a71de 0.7.8 5f705230a62ccae8718ef9b7a85b29dc569d8b5e 0.7.9 +60a3ffe734b67add80bbc58448c3f6c9036fe844 0.7.10 From pluknet at nginx.com Tue Feb 7 10:25:20 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Tue, 7 Feb 2023 14:25:20 +0400 Subject: [PATCH 2 of 3] HTTP/2: "http2" directive In-Reply-To: <139b348815b3f1975317.1675260069@arut-laptop> References: <139b348815b3f1975317.1675260069@arut-laptop> Message-ID: <021D2859-2A47-421F-A41B-C0AD6E821080@nginx.com> > On 1 Feb 2023, at 18:01, Roman Arutyunyan wrote: > > # HG changeset patch > # User Roman Arutyunyan > # Date 1675255790 -14400 > # Wed Feb 01 16:49:50 2023 +0400 > # Branch quic > # Node ID 139b348815b3f19753176988f06b590a3e0af4c0 > # Parent 2fcc1c60be1c89aad5464bcc06f1189d1adc885a > HTTP/2: "http2" directive. > > The directive enables HTTP/2 in the current server. The previous way to > enable HTTP/2 via "listen ... http2" is now deprecated. > > diff --git a/src/http/modules/ngx_http_ssl_module.c b/src/http/modules/ngx_http_ssl_module.c > --- a/src/http/modules/ngx_http_ssl_module.c > +++ b/src/http/modules/ngx_http_ssl_module.c > @@ -427,6 +427,9 @@ ngx_http_ssl_alpn_select(ngx_ssl_conn_t > #if (NGX_HTTP_V2 || NGX_HTTP_V3) > ngx_http_connection_t *hc; > #endif > +#if (NGX_HTTP_V2) > + ngx_http_v2_srv_conf_t *h2scf; > +#endif > #if (NGX_HTTP_V3) > ngx_http_v3_srv_conf_t *h3scf; > #endif > @@ -448,12 +451,9 @@ ngx_http_ssl_alpn_select(ngx_ssl_conn_t > hc = c->data; > #endif > > -#if (NGX_HTTP_V2) > - if (hc->addr_conf->http2) { > - srv = (unsigned char *) NGX_HTTP_V2_ALPN_PROTO NGX_HTTP_ALPN_PROTOS; > - srvlen = sizeof(NGX_HTTP_V2_ALPN_PROTO NGX_HTTP_ALPN_PROTOS) - 1; > - } else > -#endif > + srv = (unsigned char *) NGX_HTTP_ALPN_PROTOS; > + srvlen = sizeof(NGX_HTTP_ALPN_PROTOS) - 1; > + > #if (NGX_HTTP_V3) > if (hc->addr_conf->quic) { > > @@ -479,10 +479,16 @@ ngx_http_ssl_alpn_select(ngx_ssl_conn_t > > } else > #endif > +#if (NGX_HTTP_V2) > { > - srv = (unsigned char *) NGX_HTTP_ALPN_PROTOS; > - srvlen = sizeof(NGX_HTTP_ALPN_PROTOS) - 1; > + h2scf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_v2_module); > + > + if (h2scf->enable || hc->addr_conf->http2) { > + srv = (unsigned char *) NGX_HTTP_V2_ALPN_PROTO NGX_HTTP_ALPN_PROTOS; > + srvlen = sizeof(NGX_HTTP_V2_ALPN_PROTO NGX_HTTP_ALPN_PROTOS) - 1; > + } > } > +#endif > > if (SSL_select_next_proto((unsigned char **) out, outlen, srv, srvlen, > in, inlen) > diff --git a/src/http/ngx_http_core_module.c b/src/http/ngx_http_core_module.c > --- a/src/http/ngx_http_core_module.c > +++ b/src/http/ngx_http_core_module.c > @@ -4179,6 +4179,11 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx > > if (ngx_strcmp(value[n].data, "http2") == 0) { > #if (NGX_HTTP_V2) > + ngx_conf_log_error(NGX_LOG_WARN, cf, 0, > + "the \"listen ... http2\" directive " > + "is deprecated, use " > + "the \"http2\" directive instead"); > + > lsopt.http2 = 1; > continue; > #else > diff --git a/src/http/ngx_http_request.c b/src/http/ngx_http_request.c > --- a/src/http/ngx_http_request.c > +++ b/src/http/ngx_http_request.c > @@ -318,12 +318,6 @@ ngx_http_init_connection(ngx_connection_ > rev->handler = ngx_http_wait_request_handler; > c->write->handler = ngx_http_empty_handler; > > -#if (NGX_HTTP_V2) > - if (hc->addr_conf->http2) { > - rev->handler = ngx_http_v2_init; > - } > -#endif > - > #if (NGX_HTTP_V3) > if (hc->addr_conf->quic) { > ngx_http_v3_init_stream(c); > @@ -383,6 +377,9 @@ ngx_http_wait_request_handler(ngx_event_ > ngx_buf_t *b; > ngx_connection_t *c; > ngx_http_connection_t *hc; > +#if (NGX_HTTP_V2) > + ngx_http_v2_srv_conf_t *h2scf; > +#endif > ngx_http_core_srv_conf_t *cscf; > > c = rev->data; > @@ -429,6 +426,8 @@ ngx_http_wait_request_handler(ngx_event_ > b->end = b->last + size; > } > > + size = b->end - b->last; > + > n = c->recv(c, b->last, size); > > if (n == NGX_AGAIN) { > @@ -443,12 +442,16 @@ ngx_http_wait_request_handler(ngx_event_ > return; > } > > - /* > - * We are trying to not hold c->buffer's memory for an idle connection. > - */ > - > - if (ngx_pfree(c->pool, b->start) == NGX_OK) { > - b->start = NULL; > + if (b->pos == b->last) { > + > + /* > + * We are trying to not hold c->buffer's memory for an > + * idle connection. > + */ > + > + if (ngx_pfree(c->pool, b->start) == NGX_OK) { > + b->start = NULL; > + } > } > > return; > @@ -489,6 +492,30 @@ ngx_http_wait_request_handler(ngx_event_ > } > } > > +#if (NGX_HTTP_V2) > + > + h2scf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_v2_module); > + > + if (!c->ssl && (h2scf->enable || hc->addr_conf->http2)) { > + > + size = ngx_min(sizeof(NGX_HTTP_V2_PREFACE) - 1, > + (size_t) (b->last - b->pos)); > + > + if (ngx_memcmp(b->pos, NGX_HTTP_V2_PREFACE, size) == 0) { > + > + if (size == sizeof(NGX_HTTP_V2_PREFACE) - 1) { > + ngx_http_v2_init(rev); > + return; > + } > + > + c->log->action = "waiting for request"; > + ngx_post_event(rev, &ngx_posted_events); > + return; > + } > + } > + > +#endif > + There are two subtile changes: - if we receive bad invalid connection preface, now it is handled in the generic ngx_http_wait_request_handler(), meaning HTTP/1.1 400 will be sent on error. This is acceptable by RFC 9113, though: : A GOAWAY frame (Section 6.8) MAY be omitted in this case, since : an invalid preface indicates that the peer is not using HTTP/2. But it probably deserves to write a not in the commit message - the same is about bad PROXY protocol string Looks like we can optimize parsing PROXY after ngx_http_v2_init(), since PROXY is now always parsed in ngx_http_wait_request_handler(). > c->log->action = "reading client request line"; > > ngx_reusable_connection(c, 0); > @@ -808,13 +835,16 @@ ngx_http_ssl_handshake_handler(ngx_conne > #if (NGX_HTTP_V2 \ > && defined TLSEXT_TYPE_application_layer_protocol_negotiation) > { > - unsigned int len; > - const unsigned char *data; > - ngx_http_connection_t *hc; > + unsigned int len; > + const unsigned char *data; > + ngx_http_connection_t *hc; > + ngx_http_v2_srv_conf_t *h2scf; > > hc = c->data; > > - if (hc->addr_conf->http2) { > + h2scf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_v2_module); > + > + if (h2scf->enable || hc->addr_conf->http2) { > > SSL_get0_alpn_selected(c->ssl->connection, &data, &len); > > diff --git a/src/http/v2/ngx_http_v2.c b/src/http/v2/ngx_http_v2.c > --- a/src/http/v2/ngx_http_v2.c > +++ b/src/http/v2/ngx_http_v2.c > @@ -232,6 +232,7 @@ static ngx_http_v2_parse_header_t ngx_h > void > ngx_http_v2_init(ngx_event_t *rev) > { > + u_char *p, *end; > ngx_connection_t *c; > ngx_pool_cleanup_t *cln; > ngx_http_connection_t *hc; > @@ -335,6 +336,29 @@ ngx_http_v2_init(ngx_event_t *rev) > c->idle = 1; > ngx_reusable_connection(c, 0); > > + if (c->buffer) { > + p = c->buffer->pos; > + end = c->buffer->last; > + > + do { > + p = h2c->state.handler(h2c, p, end); > + > + if (p == NULL) { > + return; > + } > + > + } while (p != end); > + > + h2c->total_bytes += p - c->buffer->pos; > + c->buffer->pos = p; > + > + if (h2c->total_bytes / 8 > h2c->payload_bytes + 1048576) { > + ngx_log_error(NGX_LOG_INFO, c->log, 0, "http2 flood detected"); > + ngx_http_v2_finalize_connection(h2c, NGX_HTTP_V2_NO_ERROR); > + return; > + } > + } > + > ngx_http_v2_read_handler(rev); > } > > @@ -871,7 +895,7 @@ static u_char * > ngx_http_v2_state_preface(ngx_http_v2_connection_t *h2c, u_char *pos, > u_char *end) > { > - static const u_char preface[] = "PRI * HTTP/2.0\r\n"; > + static const u_char preface[] = NGX_HTTP_V2_PREFACE_START; > > if ((size_t) (end - pos) < sizeof(preface) - 1) { > return ngx_http_v2_state_save(h2c, pos, end, ngx_http_v2_state_preface); > @@ -892,7 +916,7 @@ static u_char * > ngx_http_v2_state_preface_end(ngx_http_v2_connection_t *h2c, u_char *pos, > u_char *end) > { > - static const u_char preface[] = "\r\nSM\r\n\r\n"; > + static const u_char preface[] = NGX_HTTP_V2_PREFACE_END; > > if ((size_t) (end - pos) < sizeof(preface) - 1) { > return ngx_http_v2_state_save(h2c, pos, end, > @@ -3946,10 +3970,22 @@ static void > ngx_http_v2_run_request(ngx_http_request_t *r) > { > ngx_connection_t *fc; > + ngx_http_v3_srv_conf_t *h2scf; ngx_http_v2_srv_conf_t > ngx_http_v2_connection_t *h2c; > > fc = r->connection; > > + h2scf = ngx_http_get_module_srv_conf(r, ngx_http_v2_module); > + > + if (!h2scf->enable && !r->http_connection->addr_conf->http2) { > + ngx_log_error(NGX_LOG_INFO, fc->log, 0, > + "client attempted to request the server name " > + "for which the negotitated protocol is unavailable"); - typo: negotiated (same as in 1st patch) > + > + ngx_http_finalize_request(r, NGX_HTTP_MISDIRECTED_REQUEST); > + goto failed; > + } > + > if (ngx_http_v2_construct_request_line(r) != NGX_OK) { > goto failed; > } > diff --git a/src/http/v2/ngx_http_v2.h b/src/http/v2/ngx_http_v2.h > --- a/src/http/v2/ngx_http_v2.h > +++ b/src/http/v2/ngx_http_v2.h > @@ -64,6 +64,16 @@ typedef u_char *(*ngx_http_v2_handler_pt > > > typedef struct { > + ngx_flag_t enable; > + size_t pool_size; > + ngx_uint_t concurrent_streams; > + ngx_uint_t concurrent_pushes; > + size_t preread_size; > + ngx_uint_t streams_index_mask; > +} ngx_http_v2_srv_conf_t; > + > + > +typedef struct { > ngx_str_t name; > ngx_str_t value; > } ngx_http_v2_header_t; > @@ -408,9 +418,17 @@ ngx_int_t ngx_http_v2_table_size(ngx_htt > #define NGX_HTTP_V2_USER_AGENT_INDEX 58 > #define NGX_HTTP_V2_VARY_INDEX 59 > > +#define NGX_HTTP_V2_PREFACE_START "PRI * HTTP/2.0\r\n" > +#define NGX_HTTP_V2_PREFACE_END "\r\nSM\r\n\r\n" > +#define NGX_HTTP_V2_PREFACE NGX_HTTP_V2_PREFACE_START \ > + NGX_HTTP_V2_PREFACE_END > + > > u_char *ngx_http_v2_string_encode(u_char *dst, u_char *src, size_t len, > u_char *tmp, ngx_uint_t lower); > > > +extern ngx_module_t ngx_http_v2_module; > + > + > #endif /* _NGX_HTTP_V2_H_INCLUDED_ */ > diff --git a/src/http/v2/ngx_http_v2_module.c b/src/http/v2/ngx_http_v2_module.c > --- a/src/http/v2/ngx_http_v2_module.c > +++ b/src/http/v2/ngx_http_v2_module.c > @@ -75,6 +75,13 @@ static ngx_conf_post_t ngx_http_v2_chun > > static ngx_command_t ngx_http_v2_commands[] = { > > + { ngx_string("http2"), > + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_CONF_FLAG, > + ngx_conf_set_flag_slot, > + NGX_HTTP_SRV_CONF_OFFSET, > + offsetof(ngx_http_v2_srv_conf_t, enable), > + NULL }, > + > { ngx_string("http2_recv_buffer_size"), > NGX_HTTP_MAIN_CONF|NGX_CONF_TAKE1, > ngx_conf_set_size_slot, > @@ -314,6 +321,8 @@ ngx_http_v2_create_srv_conf(ngx_conf_t * > return NULL; > } > > + h2scf->enable = NGX_CONF_UNSET; > + > h2scf->pool_size = NGX_CONF_UNSET_SIZE; > > h2scf->concurrent_streams = NGX_CONF_UNSET_UINT; > @@ -333,6 +342,8 @@ ngx_http_v2_merge_srv_conf(ngx_conf_t *c > ngx_http_v2_srv_conf_t *prev = parent; > ngx_http_v2_srv_conf_t *conf = child; > > + ngx_conf_merge_value(conf->enable, prev->enable, 0); > + > ngx_conf_merge_size_value(conf->pool_size, prev->pool_size, 4096); > > ngx_conf_merge_uint_value(conf->concurrent_streams, > diff --git a/src/http/v2/ngx_http_v2_module.h b/src/http/v2/ngx_http_v2_module.h > --- a/src/http/v2/ngx_http_v2_module.h > +++ b/src/http/v2/ngx_http_v2_module.h > @@ -21,15 +21,6 @@ typedef struct { > > > typedef struct { > - size_t pool_size; > - ngx_uint_t concurrent_streams; > - ngx_uint_t concurrent_pushes; > - size_t preread_size; > - ngx_uint_t streams_index_mask; > -} ngx_http_v2_srv_conf_t; > - > - > -typedef struct { > size_t chunk_size; > > ngx_flag_t push_preload; > @@ -39,7 +30,4 @@ typedef struct { > } ngx_http_v2_loc_conf_t; > > > -extern ngx_module_t ngx_http_v2_module; > - > - > #endif /* _NGX_HTTP_V2_MODULE_H_INCLUDED_ */ (BTW, we could make a separate ngx_http_v3_module.h as well, to keep there a private stuff.) -- Sergey Kandaurov From arut at nginx.com Tue Feb 7 13:39:55 2023 From: arut at nginx.com (Roman Arutyunyan) Date: Tue, 7 Feb 2023 17:39:55 +0400 Subject: [PATCH 1 of 3] HTTP/3: "quic" parameter of "listen" directive In-Reply-To: References: <2fcc1c60be1c89aad546.1675260068@arut-laptop> Message-ID: <20230207133955.2p7uziylqk5ssxrs@N00W24XTQX> Hi, On Tue, Feb 07, 2023 at 06:12:01AM +0300, Maxim Dounin wrote: > Hello! > > On Wed, Feb 01, 2023 at 06:01:08PM +0400, Roman Arutyunyan wrote: > > > # HG changeset patch > > # User Roman Arutyunyan > > # Date 1675254688 -14400 > > # Wed Feb 01 16:31:28 2023 +0400 > > # Branch quic > > # Node ID 2fcc1c60be1c89aad5464bcc06f1189d1adc885a > > # Parent def8e398d7c50131f8dac844814fff729da5c86c > > HTTP/3: "quic" parameter of "listen" directive. > > > > Now "listen" directve has a new "quic" parameter which enables QUIC protocol > > for the address. Further, to enable HTTP/3, a new directive "http3" is > > introduced. The hq-interop protocol is enabled by "http3_hq" as before. > > Now application protocol is chosen by ALPN. > > > > Previously used "http3" parameter of "listen" is deprecated. > > > > diff --git a/README b/README > > --- a/README > > +++ b/README > > @@ -93,13 +93,13 @@ 2. Installing > > > > 3. Configuration > > > > - The HTTP "listen" directive got a new option "http3" which enables > > - HTTP/3 over QUIC on the specified port. > > + The HTTP "listen" directive got a new option "quic" which enables > > + QUIC as client transport protocol instead of TCP. > > > > The Stream "listen" directive got a new option "quic" which enables > > QUIC as client transport protocol instead of TCP or plain UDP. > > > > - Along with "http3" or "quic", it's also possible to specify "reuseport" > > + Along with "quic", it's also possible to specify "reuseport" > > option [8] to make it work properly with multiple workers. > > > > To enable address validation: > > @@ -133,12 +133,13 @@ 3. Configuration > > > > A number of directives were added that configure HTTP/3: > > > > + http3 > > + http3_hq > > http3_stream_buffer_size > > http3_max_concurrent_pushes > > http3_max_concurrent_streams > > http3_push > > http3_push_preload > > - http3_hq (requires NGX_HTTP_V3_HQ macro) > > > > In http, an additional variable is available: $http3. > > The value of $http3 is "h3" for HTTP/3 connections, > > @@ -160,13 +161,15 @@ Example configuration: > > server { > > # for better compatibility it's recommended > > # to use the same port for quic and https > > - listen 8443 http3 reuseport; > > + listen 8443 quic reuseport; > > listen 8443 ssl; > > > > ssl_certificate certs/example.com.crt; > > ssl_certificate_key certs/example.com.key; > > ssl_protocols TLSv1.3; > > > > + http3 on; > > + > > Shouldn't "http3" be "on" by default, so it will be used for all > QUIC listening sockets automatically (unless disabled), basically > matching the "listen ... http3" behaviour? > > Not sure though, "off" by default seems to better match generic > approach and prevents if from being enabled accidentally. On the > other hand, "listen ... quic" has little to no uses without HTTP/3 > being enabled, and requirement to explicitly enable it looks > somewhat silly. OK, enabled http3 by default. Indeed, enabling http3 explicitly makes little sense for most people once "listen ... quic" is specified. > [...] > > > diff --git a/src/http/ngx_http_core_module.c b/src/http/ngx_http_core_module.c > > --- a/src/http/ngx_http_core_module.c > > +++ b/src/http/ngx_http_core_module.c > > @@ -4191,6 +4191,11 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx > > > > if (ngx_strcmp(value[n].data, "http3") == 0) { > > #if (NGX_HTTP_V3) > > + ngx_conf_log_error(NGX_LOG_WARN, cf, 0, > > + "the \"listen ... http3\" directive " > > + "is deprecated, use " > > + "the \"http3\" directive instead"); > > Note that this doesn't provide enough information to rewrite the > configuration. Something like "use the \"listen ... quic\" and > \"http3\" directives instead" should be better (assuming "http3" > is off by default). Assuming http3 is on by default, this will slightly change then. > > + lsopt.quic = 1; > > lsopt.http3 = 1; > > lsopt.type = SOCK_DGRAM; > > continue; > > [...] > > -- > Maxim Dounin > http://mdounin.ru/ > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > https://mailman.nginx.org/mailman/listinfo/nginx-devel -- Roman Arutyunyan From arut at nginx.com Tue Feb 7 13:45:10 2023 From: arut at nginx.com (Roman Arutyunyan) Date: Tue, 7 Feb 2023 17:45:10 +0400 Subject: [PATCH 2 of 3] HTTP/2: "http2" directive In-Reply-To: <021D2859-2A47-421F-A41B-C0AD6E821080@nginx.com> References: <139b348815b3f1975317.1675260069@arut-laptop> <021D2859-2A47-421F-A41B-C0AD6E821080@nginx.com> Message-ID: <20230207134510.yb6jrbydrkxl4ffr@N00W24XTQX> Hi, On Tue, Feb 07, 2023 at 02:25:20PM +0400, Sergey Kandaurov wrote: > > > On 1 Feb 2023, at 18:01, Roman Arutyunyan wrote: > > > > # HG changeset patch > > # User Roman Arutyunyan > > # Date 1675255790 -14400 > > # Wed Feb 01 16:49:50 2023 +0400 > > # Branch quic > > # Node ID 139b348815b3f19753176988f06b590a3e0af4c0 > > # Parent 2fcc1c60be1c89aad5464bcc06f1189d1adc885a > > HTTP/2: "http2" directive. > > > > The directive enables HTTP/2 in the current server. The previous way to > > enable HTTP/2 via "listen ... http2" is now deprecated. > > > > diff --git a/src/http/modules/ngx_http_ssl_module.c b/src/http/modules/ngx_http_ssl_module.c > > --- a/src/http/modules/ngx_http_ssl_module.c > > +++ b/src/http/modules/ngx_http_ssl_module.c > > @@ -427,6 +427,9 @@ ngx_http_ssl_alpn_select(ngx_ssl_conn_t > > #if (NGX_HTTP_V2 || NGX_HTTP_V3) > > ngx_http_connection_t *hc; > > #endif > > +#if (NGX_HTTP_V2) > > + ngx_http_v2_srv_conf_t *h2scf; > > +#endif > > #if (NGX_HTTP_V3) > > ngx_http_v3_srv_conf_t *h3scf; > > #endif > > @@ -448,12 +451,9 @@ ngx_http_ssl_alpn_select(ngx_ssl_conn_t > > hc = c->data; > > #endif > > > > -#if (NGX_HTTP_V2) > > - if (hc->addr_conf->http2) { > > - srv = (unsigned char *) NGX_HTTP_V2_ALPN_PROTO NGX_HTTP_ALPN_PROTOS; > > - srvlen = sizeof(NGX_HTTP_V2_ALPN_PROTO NGX_HTTP_ALPN_PROTOS) - 1; > > - } else > > -#endif > > + srv = (unsigned char *) NGX_HTTP_ALPN_PROTOS; > > + srvlen = sizeof(NGX_HTTP_ALPN_PROTOS) - 1; > > + > > #if (NGX_HTTP_V3) > > if (hc->addr_conf->quic) { > > > > @@ -479,10 +479,16 @@ ngx_http_ssl_alpn_select(ngx_ssl_conn_t > > > > } else > > #endif > > +#if (NGX_HTTP_V2) > > { > > - srv = (unsigned char *) NGX_HTTP_ALPN_PROTOS; > > - srvlen = sizeof(NGX_HTTP_ALPN_PROTOS) - 1; > > + h2scf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_v2_module); > > + > > + if (h2scf->enable || hc->addr_conf->http2) { > > + srv = (unsigned char *) NGX_HTTP_V2_ALPN_PROTO NGX_HTTP_ALPN_PROTOS; > > + srvlen = sizeof(NGX_HTTP_V2_ALPN_PROTO NGX_HTTP_ALPN_PROTOS) - 1; > > + } > > } > > +#endif > > > > if (SSL_select_next_proto((unsigned char **) out, outlen, srv, srvlen, > > in, inlen) > > diff --git a/src/http/ngx_http_core_module.c b/src/http/ngx_http_core_module.c > > --- a/src/http/ngx_http_core_module.c > > +++ b/src/http/ngx_http_core_module.c > > @@ -4179,6 +4179,11 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx > > > > if (ngx_strcmp(value[n].data, "http2") == 0) { > > #if (NGX_HTTP_V2) > > + ngx_conf_log_error(NGX_LOG_WARN, cf, 0, > > + "the \"listen ... http2\" directive " > > + "is deprecated, use " > > + "the \"http2\" directive instead"); > > + > > lsopt.http2 = 1; > > continue; > > #else > > diff --git a/src/http/ngx_http_request.c b/src/http/ngx_http_request.c > > --- a/src/http/ngx_http_request.c > > +++ b/src/http/ngx_http_request.c > > @@ -318,12 +318,6 @@ ngx_http_init_connection(ngx_connection_ > > rev->handler = ngx_http_wait_request_handler; > > c->write->handler = ngx_http_empty_handler; > > > > -#if (NGX_HTTP_V2) > > - if (hc->addr_conf->http2) { > > - rev->handler = ngx_http_v2_init; > > - } > > -#endif > > - > > #if (NGX_HTTP_V3) > > if (hc->addr_conf->quic) { > > ngx_http_v3_init_stream(c); > > @@ -383,6 +377,9 @@ ngx_http_wait_request_handler(ngx_event_ > > ngx_buf_t *b; > > ngx_connection_t *c; > > ngx_http_connection_t *hc; > > +#if (NGX_HTTP_V2) > > + ngx_http_v2_srv_conf_t *h2scf; > > +#endif > > ngx_http_core_srv_conf_t *cscf; > > > > c = rev->data; > > @@ -429,6 +426,8 @@ ngx_http_wait_request_handler(ngx_event_ > > b->end = b->last + size; > > } > > > > + size = b->end - b->last; > > + > > n = c->recv(c, b->last, size); > > > > if (n == NGX_AGAIN) { > > @@ -443,12 +442,16 @@ ngx_http_wait_request_handler(ngx_event_ > > return; > > } > > > > - /* > > - * We are trying to not hold c->buffer's memory for an idle connection. > > - */ > > - > > - if (ngx_pfree(c->pool, b->start) == NGX_OK) { > > - b->start = NULL; > > + if (b->pos == b->last) { > > + > > + /* > > + * We are trying to not hold c->buffer's memory for an > > + * idle connection. > > + */ > > + > > + if (ngx_pfree(c->pool, b->start) == NGX_OK) { > > + b->start = NULL; > > + } > > } > > > > return; > > @@ -489,6 +492,30 @@ ngx_http_wait_request_handler(ngx_event_ > > } > > } > > > > +#if (NGX_HTTP_V2) > > + > > + h2scf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_v2_module); > > + > > + if (!c->ssl && (h2scf->enable || hc->addr_conf->http2)) { > > + > > + size = ngx_min(sizeof(NGX_HTTP_V2_PREFACE) - 1, > > + (size_t) (b->last - b->pos)); > > + > > + if (ngx_memcmp(b->pos, NGX_HTTP_V2_PREFACE, size) == 0) { > > + > > + if (size == sizeof(NGX_HTTP_V2_PREFACE) - 1) { > > + ngx_http_v2_init(rev); > > + return; > > + } > > + > > + c->log->action = "waiting for request"; > > + ngx_post_event(rev, &ngx_posted_events); > > + return; > > + } > > + } > > + > > +#endif > > + > > There are two subtile changes: > > - if we receive bad invalid connection preface, > now it is handled in the generic ngx_http_wait_request_handler(), > meaning HTTP/1.1 400 will be sent on error. > This is acceptable by RFC 9113, though: > : A GOAWAY frame (Section 6.8) MAY be omitted in this case, since > : an invalid preface indicates that the peer is not using HTTP/2. > But it probably deserves to write a not in the commit message I think we can mention that unless HTTP/2 preface is matched, HTTP/1 is assumed. > - the same is about bad PROXY protocol string > > Looks like we can optimize parsing PROXY after ngx_http_v2_init(), > since PROXY is now always parsed in ngx_http_wait_request_handler(). Yes, removed reading PROXY protocol from HTTP/2. For both plain TCP and SSL, PROXY protocol header is now read outside of HTTP/2 code, which itself is a good sign. > > c->log->action = "reading client request line"; > > > > ngx_reusable_connection(c, 0); > > @@ -808,13 +835,16 @@ ngx_http_ssl_handshake_handler(ngx_conne > > #if (NGX_HTTP_V2 \ > > && defined TLSEXT_TYPE_application_layer_protocol_negotiation) > > { > > - unsigned int len; > > - const unsigned char *data; > > - ngx_http_connection_t *hc; > > + unsigned int len; > > + const unsigned char *data; > > + ngx_http_connection_t *hc; > > + ngx_http_v2_srv_conf_t *h2scf; > > > > hc = c->data; > > > > - if (hc->addr_conf->http2) { > > + h2scf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_v2_module); > > + > > + if (h2scf->enable || hc->addr_conf->http2) { > > > > SSL_get0_alpn_selected(c->ssl->connection, &data, &len); > > > > diff --git a/src/http/v2/ngx_http_v2.c b/src/http/v2/ngx_http_v2.c > > --- a/src/http/v2/ngx_http_v2.c > > +++ b/src/http/v2/ngx_http_v2.c > > @@ -232,6 +232,7 @@ static ngx_http_v2_parse_header_t ngx_h > > void > > ngx_http_v2_init(ngx_event_t *rev) > > { > > + u_char *p, *end; > > ngx_connection_t *c; > > ngx_pool_cleanup_t *cln; > > ngx_http_connection_t *hc; > > @@ -335,6 +336,29 @@ ngx_http_v2_init(ngx_event_t *rev) > > c->idle = 1; > > ngx_reusable_connection(c, 0); > > > > + if (c->buffer) { > > + p = c->buffer->pos; > > + end = c->buffer->last; > > + > > + do { > > + p = h2c->state.handler(h2c, p, end); > > + > > + if (p == NULL) { > > + return; > > + } > > + > > + } while (p != end); > > + > > + h2c->total_bytes += p - c->buffer->pos; > > + c->buffer->pos = p; > > + > > + if (h2c->total_bytes / 8 > h2c->payload_bytes + 1048576) { > > + ngx_log_error(NGX_LOG_INFO, c->log, 0, "http2 flood detected"); > > + ngx_http_v2_finalize_connection(h2c, NGX_HTTP_V2_NO_ERROR); > > + return; > > + } > > + } > > + > > ngx_http_v2_read_handler(rev); > > } > > > > @@ -871,7 +895,7 @@ static u_char * > > ngx_http_v2_state_preface(ngx_http_v2_connection_t *h2c, u_char *pos, > > u_char *end) > > { > > - static const u_char preface[] = "PRI * HTTP/2.0\r\n"; > > + static const u_char preface[] = NGX_HTTP_V2_PREFACE_START; > > > > if ((size_t) (end - pos) < sizeof(preface) - 1) { > > return ngx_http_v2_state_save(h2c, pos, end, ngx_http_v2_state_preface); > > @@ -892,7 +916,7 @@ static u_char * > > ngx_http_v2_state_preface_end(ngx_http_v2_connection_t *h2c, u_char *pos, > > u_char *end) > > { > > - static const u_char preface[] = "\r\nSM\r\n\r\n"; > > + static const u_char preface[] = NGX_HTTP_V2_PREFACE_END; > > > > if ((size_t) (end - pos) < sizeof(preface) - 1) { > > return ngx_http_v2_state_save(h2c, pos, end, > > @@ -3946,10 +3970,22 @@ static void > > ngx_http_v2_run_request(ngx_http_request_t *r) > > { > > ngx_connection_t *fc; > > + ngx_http_v3_srv_conf_t *h2scf; > > ngx_http_v2_srv_conf_t Fixed, thanks. > > ngx_http_v2_connection_t *h2c; > > > > fc = r->connection; > > > > + h2scf = ngx_http_get_module_srv_conf(r, ngx_http_v2_module); > > + > > + if (!h2scf->enable && !r->http_connection->addr_conf->http2) { > > + ngx_log_error(NGX_LOG_INFO, fc->log, 0, > > + "client attempted to request the server name " > > + "for which the negotitated protocol is unavailable"); > > - typo: negotiated > (same as in 1st patch) Changed to this (same in the 1st patch): "for which the negotiated protocol is disabled" > > + > > + ngx_http_finalize_request(r, NGX_HTTP_MISDIRECTED_REQUEST); > > + goto failed; > > + } > > + > > if (ngx_http_v2_construct_request_line(r) != NGX_OK) { > > goto failed; > > } > > diff --git a/src/http/v2/ngx_http_v2.h b/src/http/v2/ngx_http_v2.h > > --- a/src/http/v2/ngx_http_v2.h > > +++ b/src/http/v2/ngx_http_v2.h > > @@ -64,6 +64,16 @@ typedef u_char *(*ngx_http_v2_handler_pt > > > > > > typedef struct { > > + ngx_flag_t enable; > > + size_t pool_size; > > + ngx_uint_t concurrent_streams; > > + ngx_uint_t concurrent_pushes; > > + size_t preread_size; > > + ngx_uint_t streams_index_mask; > > +} ngx_http_v2_srv_conf_t; > > + > > + > > +typedef struct { > > ngx_str_t name; > > ngx_str_t value; > > } ngx_http_v2_header_t; > > @@ -408,9 +418,17 @@ ngx_int_t ngx_http_v2_table_size(ngx_htt > > #define NGX_HTTP_V2_USER_AGENT_INDEX 58 > > #define NGX_HTTP_V2_VARY_INDEX 59 > > > > +#define NGX_HTTP_V2_PREFACE_START "PRI * HTTP/2.0\r\n" > > +#define NGX_HTTP_V2_PREFACE_END "\r\nSM\r\n\r\n" > > +#define NGX_HTTP_V2_PREFACE NGX_HTTP_V2_PREFACE_START \ > > + NGX_HTTP_V2_PREFACE_END > > + > > > > u_char *ngx_http_v2_string_encode(u_char *dst, u_char *src, size_t len, > > u_char *tmp, ngx_uint_t lower); > > > > > > +extern ngx_module_t ngx_http_v2_module; > > + > > + > > #endif /* _NGX_HTTP_V2_H_INCLUDED_ */ > > diff --git a/src/http/v2/ngx_http_v2_module.c b/src/http/v2/ngx_http_v2_module.c > > --- a/src/http/v2/ngx_http_v2_module.c > > +++ b/src/http/v2/ngx_http_v2_module.c > > @@ -75,6 +75,13 @@ static ngx_conf_post_t ngx_http_v2_chun > > > > static ngx_command_t ngx_http_v2_commands[] = { > > > > + { ngx_string("http2"), > > + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_CONF_FLAG, > > + ngx_conf_set_flag_slot, > > + NGX_HTTP_SRV_CONF_OFFSET, > > + offsetof(ngx_http_v2_srv_conf_t, enable), > > + NULL }, > > + > > { ngx_string("http2_recv_buffer_size"), > > NGX_HTTP_MAIN_CONF|NGX_CONF_TAKE1, > > ngx_conf_set_size_slot, > > @@ -314,6 +321,8 @@ ngx_http_v2_create_srv_conf(ngx_conf_t * > > return NULL; > > } > > > > + h2scf->enable = NGX_CONF_UNSET; > > + > > h2scf->pool_size = NGX_CONF_UNSET_SIZE; > > > > h2scf->concurrent_streams = NGX_CONF_UNSET_UINT; > > @@ -333,6 +342,8 @@ ngx_http_v2_merge_srv_conf(ngx_conf_t *c > > ngx_http_v2_srv_conf_t *prev = parent; > > ngx_http_v2_srv_conf_t *conf = child; > > > > + ngx_conf_merge_value(conf->enable, prev->enable, 0); > > + > > ngx_conf_merge_size_value(conf->pool_size, prev->pool_size, 4096); > > > > ngx_conf_merge_uint_value(conf->concurrent_streams, > > diff --git a/src/http/v2/ngx_http_v2_module.h b/src/http/v2/ngx_http_v2_module.h > > --- a/src/http/v2/ngx_http_v2_module.h > > +++ b/src/http/v2/ngx_http_v2_module.h > > @@ -21,15 +21,6 @@ typedef struct { > > > > > > typedef struct { > > - size_t pool_size; > > - ngx_uint_t concurrent_streams; > > - ngx_uint_t concurrent_pushes; > > - size_t preread_size; > > - ngx_uint_t streams_index_mask; > > -} ngx_http_v2_srv_conf_t; > > - > > - > > -typedef struct { > > size_t chunk_size; > > > > ngx_flag_t push_preload; > > @@ -39,7 +30,4 @@ typedef struct { > > } ngx_http_v2_loc_conf_t; > > > > > > -extern ngx_module_t ngx_http_v2_module; > > - > > - > > #endif /* _NGX_HTTP_V2_MODULE_H_INCLUDED_ */ > > (BTW, we could make a separate ngx_http_v3_module.h as well, > to keep there a private stuff.) > > -- > Sergey Kandaurov > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > https://mailman.nginx.org/mailman/listinfo/nginx-devel -- Roman Arutyunyan From arut at nginx.com Tue Feb 7 14:05:53 2023 From: arut at nginx.com (Roman Arutyunyan) Date: Tue, 7 Feb 2023 18:05:53 +0400 Subject: [PATCH 2 of 3] HTTP/2: "http2" directive In-Reply-To: References: <139b348815b3f1975317.1675260069@arut-laptop> Message-ID: <20230207140553.s3yhh4w4pz42ji5h@N00W24XTQX> Hi, On Tue, Feb 07, 2023 at 06:12:50AM +0300, Maxim Dounin wrote: > Hello! > > On Wed, Feb 01, 2023 at 06:01:09PM +0400, Roman Arutyunyan wrote: > > > # HG changeset patch > > # User Roman Arutyunyan > > # Date 1675255790 -14400 > > # Wed Feb 01 16:49:50 2023 +0400 > > # Branch quic > > # Node ID 139b348815b3f19753176988f06b590a3e0af4c0 > > # Parent 2fcc1c60be1c89aad5464bcc06f1189d1adc885a > > HTTP/2: "http2" directive. > > > > The directive enables HTTP/2 in the current server. The previous way to > > enable HTTP/2 via "listen ... http2" is now deprecated. > > It might be a good idea to describe here how it works, notably > ALPN negotiation (and OpenSSL version where it works for > non-default virtual servers) and preface-based protocol detection > for non-SSL listening sockets. As well as benefits of this > approach. OK, added more details. > > > > diff --git a/src/http/modules/ngx_http_ssl_module.c b/src/http/modules/ngx_http_ssl_module.c > > --- a/src/http/modules/ngx_http_ssl_module.c > > +++ b/src/http/modules/ngx_http_ssl_module.c > > @@ -427,6 +427,9 @@ ngx_http_ssl_alpn_select(ngx_ssl_conn_t > > #if (NGX_HTTP_V2 || NGX_HTTP_V3) > > ngx_http_connection_t *hc; > > #endif > > +#if (NGX_HTTP_V2) > > + ngx_http_v2_srv_conf_t *h2scf; > > +#endif > > #if (NGX_HTTP_V3) > > ngx_http_v3_srv_conf_t *h3scf; > > #endif > > @@ -448,12 +451,9 @@ ngx_http_ssl_alpn_select(ngx_ssl_conn_t > > hc = c->data; > > #endif > > > > -#if (NGX_HTTP_V2) > > - if (hc->addr_conf->http2) { > > - srv = (unsigned char *) NGX_HTTP_V2_ALPN_PROTO NGX_HTTP_ALPN_PROTOS; > > - srvlen = sizeof(NGX_HTTP_V2_ALPN_PROTO NGX_HTTP_ALPN_PROTOS) - 1; > > - } else > > -#endif > > + srv = (unsigned char *) NGX_HTTP_ALPN_PROTOS; > > + srvlen = sizeof(NGX_HTTP_ALPN_PROTOS) - 1; > > + > > #if (NGX_HTTP_V3) > > if (hc->addr_conf->quic) { > > > > @@ -479,10 +479,16 @@ ngx_http_ssl_alpn_select(ngx_ssl_conn_t > > > > } else > > #endif > > +#if (NGX_HTTP_V2) > > { > > - srv = (unsigned char *) NGX_HTTP_ALPN_PROTOS; > > - srvlen = sizeof(NGX_HTTP_ALPN_PROTOS) - 1; > > + h2scf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_v2_module); > > + > > + if (h2scf->enable || hc->addr_conf->http2) { > > + srv = (unsigned char *) NGX_HTTP_V2_ALPN_PROTO NGX_HTTP_ALPN_PROTOS; > > + srvlen = sizeof(NGX_HTTP_V2_ALPN_PROTO NGX_HTTP_ALPN_PROTOS) - 1; > > + } > > } > > +#endif > > > > if (SSL_select_next_proto((unsigned char **) out, outlen, srv, srvlen, > > in, inlen) > > diff --git a/src/http/ngx_http_core_module.c b/src/http/ngx_http_core_module.c > > --- a/src/http/ngx_http_core_module.c > > +++ b/src/http/ngx_http_core_module.c > > @@ -4179,6 +4179,11 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx > > > > if (ngx_strcmp(value[n].data, "http2") == 0) { > > #if (NGX_HTTP_V2) > > + ngx_conf_log_error(NGX_LOG_WARN, cf, 0, > > + "the \"listen ... http2\" directive " > > + "is deprecated, use " > > + "the \"http2\" directive instead"); > > + > > lsopt.http2 = 1; > > continue; > > #else > > diff --git a/src/http/ngx_http_request.c b/src/http/ngx_http_request.c > > --- a/src/http/ngx_http_request.c > > +++ b/src/http/ngx_http_request.c > > @@ -318,12 +318,6 @@ ngx_http_init_connection(ngx_connection_ > > rev->handler = ngx_http_wait_request_handler; > > c->write->handler = ngx_http_empty_handler; > > > > -#if (NGX_HTTP_V2) > > - if (hc->addr_conf->http2) { > > - rev->handler = ngx_http_v2_init; > > - } > > -#endif > > - > > #if (NGX_HTTP_V3) > > if (hc->addr_conf->quic) { > > ngx_http_v3_init_stream(c); > > @@ -383,6 +377,9 @@ ngx_http_wait_request_handler(ngx_event_ > > ngx_buf_t *b; > > ngx_connection_t *c; > > ngx_http_connection_t *hc; > > +#if (NGX_HTTP_V2) > > + ngx_http_v2_srv_conf_t *h2scf; > > +#endif > > ngx_http_core_srv_conf_t *cscf; > > > > c = rev->data; > > @@ -429,6 +426,8 @@ ngx_http_wait_request_handler(ngx_event_ > > b->end = b->last + size; > > } > > > > + size = b->end - b->last; > > + > > n = c->recv(c, b->last, size); > > > > if (n == NGX_AGAIN) { > > @@ -443,12 +442,16 @@ ngx_http_wait_request_handler(ngx_event_ > > return; > > } > > > > - /* > > - * We are trying to not hold c->buffer's memory for an idle connection. > > - */ > > - > > - if (ngx_pfree(c->pool, b->start) == NGX_OK) { > > - b->start = NULL; > > + if (b->pos == b->last) { > > + > > + /* > > + * We are trying to not hold c->buffer's memory for an > > + * idle connection. > > + */ > > + > > + if (ngx_pfree(c->pool, b->start) == NGX_OK) { > > + b->start = NULL; > > + } > > } > > > > return; > > @@ -489,6 +492,30 @@ ngx_http_wait_request_handler(ngx_event_ > > } > > } > > > > +#if (NGX_HTTP_V2) > > + > > + h2scf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_v2_module); > > + > > + if (!c->ssl && (h2scf->enable || hc->addr_conf->http2)) { > > + > > + size = ngx_min(sizeof(NGX_HTTP_V2_PREFACE) - 1, > > + (size_t) (b->last - b->pos)); > > + > > + if (ngx_memcmp(b->pos, NGX_HTTP_V2_PREFACE, size) == 0) { > > + > > + if (size == sizeof(NGX_HTTP_V2_PREFACE) - 1) { > > + ngx_http_v2_init(rev); > > + return; > > + } > > Note that ngx_http_wait_request_handler() is used for both > new and keepalive connections. It does not look like a good idea > to start an HTTP/2 connection after processing some HTTP/1.x > requests. Keepalive has its own handler ngx_http_keepalive_handler(), which is similar to ngx_http_wait_request_handler(), but does not try to recognize HTTP/2. > > + > > + c->log->action = "waiting for request"; > > + ngx_post_event(rev, &ngx_posted_events); > > + return; > > + } > > + } > > + > > +#endif > > + > > c->log->action = "reading client request line"; > > > > ngx_reusable_connection(c, 0); > > @@ -808,13 +835,16 @@ ngx_http_ssl_handshake_handler(ngx_conne > > #if (NGX_HTTP_V2 \ > > && defined TLSEXT_TYPE_application_layer_protocol_negotiation) > > { > > - unsigned int len; > > - const unsigned char *data; > > - ngx_http_connection_t *hc; > > + unsigned int len; > > + const unsigned char *data; > > + ngx_http_connection_t *hc; > > + ngx_http_v2_srv_conf_t *h2scf; > > > > hc = c->data; > > > > - if (hc->addr_conf->http2) { > > + h2scf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_v2_module); > > + > > + if (h2scf->enable || hc->addr_conf->http2) { > > > > SSL_get0_alpn_selected(c->ssl->connection, &data, &len); > > > > diff --git a/src/http/v2/ngx_http_v2.c b/src/http/v2/ngx_http_v2.c > > --- a/src/http/v2/ngx_http_v2.c > > +++ b/src/http/v2/ngx_http_v2.c > > @@ -232,6 +232,7 @@ static ngx_http_v2_parse_header_t ngx_h > > void > > ngx_http_v2_init(ngx_event_t *rev) > > { > > + u_char *p, *end; > > ngx_connection_t *c; > > ngx_pool_cleanup_t *cln; > > ngx_http_connection_t *hc; > > @@ -335,6 +336,29 @@ ngx_http_v2_init(ngx_event_t *rev) > > c->idle = 1; > > ngx_reusable_connection(c, 0); > > > > + if (c->buffer) { > > + p = c->buffer->pos; > > + end = c->buffer->last; > > + > > + do { > > + p = h2c->state.handler(h2c, p, end); > > + > > + if (p == NULL) { > > + return; > > + } > > + > > + } while (p != end); > > + > > + h2c->total_bytes += p - c->buffer->pos; > > + c->buffer->pos = p; > > + > > + if (h2c->total_bytes / 8 > h2c->payload_bytes + 1048576) { > > + ngx_log_error(NGX_LOG_INFO, c->log, 0, "http2 flood detected"); > > + ngx_http_v2_finalize_connection(h2c, NGX_HTTP_V2_NO_ERROR); > > + return; > > + } > > I think flood protection can be safely omitted here, given that > client_header_buffer_size is expected to be small (1k by default), > and flood protection is not going to kick in till at least 8M of > total bytes from the client. Yes, let's remove this block. > > + } > > + > > ngx_http_v2_read_handler(rev); > > } > > > > @@ -871,7 +895,7 @@ static u_char * > > ngx_http_v2_state_preface(ngx_http_v2_connection_t *h2c, u_char *pos, > > u_char *end) > > { > > - static const u_char preface[] = "PRI * HTTP/2.0\r\n"; > > + static const u_char preface[] = NGX_HTTP_V2_PREFACE_START; > > > > if ((size_t) (end - pos) < sizeof(preface) - 1) { > > return ngx_http_v2_state_save(h2c, pos, end, ngx_http_v2_state_preface); > > @@ -892,7 +916,7 @@ static u_char * > > ngx_http_v2_state_preface_end(ngx_http_v2_connection_t *h2c, u_char *pos, > > u_char *end) > > { > > - static const u_char preface[] = "\r\nSM\r\n\r\n"; > > + static const u_char preface[] = NGX_HTTP_V2_PREFACE_END; > > > > if ((size_t) (end - pos) < sizeof(preface) - 1) { > > return ngx_http_v2_state_save(h2c, pos, end, > > @@ -3946,10 +3970,22 @@ static void > > ngx_http_v2_run_request(ngx_http_request_t *r) > > { > > ngx_connection_t *fc; > > + ngx_http_v3_srv_conf_t *h2scf; > > ngx_http_v2_connection_t *h2c; > > > > fc = r->connection; > > > > + h2scf = ngx_http_get_module_srv_conf(r, ngx_http_v2_module); > > + > > + if (!h2scf->enable && !r->http_connection->addr_conf->http2) { > > + ngx_log_error(NGX_LOG_INFO, fc->log, 0, > > + "client attempted to request the server name " > > + "for which the negotitated protocol is unavailable"); > > Typo, should be "negotiated". Thanks, changed to: "for which the negotiated protocol is disabled" > > + > > + ngx_http_finalize_request(r, NGX_HTTP_MISDIRECTED_REQUEST); > > + goto failed; > > + } > > + > > if (ngx_http_v2_construct_request_line(r) != NGX_OK) { > > goto failed; > > } > > diff --git a/src/http/v2/ngx_http_v2.h b/src/http/v2/ngx_http_v2.h > > --- a/src/http/v2/ngx_http_v2.h > > +++ b/src/http/v2/ngx_http_v2.h > > @@ -64,6 +64,16 @@ typedef u_char *(*ngx_http_v2_handler_pt > > > > > > typedef struct { > > + ngx_flag_t enable; > > + size_t pool_size; > > + ngx_uint_t concurrent_streams; > > + ngx_uint_t concurrent_pushes; > > + size_t preread_size; > > + ngx_uint_t streams_index_mask; > > +} ngx_http_v2_srv_conf_t; > > + > > + > > +typedef struct { > > ngx_str_t name; > > ngx_str_t value; > > } ngx_http_v2_header_t; > > @@ -408,9 +418,17 @@ ngx_int_t ngx_http_v2_table_size(ngx_htt > > #define NGX_HTTP_V2_USER_AGENT_INDEX 58 > > #define NGX_HTTP_V2_VARY_INDEX 59 > > > > +#define NGX_HTTP_V2_PREFACE_START "PRI * HTTP/2.0\r\n" > > +#define NGX_HTTP_V2_PREFACE_END "\r\nSM\r\n\r\n" > > +#define NGX_HTTP_V2_PREFACE NGX_HTTP_V2_PREFACE_START \ > > + NGX_HTTP_V2_PREFACE_END > > + > > > > u_char *ngx_http_v2_string_encode(u_char *dst, u_char *src, size_t len, > > u_char *tmp, ngx_uint_t lower); > > > > > > +extern ngx_module_t ngx_http_v2_module; > > + > > + > > #endif /* _NGX_HTTP_V2_H_INCLUDED_ */ > > diff --git a/src/http/v2/ngx_http_v2_module.c b/src/http/v2/ngx_http_v2_module.c > > --- a/src/http/v2/ngx_http_v2_module.c > > +++ b/src/http/v2/ngx_http_v2_module.c > > @@ -75,6 +75,13 @@ static ngx_conf_post_t ngx_http_v2_chun > > > > static ngx_command_t ngx_http_v2_commands[] = { > > > > + { ngx_string("http2"), > > + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_CONF_FLAG, > > + ngx_conf_set_flag_slot, > > + NGX_HTTP_SRV_CONF_OFFSET, > > + offsetof(ngx_http_v2_srv_conf_t, enable), > > + NULL }, > > + > > { ngx_string("http2_recv_buffer_size"), > > NGX_HTTP_MAIN_CONF|NGX_CONF_TAKE1, > > ngx_conf_set_size_slot, > > @@ -314,6 +321,8 @@ ngx_http_v2_create_srv_conf(ngx_conf_t * > > return NULL; > > } > > > > + h2scf->enable = NGX_CONF_UNSET; > > + > > h2scf->pool_size = NGX_CONF_UNSET_SIZE; > > > > h2scf->concurrent_streams = NGX_CONF_UNSET_UINT; > > @@ -333,6 +342,8 @@ ngx_http_v2_merge_srv_conf(ngx_conf_t *c > > ngx_http_v2_srv_conf_t *prev = parent; > > ngx_http_v2_srv_conf_t *conf = child; > > > > + ngx_conf_merge_value(conf->enable, prev->enable, 0); > > + > > ngx_conf_merge_size_value(conf->pool_size, prev->pool_size, 4096); > > > > ngx_conf_merge_uint_value(conf->concurrent_streams, > > diff --git a/src/http/v2/ngx_http_v2_module.h b/src/http/v2/ngx_http_v2_module.h > > --- a/src/http/v2/ngx_http_v2_module.h > > +++ b/src/http/v2/ngx_http_v2_module.h > > @@ -21,15 +21,6 @@ typedef struct { > > > > > > typedef struct { > > - size_t pool_size; > > - ngx_uint_t concurrent_streams; > > - ngx_uint_t concurrent_pushes; > > - size_t preread_size; > > - ngx_uint_t streams_index_mask; > > -} ngx_http_v2_srv_conf_t; > > - > > - > > -typedef struct { > > size_t chunk_size; > > > > ngx_flag_t push_preload; > > @@ -39,7 +30,4 @@ typedef struct { > > } ngx_http_v2_loc_conf_t; > > > > > > -extern ngx_module_t ngx_http_v2_module; > > - > > - > > #endif /* _NGX_HTTP_V2_MODULE_H_INCLUDED_ */ > > Otherwise looks good. Also, ngx_http_wait_request_handler() disables reusable mode for the connection prior to creating the request, and ngx_http_v2_init() does the same. It seems a good idea to move ngx_reusable_connection(c, 0) up before detecting HTTP/2 preface and remove this call from ngx_http_v2_init(). This will simplify HTTP/2 code and also keep the connection from being closed while reading the preface, which is what we do while reading HTTP/1 request line. > -- > Maxim Dounin > http://mdounin.ru/ > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > https://mailman.nginx.org/mailman/listinfo/nginx-devel -- Roman Arutyunyan From arut at nginx.com Tue Feb 7 14:50:29 2023 From: arut at nginx.com (=?iso-8859-1?q?Roman_Arutyunyan?=) Date: Tue, 07 Feb 2023 18:50:29 +0400 Subject: [PATCH 0 of 3] Directives for enabling http2 and http3 In-Reply-To: References: Message-ID: Hi, Updated the patches based on latest comments. -- Roman Arutyunyan From arut at nginx.com Tue Feb 7 14:50:30 2023 From: arut at nginx.com (=?iso-8859-1?q?Roman_Arutyunyan?=) Date: Tue, 07 Feb 2023 18:50:30 +0400 Subject: [PATCH 1 of 3] HTTP/3: "quic" parameter of "listen" directive In-Reply-To: References: Message-ID: <38eec3d9f2c0d2e6d041.1675781430@arut-laptop> # HG changeset patch # User Roman Arutyunyan # Date 1675779344 -14400 # Tue Feb 07 18:15:44 2023 +0400 # Branch quic # Node ID 38eec3d9f2c0d2e6d041efe3ee6d9c1618d8f1e6 # Parent def8e398d7c50131f8dac844814fff729da5c86c HTTP/3: "quic" parameter of "listen" directive. Now "listen" directve has a new "quic" parameter which enables QUIC protocol for the address. Further, to enable HTTP/3, a new directive "http3" is introduced. The hq-interop protocol is enabled by "http3_hq" as before. Now application protocol is chosen by ALPN. Previously used "http3" parameter of "listen" is deprecated. diff --git a/README b/README --- a/README +++ b/README @@ -93,13 +93,13 @@ 2. Installing 3. Configuration - The HTTP "listen" directive got a new option "http3" which enables - HTTP/3 over QUIC on the specified port. + The HTTP "listen" directive got a new option "quic" which enables + QUIC as client transport protocol instead of TCP. The Stream "listen" directive got a new option "quic" which enables QUIC as client transport protocol instead of TCP or plain UDP. - Along with "http3" or "quic", it's also possible to specify "reuseport" + Along with "quic", it's also possible to specify "reuseport" option [8] to make it work properly with multiple workers. To enable address validation: @@ -133,12 +133,13 @@ 3. Configuration A number of directives were added that configure HTTP/3: + http3 + http3_hq http3_stream_buffer_size http3_max_concurrent_pushes http3_max_concurrent_streams http3_push http3_push_preload - http3_hq (requires NGX_HTTP_V3_HQ macro) In http, an additional variable is available: $http3. The value of $http3 is "h3" for HTTP/3 connections, @@ -160,7 +161,7 @@ Example configuration: server { # for better compatibility it's recommended # to use the same port for quic and https - listen 8443 http3 reuseport; + listen 8443 quic reuseport; listen 8443 ssl; ssl_certificate certs/example.com.crt; diff --git a/src/http/modules/ngx_http_ssl_module.c b/src/http/modules/ngx_http_ssl_module.c --- a/src/http/modules/ngx_http_ssl_module.c +++ b/src/http/modules/ngx_http_ssl_module.c @@ -427,7 +427,7 @@ ngx_http_ssl_alpn_select(ngx_ssl_conn_t #if (NGX_HTTP_V2 || NGX_HTTP_V3) ngx_http_connection_t *hc; #endif -#if (NGX_HTTP_V3 && NGX_HTTP_V3_HQ) +#if (NGX_HTTP_V3) ngx_http_v3_srv_conf_t *h3scf; #endif #if (NGX_HTTP_V2 || NGX_HTTP_V3 || NGX_DEBUG) @@ -455,19 +455,26 @@ ngx_http_ssl_alpn_select(ngx_ssl_conn_t } else #endif #if (NGX_HTTP_V3) - if (hc->addr_conf->http3) { + if (hc->addr_conf->quic) { -#if (NGX_HTTP_V3_HQ) h3scf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_v3_module); - if (h3scf->hq) { + if (h3scf->enable && h3scf->enable_hq) { + srv = (unsigned char *) NGX_HTTP_V3_ALPN_PROTO + NGX_HTTP_V3_HQ_ALPN_PROTO; + srvlen = sizeof(NGX_HTTP_V3_ALPN_PROTO NGX_HTTP_V3_HQ_ALPN_PROTO) + - 1; + + } else if (h3scf->enable_hq) { srv = (unsigned char *) NGX_HTTP_V3_HQ_ALPN_PROTO; srvlen = sizeof(NGX_HTTP_V3_HQ_ALPN_PROTO) - 1; - } else -#endif - { + + } else if (h3scf->enable || hc->addr_conf->http3) { srv = (unsigned char *) NGX_HTTP_V3_ALPN_PROTO; srvlen = sizeof(NGX_HTTP_V3_ALPN_PROTO) - 1; + + } else { + return SSL_TLSEXT_ERR_ALERT_FATAL; } } else @@ -1313,12 +1320,12 @@ ngx_http_ssl_init(ngx_conf_t *cf) addr = port[p].addrs.elts; for (a = 0; a < port[p].addrs.nelts; a++) { - if (!addr[a].opt.ssl && !addr[a].opt.http3) { + if (!addr[a].opt.ssl && !addr[a].opt.quic) { continue; } - if (addr[a].opt.http3) { - name = "http3"; + if (addr[a].opt.quic) { + name = "quic"; } else { name = "ssl"; @@ -1329,7 +1336,7 @@ ngx_http_ssl_init(ngx_conf_t *cf) if (sscf->certificates) { - if (addr[a].opt.http3 && !(sscf->protocols & NGX_SSL_TLSv1_3)) { + if (addr[a].opt.quic && !(sscf->protocols & NGX_SSL_TLSv1_3)) { ngx_log_error(NGX_LOG_EMERG, cf->log, 0, "\"ssl_protocols\" must enable TLSv1.3 for " "the \"listen ... %s\" directive in %s:%ui", diff --git a/src/http/ngx_http.c b/src/http/ngx_http.c --- a/src/http/ngx_http.c +++ b/src/http/ngx_http.c @@ -1242,6 +1242,7 @@ ngx_http_add_addresses(ngx_conf_t *cf, n #endif #if (NGX_HTTP_V3) ngx_uint_t http3; + ngx_uint_t quic; #endif /* @@ -1280,6 +1281,7 @@ ngx_http_add_addresses(ngx_conf_t *cf, n #endif #if (NGX_HTTP_V3) http3 = lsopt->http3 || addr[i].opt.http3; + quic = lsopt->quic || addr[i].opt.quic; #endif if (lsopt->set) { @@ -1319,6 +1321,7 @@ ngx_http_add_addresses(ngx_conf_t *cf, n #endif #if (NGX_HTTP_V3) addr[i].opt.http3 = http3; + addr[i].opt.quic = quic; #endif return NGX_OK; @@ -1823,7 +1826,7 @@ ngx_http_add_listening(ngx_conf_t *cf, n #if (NGX_HTTP_V3) - ls->quic = addr->opt.http3; + ls->quic = addr->opt.quic; if (ls->quic) { ngx_rbtree_init(&ls->rbtree, &ls->sentinel, @@ -1866,6 +1869,7 @@ ngx_http_add_addrs(ngx_conf_t *cf, ngx_h #endif #if (NGX_HTTP_V3) addrs[i].conf.http3 = addr[i].opt.http3; + addrs[i].conf.quic = addr[i].opt.quic; #endif addrs[i].conf.proxy_protocol = addr[i].opt.proxy_protocol; @@ -1934,6 +1938,7 @@ ngx_http_add_addrs6(ngx_conf_t *cf, ngx_ #endif #if (NGX_HTTP_V3) addrs6[i].conf.http3 = addr[i].opt.http3; + addrs6[i].conf.quic = addr[i].opt.quic; #endif addrs6[i].conf.proxy_protocol = addr[i].opt.proxy_protocol; diff --git a/src/http/ngx_http_core_module.c b/src/http/ngx_http_core_module.c --- a/src/http/ngx_http_core_module.c +++ b/src/http/ngx_http_core_module.c @@ -4191,6 +4191,10 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx if (ngx_strcmp(value[n].data, "http3") == 0) { #if (NGX_HTTP_V3) + ngx_conf_log_error(NGX_LOG_WARN, cf, 0, + "the \"http3\" parameter is deprecated, " + "use \"quic\" parameter instead"); + lsopt.quic = 1; lsopt.http3 = 1; lsopt.type = SOCK_DGRAM; continue; @@ -4202,6 +4206,19 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx #endif } + if (ngx_strcmp(value[n].data, "quic") == 0) { +#if (NGX_HTTP_V3) + lsopt.quic = 1; + lsopt.type = SOCK_DGRAM; + continue; +#else + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, + "the \"quic\" parameter requires " + "ngx_http_v3_module"); + return NGX_CONF_ERROR; +#endif + } + if (ngx_strncmp(value[n].data, "so_keepalive=", 13) == 0) { if (ngx_strcmp(&value[n].data[13], "on") == 0) { @@ -4304,8 +4321,8 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx } #if (NGX_HTTP_SSL && NGX_HTTP_V3) - if (lsopt.ssl && lsopt.http3) { - return "\"ssl\" parameter is incompatible with \"http3\""; + if (lsopt.ssl && lsopt.quic) { + return "\"ssl\" parameter is incompatible with \"quic\""; } #endif diff --git a/src/http/ngx_http_core_module.h b/src/http/ngx_http_core_module.h --- a/src/http/ngx_http_core_module.h +++ b/src/http/ngx_http_core_module.h @@ -76,6 +76,7 @@ typedef struct { unsigned ssl:1; unsigned http2:1; unsigned http3:1; + unsigned quic:1; #if (NGX_HAVE_INET6) unsigned ipv6only:1; #endif @@ -240,6 +241,7 @@ struct ngx_http_addr_conf_s { unsigned ssl:1; unsigned http2:1; unsigned http3:1; + unsigned quic:1; unsigned proxy_protocol:1; }; diff --git a/src/http/ngx_http_request.c b/src/http/ngx_http_request.c --- a/src/http/ngx_http_request.c +++ b/src/http/ngx_http_request.c @@ -325,7 +325,7 @@ ngx_http_init_connection(ngx_connection_ #endif #if (NGX_HTTP_V3) - if (hc->addr_conf->http3) { + if (hc->addr_conf->quic) { ngx_http_v3_init_stream(c); return; } diff --git a/src/http/v3/ngx_http_v3.c b/src/http/v3/ngx_http_v3.c --- a/src/http/v3/ngx_http_v3.c +++ b/src/http/v3/ngx_http_v3.c @@ -17,12 +17,9 @@ static void ngx_http_v3_cleanup_session( ngx_int_t ngx_http_v3_init_session(ngx_connection_t *c) { - ngx_pool_cleanup_t *cln; - ngx_http_connection_t *hc; - ngx_http_v3_session_t *h3c; -#if (NGX_HTTP_V3_HQ) - ngx_http_v3_srv_conf_t *h3scf; -#endif + ngx_pool_cleanup_t *cln; + ngx_http_connection_t *hc; + ngx_http_v3_session_t *h3c; hc = c->data; @@ -36,13 +33,6 @@ ngx_http_v3_init_session(ngx_connection_ h3c->max_push_id = (uint64_t) -1; h3c->goaway_push_id = (uint64_t) -1; -#if (NGX_HTTP_V3_HQ) - h3scf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_v3_module); - if (h3scf->hq) { - h3c->hq = 1; - } -#endif - ngx_queue_init(&h3c->blocked); ngx_queue_init(&h3c->pushing); diff --git a/src/http/v3/ngx_http_v3.h b/src/http/v3/ngx_http_v3.h --- a/src/http/v3/ngx_http_v3.h +++ b/src/http/v3/ngx_http_v3.h @@ -21,6 +21,7 @@ #define NGX_HTTP_V3_ALPN_PROTO "\x02h3" #define NGX_HTTP_V3_HQ_ALPN_PROTO "\x0Ahq-interop" +#define NGX_HTTP_V3_HQ_PROTO "hq-interop" #define NGX_HTTP_V3_VARLEN_INT_LEN 4 #define NGX_HTTP_V3_PREFIX_INT_LEN 11 @@ -101,13 +102,12 @@ typedef struct { + ngx_flag_t enable; + ngx_flag_t enable_hq; size_t max_table_capacity; ngx_uint_t max_blocked_streams; ngx_uint_t max_concurrent_pushes; ngx_uint_t max_concurrent_streams; -#if (NGX_HTTP_V3_HQ) - ngx_flag_t hq; -#endif ngx_quic_conf_t quic; } ngx_http_v3_srv_conf_t; @@ -147,9 +147,7 @@ struct ngx_http_v3_session_s { off_t payload_bytes; unsigned goaway:1; -#if (NGX_HTTP_V3_HQ) unsigned hq:1; -#endif ngx_connection_t *known_streams[NGX_HTTP_V3_MAX_KNOWN_STREAM]; }; diff --git a/src/http/v3/ngx_http_v3_module.c b/src/http/v3/ngx_http_v3_module.c --- a/src/http/v3/ngx_http_v3_module.c +++ b/src/http/v3/ngx_http_v3_module.c @@ -32,6 +32,20 @@ static ngx_conf_post_t ngx_http_quic_mt static ngx_command_t ngx_http_v3_commands[] = { + { ngx_string("http3"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_CONF_FLAG, + ngx_conf_set_flag_slot, + NGX_HTTP_SRV_CONF_OFFSET, + offsetof(ngx_http_v3_srv_conf_t, enable), + NULL }, + + { ngx_string("http3_hq"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_CONF_FLAG, + ngx_conf_set_flag_slot, + NGX_HTTP_SRV_CONF_OFFSET, + offsetof(ngx_http_v3_srv_conf_t, enable_hq), + NULL }, + { ngx_string("http3_max_concurrent_pushes"), NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_CONF_TAKE1, ngx_conf_set_num_slot, @@ -46,15 +60,6 @@ static ngx_command_t ngx_http_v3_comman offsetof(ngx_http_v3_srv_conf_t, max_concurrent_streams), NULL }, -#if (NGX_HTTP_V3_HQ) - { ngx_string("http3_hq"), - NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_CONF_FLAG, - ngx_conf_set_flag_slot, - NGX_HTTP_SRV_CONF_OFFSET, - offsetof(ngx_http_v3_srv_conf_t, hq), - NULL }, -#endif - { ngx_string("http3_push"), NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1, ngx_http_v3_push, @@ -160,14 +165,12 @@ static ngx_int_t ngx_http_v3_variable(ngx_http_request_t *r, ngx_http_variable_value_t *v, uintptr_t data) { - if (r->connection->quic) { -#if (NGX_HTTP_V3_HQ) + ngx_http_v3_session_t *h3c; - ngx_http_v3_srv_conf_t *h3scf; + if (r->connection->quic) { + h3c = ngx_http_v3_get_session(r->connection); - h3scf = ngx_http_get_module_srv_conf(r, ngx_http_v3_module); - - if (h3scf->hq) { + if (h3c->hq) { v->len = sizeof("hq") - 1; v->valid = 1; v->no_cacheable = 0; @@ -177,8 +180,6 @@ ngx_http_v3_variable(ngx_http_request_t return NGX_OK; } -#endif - v->len = sizeof("h3") - 1; v->valid = 1; v->no_cacheable = 0; @@ -232,12 +233,12 @@ ngx_http_v3_create_srv_conf(ngx_conf_t * * h3scf->quic.timeout = 0; * h3scf->max_blocked_streams = 0; */ + + h3scf->enable = NGX_CONF_UNSET; + h3scf->enable_hq = NGX_CONF_UNSET; h3scf->max_table_capacity = NGX_HTTP_V3_MAX_TABLE_CAPACITY; h3scf->max_concurrent_pushes = NGX_CONF_UNSET_UINT; h3scf->max_concurrent_streams = NGX_CONF_UNSET_UINT; -#if (NGX_HTTP_V3_HQ) - h3scf->hq = NGX_CONF_UNSET; -#endif h3scf->quic.mtu = NGX_CONF_UNSET_SIZE; h3scf->quic.stream_buffer_size = NGX_CONF_UNSET_SIZE; @@ -264,6 +265,10 @@ ngx_http_v3_merge_srv_conf(ngx_conf_t *c ngx_http_ssl_srv_conf_t *sscf; + ngx_conf_merge_value(conf->enable, prev->enable, 1); + + ngx_conf_merge_value(conf->enable_hq, prev->enable_hq, 0); + ngx_conf_merge_uint_value(conf->max_concurrent_pushes, prev->max_concurrent_pushes, 10); @@ -272,11 +277,6 @@ ngx_http_v3_merge_srv_conf(ngx_conf_t *c conf->max_blocked_streams = conf->max_concurrent_streams; -#if (NGX_HTTP_V3_HQ) - ngx_conf_merge_value(conf->hq, prev->hq, 0); -#endif - - ngx_conf_merge_size_value(conf->quic.mtu, prev->quic.mtu, NGX_QUIC_MAX_UDP_PAYLOAD_SIZE); diff --git a/src/http/v3/ngx_http_v3_request.c b/src/http/v3/ngx_http_v3_request.c --- a/src/http/v3/ngx_http_v3_request.c +++ b/src/http/v3/ngx_http_v3_request.c @@ -110,7 +110,10 @@ ngx_http_v3_init_stream(ngx_connection_t ngx_int_t ngx_http_v3_init(ngx_connection_t *c) { + unsigned int len; + const unsigned char *data; ngx_http_v3_session_t *h3c; + ngx_http_v3_srv_conf_t *h3scf; ngx_http_core_loc_conf_t *clcf; ngx_log_debug0(NGX_LOG_DEBUG_HTTP, c->log, 0, "http3 init"); @@ -119,11 +122,23 @@ ngx_http_v3_init(ngx_connection_t *c) clcf = ngx_http_v3_get_module_loc_conf(c, ngx_http_core_module); ngx_add_timer(&h3c->keepalive, clcf->keepalive_timeout); -#if (NGX_HTTP_V3_HQ) - if (h3c->hq) { - return NGX_OK; + h3scf = ngx_http_v3_get_module_srv_conf(c, ngx_http_v3_module); + + if (h3scf->enable_hq) { + if (!h3scf->enable) { + h3c->hq = 1; + return NGX_OK; + } + + SSL_get0_alpn_selected(c->ssl->connection, &data, &len); + + if (len == sizeof(NGX_HTTP_V3_HQ_PROTO) - 1 + && ngx_strncmp(data, NGX_HTTP_V3_HQ_PROTO, len) == 0) + { + h3c->hq = 1; + return NGX_OK; + } } -#endif return ngx_http_v3_send_settings(c); } @@ -147,10 +162,7 @@ ngx_http_v3_shutdown(ngx_connection_t *c if (!h3c->goaway) { h3c->goaway = 1; -#if (NGX_HTTP_V3_HQ) - if (!h3c->hq) -#endif - { + if (!h3c->hq) { (void) ngx_http_v3_send_goaway(c, h3c->next_request_id); } @@ -205,10 +217,7 @@ ngx_http_v3_init_request_stream(ngx_conn { h3c->goaway = 1; -#if (NGX_HTTP_V3_HQ) - if (!h3c->hq) -#endif - { + if (!h3c->hq) { if (ngx_http_v3_send_goaway(c, h3c->next_request_id) != NGX_OK) { ngx_http_close_connection(c); return; @@ -236,10 +245,7 @@ ngx_http_v3_init_request_stream(ngx_conn rev = c->read; -#if (NGX_HTTP_V3_HQ) - if (!h3c->hq) -#endif - { + if (!h3c->hq) { rev->handler = ngx_http_v3_wait_request_handler; c->write->handler = ngx_http_empty_handler; } @@ -398,14 +404,14 @@ ngx_http_v3_wait_request_handler(ngx_eve void ngx_http_v3_reset_stream(ngx_connection_t *c) { + ngx_http_v3_session_t *h3c; ngx_http_v3_srv_conf_t *h3scf; h3scf = ngx_http_v3_get_module_srv_conf(c, ngx_http_v3_module); - if (h3scf->max_table_capacity > 0 && !c->read->eof -#if (NGX_HTTP_V3_HQ) - && !h3scf->hq -#endif + h3c = ngx_http_v3_get_session(c); + + if (h3scf->max_table_capacity > 0 && !c->read->eof && !h3c->hq && (c->quic->id & NGX_QUIC_STREAM_UNIDIRECTIONAL) == 0) { (void) ngx_http_v3_send_cancel_stream(c, c->quic->id); @@ -993,9 +999,11 @@ failed: static ngx_int_t ngx_http_v3_process_request_header(ngx_http_request_t *r) { - ssize_t n; - ngx_buf_t *b; - ngx_connection_t *c; + ssize_t n; + ngx_buf_t *b; + ngx_connection_t *c; + ngx_http_v3_session_t *h3c; + ngx_http_v3_srv_conf_t *h3scf; c = r->connection; @@ -1003,6 +1011,19 @@ ngx_http_v3_process_request_header(ngx_h return NGX_ERROR; } + h3c = ngx_http_v3_get_session(c); + h3scf = ngx_http_get_module_srv_conf(r, ngx_http_v3_module); + + if (!r->http_connection->addr_conf->http3) { + if ((h3c->hq && !h3scf->enable_hq) || (!h3c->hq && !h3scf->enable)) { + ngx_log_error(NGX_LOG_INFO, c->log, 0, + "client attempted to request the server name " + "for which the negotiated protocol is disabled"); + ngx_http_finalize_request(r, NGX_HTTP_MISDIRECTED_REQUEST); + return NGX_ERROR; + } + } + if (ngx_http_v3_construct_cookie_header(r) != NGX_OK) { return NGX_ERROR; } diff --git a/src/http/v3/ngx_http_v3_uni.c b/src/http/v3/ngx_http_v3_uni.c --- a/src/http/v3/ngx_http_v3_uni.c +++ b/src/http/v3/ngx_http_v3_uni.c @@ -37,12 +37,9 @@ void ngx_http_v3_init_uni_stream(ngx_connection_t *c) { uint64_t n; -#if (NGX_HTTP_V3_HQ) ngx_http_v3_session_t *h3c; -#endif ngx_http_v3_uni_stream_t *us; -#if (NGX_HTTP_V3_HQ) h3c = ngx_http_v3_get_session(c); if (h3c->hq) { ngx_http_v3_finalize_connection(c, @@ -52,7 +49,6 @@ ngx_http_v3_init_uni_stream(ngx_connecti ngx_http_v3_close_uni_stream(c); return; } -#endif ngx_log_debug0(NGX_LOG_DEBUG_HTTP, c->log, 0, "http3 init uni stream"); From arut at nginx.com Tue Feb 7 14:50:31 2023 From: arut at nginx.com (=?iso-8859-1?q?Roman_Arutyunyan?=) Date: Tue, 07 Feb 2023 18:50:31 +0400 Subject: [PATCH 2 of 3] HTTP/2: "http2" directive In-Reply-To: References: Message-ID: <735f9e501922e4b0a1b2.1675781431@arut-laptop> # HG changeset patch # User Roman Arutyunyan # Date 1675781276 -14400 # Tue Feb 07 18:47:56 2023 +0400 # Branch quic # Node ID 735f9e501922e4b0a1b20730d62bac35ea398336 # Parent 38eec3d9f2c0d2e6d041efe3ee6d9c1618d8f1e6 HTTP/2: "http2" directive. The directive enables HTTP/2 in the current server. The previous way to enable HTTP/2 via "listen ... http2" is now deprecated. The new approach allows to share HTTP/2 and HTTP/0.9-1.1 on the same port. For SSL connections, HTTP/2 is now selected by ALPN callback based on whether the protocol is enabled in the virtual server chosen by SNI. This however only works since OpenSSL 1.0.2h, where ALPN callback is invoked after SNI callback. For older versions of OpenSSL, HTTP/2 is enabled based on the default virtual server configuration. For plain TCP connections, HTTP/2 is now auto-detected by HTTP/2 preface, if HTTP/2 is enabled in the default virtual server. If preface is not matched, HTTP/0.9-1.1 is assumed. diff --git a/src/http/modules/ngx_http_ssl_module.c b/src/http/modules/ngx_http_ssl_module.c --- a/src/http/modules/ngx_http_ssl_module.c +++ b/src/http/modules/ngx_http_ssl_module.c @@ -427,6 +427,9 @@ ngx_http_ssl_alpn_select(ngx_ssl_conn_t #if (NGX_HTTP_V2 || NGX_HTTP_V3) ngx_http_connection_t *hc; #endif +#if (NGX_HTTP_V2) + ngx_http_v2_srv_conf_t *h2scf; +#endif #if (NGX_HTTP_V3) ngx_http_v3_srv_conf_t *h3scf; #endif @@ -448,12 +451,9 @@ ngx_http_ssl_alpn_select(ngx_ssl_conn_t hc = c->data; #endif -#if (NGX_HTTP_V2) - if (hc->addr_conf->http2) { - srv = (unsigned char *) NGX_HTTP_V2_ALPN_PROTO NGX_HTTP_ALPN_PROTOS; - srvlen = sizeof(NGX_HTTP_V2_ALPN_PROTO NGX_HTTP_ALPN_PROTOS) - 1; - } else -#endif + srv = (unsigned char *) NGX_HTTP_ALPN_PROTOS; + srvlen = sizeof(NGX_HTTP_ALPN_PROTOS) - 1; + #if (NGX_HTTP_V3) if (hc->addr_conf->quic) { @@ -479,10 +479,16 @@ ngx_http_ssl_alpn_select(ngx_ssl_conn_t } else #endif +#if (NGX_HTTP_V2) { - srv = (unsigned char *) NGX_HTTP_ALPN_PROTOS; - srvlen = sizeof(NGX_HTTP_ALPN_PROTOS) - 1; + h2scf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_v2_module); + + if (h2scf->enable || hc->addr_conf->http2) { + srv = (unsigned char *) NGX_HTTP_V2_ALPN_PROTO NGX_HTTP_ALPN_PROTOS; + srvlen = sizeof(NGX_HTTP_V2_ALPN_PROTO NGX_HTTP_ALPN_PROTOS) - 1; + } } +#endif if (SSL_select_next_proto((unsigned char **) out, outlen, srv, srvlen, in, inlen) diff --git a/src/http/ngx_http_core_module.c b/src/http/ngx_http_core_module.c --- a/src/http/ngx_http_core_module.c +++ b/src/http/ngx_http_core_module.c @@ -4179,6 +4179,11 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx if (ngx_strcmp(value[n].data, "http2") == 0) { #if (NGX_HTTP_V2) + ngx_conf_log_error(NGX_LOG_WARN, cf, 0, + "the \"listen ... http2\" directive " + "is deprecated, use " + "the \"http2\" directive instead"); + lsopt.http2 = 1; continue; #else diff --git a/src/http/ngx_http_request.c b/src/http/ngx_http_request.c --- a/src/http/ngx_http_request.c +++ b/src/http/ngx_http_request.c @@ -318,12 +318,6 @@ ngx_http_init_connection(ngx_connection_ rev->handler = ngx_http_wait_request_handler; c->write->handler = ngx_http_empty_handler; -#if (NGX_HTTP_V2) - if (hc->addr_conf->http2) { - rev->handler = ngx_http_v2_init; - } -#endif - #if (NGX_HTTP_V3) if (hc->addr_conf->quic) { ngx_http_v3_init_stream(c); @@ -383,6 +377,9 @@ ngx_http_wait_request_handler(ngx_event_ ngx_buf_t *b; ngx_connection_t *c; ngx_http_connection_t *hc; +#if (NGX_HTTP_V2) + ngx_http_v2_srv_conf_t *h2scf; +#endif ngx_http_core_srv_conf_t *cscf; c = rev->data; @@ -429,6 +426,8 @@ ngx_http_wait_request_handler(ngx_event_ b->end = b->last + size; } + size = b->end - b->last; + n = c->recv(c, b->last, size); if (n == NGX_AGAIN) { @@ -443,12 +442,16 @@ ngx_http_wait_request_handler(ngx_event_ return; } - /* - * We are trying to not hold c->buffer's memory for an idle connection. - */ - - if (ngx_pfree(c->pool, b->start) == NGX_OK) { - b->start = NULL; + if (b->pos == b->last) { + + /* + * We are trying to not hold c->buffer's memory for an + * idle connection. + */ + + if (ngx_pfree(c->pool, b->start) == NGX_OK) { + b->start = NULL; + } } return; @@ -489,10 +492,34 @@ ngx_http_wait_request_handler(ngx_event_ } } + ngx_reusable_connection(c, 0); + +#if (NGX_HTTP_V2) + + h2scf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_v2_module); + + if (!c->ssl && (h2scf->enable || hc->addr_conf->http2)) { + + size = ngx_min(sizeof(NGX_HTTP_V2_PREFACE) - 1, + (size_t) (b->last - b->pos)); + + if (ngx_memcmp(b->pos, NGX_HTTP_V2_PREFACE, size) == 0) { + + if (size == sizeof(NGX_HTTP_V2_PREFACE) - 1) { + ngx_http_v2_init(rev); + return; + } + + c->log->action = "waiting for request"; + ngx_post_event(rev, &ngx_posted_events); + return; + } + } + +#endif + c->log->action = "reading client request line"; - ngx_reusable_connection(c, 0); - c->data = ngx_http_create_request(c); if (c->data == NULL) { ngx_http_close_connection(c); @@ -808,13 +835,16 @@ ngx_http_ssl_handshake_handler(ngx_conne #if (NGX_HTTP_V2 \ && defined TLSEXT_TYPE_application_layer_protocol_negotiation) { - unsigned int len; - const unsigned char *data; - ngx_http_connection_t *hc; + unsigned int len; + const unsigned char *data; + ngx_http_connection_t *hc; + ngx_http_v2_srv_conf_t *h2scf; hc = c->data; - if (hc->addr_conf->http2) { + h2scf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_v2_module); + + if (h2scf->enable || hc->addr_conf->http2) { SSL_get0_alpn_selected(c->ssl->connection, &data, &len); diff --git a/src/http/v2/ngx_http_v2.c b/src/http/v2/ngx_http_v2.c --- a/src/http/v2/ngx_http_v2.c +++ b/src/http/v2/ngx_http_v2.c @@ -63,8 +63,6 @@ static void ngx_http_v2_handle_connectio static void ngx_http_v2_lingering_close(ngx_connection_t *c); static void ngx_http_v2_lingering_close_handler(ngx_event_t *rev); -static u_char *ngx_http_v2_state_proxy_protocol(ngx_http_v2_connection_t *h2c, - u_char *pos, u_char *end); static u_char *ngx_http_v2_state_preface(ngx_http_v2_connection_t *h2c, u_char *pos, u_char *end); static u_char *ngx_http_v2_state_preface_end(ngx_http_v2_connection_t *h2c, @@ -232,6 +230,7 @@ static ngx_http_v2_parse_header_t ngx_h void ngx_http_v2_init(ngx_event_t *rev) { + u_char *p, *end; ngx_connection_t *c; ngx_pool_cleanup_t *cln; ngx_http_connection_t *hc; @@ -314,8 +313,7 @@ ngx_http_v2_init(ngx_event_t *rev) return; } - h2c->state.handler = hc->proxy_protocol ? ngx_http_v2_state_proxy_protocol - : ngx_http_v2_state_preface; + h2c->state.handler = ngx_http_v2_state_preface; ngx_queue_init(&h2c->waiting); ngx_queue_init(&h2c->dependencies); @@ -333,7 +331,23 @@ ngx_http_v2_init(ngx_event_t *rev) } c->idle = 1; - ngx_reusable_connection(c, 0); + + if (c->buffer) { + p = c->buffer->pos; + end = c->buffer->last; + + do { + p = h2c->state.handler(h2c, p, end); + + if (p == NULL) { + return; + } + + } while (p != end); + + h2c->total_bytes += p - c->buffer->pos; + c->buffer->pos = p; + } ngx_http_v2_read_handler(rev); } @@ -847,31 +861,10 @@ ngx_http_v2_lingering_close_handler(ngx_ static u_char * -ngx_http_v2_state_proxy_protocol(ngx_http_v2_connection_t *h2c, u_char *pos, - u_char *end) -{ - ngx_log_t *log; - - log = h2c->connection->log; - log->action = "reading PROXY protocol"; - - pos = ngx_proxy_protocol_read(h2c->connection, pos, end); - - log->action = "processing HTTP/2 connection"; - - if (pos == NULL) { - return ngx_http_v2_connection_error(h2c, NGX_HTTP_V2_PROTOCOL_ERROR); - } - - return ngx_http_v2_state_preface(h2c, pos, end); -} - - -static u_char * ngx_http_v2_state_preface(ngx_http_v2_connection_t *h2c, u_char *pos, u_char *end) { - static const u_char preface[] = "PRI * HTTP/2.0\r\n"; + static const u_char preface[] = NGX_HTTP_V2_PREFACE_START; if ((size_t) (end - pos) < sizeof(preface) - 1) { return ngx_http_v2_state_save(h2c, pos, end, ngx_http_v2_state_preface); @@ -892,7 +885,7 @@ static u_char * ngx_http_v2_state_preface_end(ngx_http_v2_connection_t *h2c, u_char *pos, u_char *end) { - static const u_char preface[] = "\r\nSM\r\n\r\n"; + static const u_char preface[] = NGX_HTTP_V2_PREFACE_END; if ((size_t) (end - pos) < sizeof(preface) - 1) { return ngx_http_v2_state_save(h2c, pos, end, @@ -3946,10 +3939,22 @@ static void ngx_http_v2_run_request(ngx_http_request_t *r) { ngx_connection_t *fc; + ngx_http_v2_srv_conf_t *h2scf; ngx_http_v2_connection_t *h2c; fc = r->connection; + h2scf = ngx_http_get_module_srv_conf(r, ngx_http_v2_module); + + if (!h2scf->enable && !r->http_connection->addr_conf->http2) { + ngx_log_error(NGX_LOG_INFO, fc->log, 0, + "client attempted to request the server name " + "for which the negotiated protocol is disabled"); + + ngx_http_finalize_request(r, NGX_HTTP_MISDIRECTED_REQUEST); + goto failed; + } + if (ngx_http_v2_construct_request_line(r) != NGX_OK) { goto failed; } diff --git a/src/http/v2/ngx_http_v2.h b/src/http/v2/ngx_http_v2.h --- a/src/http/v2/ngx_http_v2.h +++ b/src/http/v2/ngx_http_v2.h @@ -64,6 +64,16 @@ typedef u_char *(*ngx_http_v2_handler_pt typedef struct { + ngx_flag_t enable; + size_t pool_size; + ngx_uint_t concurrent_streams; + ngx_uint_t concurrent_pushes; + size_t preread_size; + ngx_uint_t streams_index_mask; +} ngx_http_v2_srv_conf_t; + + +typedef struct { ngx_str_t name; ngx_str_t value; } ngx_http_v2_header_t; @@ -408,9 +418,17 @@ ngx_int_t ngx_http_v2_table_size(ngx_htt #define NGX_HTTP_V2_USER_AGENT_INDEX 58 #define NGX_HTTP_V2_VARY_INDEX 59 +#define NGX_HTTP_V2_PREFACE_START "PRI * HTTP/2.0\r\n" +#define NGX_HTTP_V2_PREFACE_END "\r\nSM\r\n\r\n" +#define NGX_HTTP_V2_PREFACE NGX_HTTP_V2_PREFACE_START \ + NGX_HTTP_V2_PREFACE_END + u_char *ngx_http_v2_string_encode(u_char *dst, u_char *src, size_t len, u_char *tmp, ngx_uint_t lower); +extern ngx_module_t ngx_http_v2_module; + + #endif /* _NGX_HTTP_V2_H_INCLUDED_ */ diff --git a/src/http/v2/ngx_http_v2_module.c b/src/http/v2/ngx_http_v2_module.c --- a/src/http/v2/ngx_http_v2_module.c +++ b/src/http/v2/ngx_http_v2_module.c @@ -75,6 +75,13 @@ static ngx_conf_post_t ngx_http_v2_chun static ngx_command_t ngx_http_v2_commands[] = { + { ngx_string("http2"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_CONF_FLAG, + ngx_conf_set_flag_slot, + NGX_HTTP_SRV_CONF_OFFSET, + offsetof(ngx_http_v2_srv_conf_t, enable), + NULL }, + { ngx_string("http2_recv_buffer_size"), NGX_HTTP_MAIN_CONF|NGX_CONF_TAKE1, ngx_conf_set_size_slot, @@ -314,6 +321,8 @@ ngx_http_v2_create_srv_conf(ngx_conf_t * return NULL; } + h2scf->enable = NGX_CONF_UNSET; + h2scf->pool_size = NGX_CONF_UNSET_SIZE; h2scf->concurrent_streams = NGX_CONF_UNSET_UINT; @@ -333,6 +342,8 @@ ngx_http_v2_merge_srv_conf(ngx_conf_t *c ngx_http_v2_srv_conf_t *prev = parent; ngx_http_v2_srv_conf_t *conf = child; + ngx_conf_merge_value(conf->enable, prev->enable, 0); + ngx_conf_merge_size_value(conf->pool_size, prev->pool_size, 4096); ngx_conf_merge_uint_value(conf->concurrent_streams, diff --git a/src/http/v2/ngx_http_v2_module.h b/src/http/v2/ngx_http_v2_module.h --- a/src/http/v2/ngx_http_v2_module.h +++ b/src/http/v2/ngx_http_v2_module.h @@ -21,15 +21,6 @@ typedef struct { typedef struct { - size_t pool_size; - ngx_uint_t concurrent_streams; - ngx_uint_t concurrent_pushes; - size_t preread_size; - ngx_uint_t streams_index_mask; -} ngx_http_v2_srv_conf_t; - - -typedef struct { size_t chunk_size; ngx_flag_t push_preload; @@ -39,7 +30,4 @@ typedef struct { } ngx_http_v2_loc_conf_t; -extern ngx_module_t ngx_http_v2_module; - - #endif /* _NGX_HTTP_V2_MODULE_H_INCLUDED_ */ From arut at nginx.com Tue Feb 7 14:50:32 2023 From: arut at nginx.com (=?iso-8859-1?q?Roman_Arutyunyan?=) Date: Tue, 07 Feb 2023 18:50:32 +0400 Subject: [PATCH 3 of 3] HTTP/3: trigger more compatibility errors for "listen quic" In-Reply-To: References: Message-ID: <09fd10e9b2e80c998987.1675781432@arut-laptop> # HG changeset patch # User Roman Arutyunyan # Date 1674732333 -14400 # Thu Jan 26 15:25:33 2023 +0400 # Branch quic # Node ID 09fd10e9b2e80c998987fc88922f3045c9cb3a3c # Parent 735f9e501922e4b0a1b20730d62bac35ea398336 HTTP/3: trigger more compatibility errors for "listen quic". Now "ssl", "proxy_protocol" and "http2" are not allowed with "quic" in "listen" directive. Previously, only "ssl" was not allowed. diff --git a/src/http/ngx_http_core_module.c b/src/http/ngx_http_core_module.c --- a/src/http/ngx_http_core_module.c +++ b/src/http/ngx_http_core_module.c @@ -4325,10 +4325,26 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx return NGX_CONF_ERROR; } -#if (NGX_HTTP_SSL && NGX_HTTP_V3) - if (lsopt.ssl && lsopt.quic) { - return "\"ssl\" parameter is incompatible with \"quic\""; - } +#if (NGX_HTTP_V3) + + if (lsopt.quic) { +#if (NGX_HTTP_SSL) + if (lsopt.ssl) { + return "\"ssl\" parameter is incompatible with \"quic\""; + } +#endif + +#if (NGX_HTTP_V2) + if (lsopt.http2) { + return "\"http2\" parameter is incompatible with \"quic\""; + } +#endif + + if (lsopt.proxy_protocol) { + return "\"proxy_protocol\" parameter is incompatible with \"quic\""; + } + } + #endif for (n = 0; n < u.naddrs; n++) { From xeioex at nginx.com Tue Feb 7 17:17:57 2023 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Tue, 07 Feb 2023 17:17:57 +0000 Subject: [njs] Removed tag 0.7.10 Message-ID: details: https://hg.nginx.org/njs/rev/3a1b46d51f04 branches: changeset: 2042:3a1b46d51f04 user: Dmitry Volyntsev date: Tue Feb 07 09:02:59 2023 -0800 description: Removed tag 0.7.10 diffstat: .hgtags | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diffs (9 lines): diff -r d20fb5d9d28f -r 3a1b46d51f04 .hgtags --- a/.hgtags Mon Feb 06 10:50:08 2023 -0800 +++ b/.hgtags Tue Feb 07 09:02:59 2023 -0800 @@ -56,3 +56,5 @@ 1592d46d9076aa832b2d37d50b90f5edfca67030 3308415d7de83c3c0c7c65405bec4836685a71de 0.7.8 5f705230a62ccae8718ef9b7a85b29dc569d8b5e 0.7.9 60a3ffe734b67add80bbc58448c3f6c9036fe844 0.7.10 +0000000000000000000000000000000000000000 0.7.10 +0000000000000000000000000000000000000000 0.7.10 From xeioex at nginx.com Tue Feb 7 17:17:59 2023 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Tue, 07 Feb 2023 17:17:59 +0000 Subject: [njs] Added tag 0.7.10 for changeset 3a1b46d51f04 Message-ID: details: https://hg.nginx.org/njs/rev/083feee041d0 branches: changeset: 2043:083feee041d0 user: Dmitry Volyntsev date: Tue Feb 07 09:03:04 2023 -0800 description: Added tag 0.7.10 for changeset 3a1b46d51f04 diffstat: .hgtags | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diffs (9 lines): diff -r 3a1b46d51f04 -r 083feee041d0 .hgtags --- a/.hgtags Tue Feb 07 09:02:59 2023 -0800 +++ b/.hgtags Tue Feb 07 09:03:04 2023 -0800 @@ -58,3 +58,5 @@ 5f705230a62ccae8718ef9b7a85b29dc569d8b5e 60a3ffe734b67add80bbc58448c3f6c9036fe844 0.7.10 0000000000000000000000000000000000000000 0.7.10 0000000000000000000000000000000000000000 0.7.10 +0000000000000000000000000000000000000000 0.7.10 +3a1b46d51f040f5e7b9b81c3b2b312a2d272f0a3 0.7.10 From pluknet at nginx.com Wed Feb 8 12:28:10 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Wed, 8 Feb 2023 16:28:10 +0400 Subject: [PATCH] QUIC: OpenSSL compatibility layer In-Reply-To: <20230206142701.pxzaq3irckqads7y@N00W24XTQX> References: <64a365dcb52503e91d91.1671606452@arut-laptop> <20230109111316.yfqj24unoubibihz@N00W24XTQX> <20230202183522.o4w5xbjvegbchts4@Y9MQ9X2QVV> <20230206142701.pxzaq3irckqads7y@N00W24XTQX> Message-ID: <20230208122810.tsxynyhlqgt2ubsm@Y9MQ9X2QVV> On Mon, Feb 06, 2023 at 06:27:01PM +0400, Roman Arutyunyan wrote: > # HG changeset patch > # User Roman Arutyunyan > # Date 1675427689 -14400 > # Fri Feb 03 16:34:49 2023 +0400 > # Branch quic > # Node ID 9cf1fc42260e7e0e19fe5707f1b054d6499a4157 > # Parent def8e398d7c50131f8dac844814fff729da5c86c > QUIC: OpenSSL compatibility layer. > > The change allows to compile QUIC with OpenSSL which lacks BoringSSL QUIC API. > > This implementation does not support 0-RTT. > > diff --git a/README b/README > --- a/README > +++ b/README > @@ -53,7 +53,7 @@ 1. Introduction > > 2. Installing > > - A library that provides QUIC support is required to build nginx, there > + A library that provides QUIC support is recommended to build nginx, there > are several of those available on the market: > + BoringSSL [4] > + LibreSSL [5] > @@ -85,6 +85,10 @@ 2. Installing > --with-cc-opt="-I../libressl/build/include" \ > --with-ld-opt="-L../libressl/build/lib" > > + Alternatively, nginx can be configured with OpenSSL compatibility > + layer, which emulates BoringSSL QUIC API for OpenSSL. This mode is > + enabled by default if native QUIC support is not detected. > + > When configuring nginx, it's possible to enable QUIC and HTTP/3 > using the following new configuration options: > > diff --git a/auto/lib/openssl/conf b/auto/lib/openssl/conf > --- a/auto/lib/openssl/conf > +++ b/auto/lib/openssl/conf > @@ -10,6 +10,7 @@ if [ $OPENSSL != NONE ]; then > > if [ $USE_OPENSSL_QUIC = YES ]; then > have=NGX_QUIC . auto/have > + have=NGX_QUIC_OPENSSL_COMPAT . auto/have > fi > > case "$CC" in > @@ -124,6 +125,43 @@ else > CORE_INCS="$CORE_INCS $ngx_feature_path" > CORE_LIBS="$CORE_LIBS $ngx_feature_libs" > OPENSSL=YES Given that you moved these tests under OpenSSL tests, some things can be simplified, such as ngx_feature_run and ngx_feature_incs. > + > + if [ $USE_OPENSSL_QUIC = YES ]; then > + > + ngx_feature="OpenSSL QUIC support" > + ngx_feature_name="NGX_OPENSSL_QUIC" This seems to revive NGX_OPENSSL_QUIC unused since 7603284f7af5. It could be replaced with NGX_QUIC feature name, but this makes hard to co-exist with NGX_QUIC_OPENSSL_COMPAT feature name below. So the simplifiest is just to remove this line. > + ngx_feature_run=no > + ngx_feature_incs="#include " > + ngx_feature_test="SSL_set_quic_method(NULL, NULL)" > + . auto/feature > + > + if [ $ngx_found = no ]; then > + > + ngx_feature="OpenSSL QUIC compatibility" > + ngx_feature_name="NGX_QUIC_OPENSSL_COMPAT" > + ngx_feature_run=no > + ngx_feature_incs="#include " > + ngx_feature_test=" > + (void) TLS1_3_VERSION; Note that SSL_CTX_add_custom_ext() seems to be enough there, because this API was added in OpenSSL 1.1.1 only, which has TLSv1.3 support. This makes testing TLS1_3_VERSION redundant. Though, I don't insist against such explicit test. So this can be simplified to: diff --git a/auto/lib/openssl/conf b/auto/lib/openssl/conf --- a/auto/lib/openssl/conf +++ b/auto/lib/openssl/conf @@ -129,9 +129,6 @@ else if [ $USE_OPENSSL_QUIC = YES ]; then ngx_feature="OpenSSL QUIC support" - ngx_feature_name="NGX_OPENSSL_QUIC" - ngx_feature_run=no - ngx_feature_incs="#include " ngx_feature_test="SSL_set_quic_method(NULL, NULL)" . auto/feature @@ -139,10 +136,7 @@ else ngx_feature="OpenSSL QUIC compatibility" ngx_feature_name="NGX_QUIC_OPENSSL_COMPAT" - ngx_feature_run=no - ngx_feature_incs="#include " ngx_feature_test=" - (void) TLS1_3_VERSION; SSL_CTX_add_custom_ext(NULL, 0, 0, NULL, NULL, NULL, NULL, NULL)" . auto/feature > + SSL_CTX_add_custom_ext(NULL, 0, 0, NULL, NULL, > + NULL, NULL, NULL)" > + . auto/feature > + fi > + > + if [ $ngx_found = no ]; then > +cat << END > + > +$0: error: certain modules require OpenSSL QUIC support. > +You can either do not enable the modules, or install the OpenSSL library with > +QUIC support into the system, or build the OpenSSL library with QUIC support > +statically from the source with nginx by using --with-openssl= option. > + > +END > + exit 1 > + fi > + > + have=NGX_QUIC . auto/have > + fi > fi > fi > > @@ -139,29 +177,4 @@ with nginx by using --with-openssl= END > exit 1 > fi > - > - if [ $USE_OPENSSL_QUIC = YES ]; then > - > - ngx_feature="OpenSSL QUIC support" > - ngx_feature_name="NGX_QUIC" > - ngx_feature_run=no > - ngx_feature_incs="#include " > - ngx_feature_path= > - ngx_feature_libs="-lssl -lcrypto $NGX_LIBDL $NGX_LIBPTHREAD" > - ngx_feature_test="SSL_set_quic_method(NULL, NULL)" > - . auto/feature > - > - if [ $ngx_found = no ]; then > - > -cat << END > - > -$0: error: certain modules require OpenSSL QUIC support. > -You can either do not enable the modules, or install the OpenSSL library with > -QUIC support into the system, or build the OpenSSL library with QUIC support > -statically from the source with nginx by using --with-openssl= option. > - > -END > - exit 1 > - fi > - fi N.B. there is an empty line in the default branch > fi > diff --git a/auto/modules b/auto/modules > --- a/auto/modules > +++ b/auto/modules > @@ -1342,7 +1342,8 @@ if [ $USE_OPENSSL_QUIC = YES ]; then > src/event/quic/ngx_event_quic_tokens.h \ > src/event/quic/ngx_event_quic_ack.h \ > src/event/quic/ngx_event_quic_output.h \ > - src/event/quic/ngx_event_quic_socket.h" > + src/event/quic/ngx_event_quic_socket.h \ > + src/event/quic/ngx_event_quic_openssl_compat.h" > ngx_module_srcs="src/event/quic/ngx_event_quic.c \ > src/event/quic/ngx_event_quic_udp.c \ > src/event/quic/ngx_event_quic_transport.c \ > @@ -1355,7 +1356,8 @@ if [ $USE_OPENSSL_QUIC = YES ]; then > src/event/quic/ngx_event_quic_tokens.c \ > src/event/quic/ngx_event_quic_ack.c \ > src/event/quic/ngx_event_quic_output.c \ > - src/event/quic/ngx_event_quic_socket.c" > + src/event/quic/ngx_event_quic_socket.c \ > + src/event/quic/ngx_event_quic_openssl_compat.c" > > ngx_module_libs= > ngx_module_link=YES > diff --git a/src/event/ngx_event_openssl.c b/src/event/ngx_event_openssl.c > --- a/src/event/ngx_event_openssl.c > +++ b/src/event/ngx_event_openssl.c > @@ -9,6 +9,10 @@ > #include > #include > > +#if (NGX_QUIC_OPENSSL_COMPAT) > +#include > +#endif > + This is unused now and can be removed. [..] The rest of the patch looks good. From arut at nginx.com Wed Feb 8 14:41:38 2023 From: arut at nginx.com (Roman Arutyunyan) Date: Wed, 8 Feb 2023 18:41:38 +0400 Subject: [PATCH] QUIC: OpenSSL compatibility layer In-Reply-To: <20230208122810.tsxynyhlqgt2ubsm@Y9MQ9X2QVV> References: <64a365dcb52503e91d91.1671606452@arut-laptop> <20230109111316.yfqj24unoubibihz@N00W24XTQX> <20230202183522.o4w5xbjvegbchts4@Y9MQ9X2QVV> <20230206142701.pxzaq3irckqads7y@N00W24XTQX> <20230208122810.tsxynyhlqgt2ubsm@Y9MQ9X2QVV> Message-ID: <20230208144138.u7pozeds34nqr6mi@N00W24XTQX> Hi, On Wed, Feb 08, 2023 at 04:28:10PM +0400, Sergey Kandaurov wrote: > On Mon, Feb 06, 2023 at 06:27:01PM +0400, Roman Arutyunyan wrote: > > # HG changeset patch > > # User Roman Arutyunyan > > # Date 1675427689 -14400 > > # Fri Feb 03 16:34:49 2023 +0400 > > # Branch quic > > # Node ID 9cf1fc42260e7e0e19fe5707f1b054d6499a4157 > > # Parent def8e398d7c50131f8dac844814fff729da5c86c > > QUIC: OpenSSL compatibility layer. > > > > The change allows to compile QUIC with OpenSSL which lacks BoringSSL QUIC API. > > > > This implementation does not support 0-RTT. > > > > diff --git a/README b/README > > --- a/README > > +++ b/README > > @@ -53,7 +53,7 @@ 1. Introduction > > > > 2. Installing > > > > - A library that provides QUIC support is required to build nginx, there > > + A library that provides QUIC support is recommended to build nginx, there > > are several of those available on the market: > > + BoringSSL [4] > > + LibreSSL [5] > > @@ -85,6 +85,10 @@ 2. Installing > > --with-cc-opt="-I../libressl/build/include" \ > > --with-ld-opt="-L../libressl/build/lib" > > > > + Alternatively, nginx can be configured with OpenSSL compatibility > > + layer, which emulates BoringSSL QUIC API for OpenSSL. This mode is > > + enabled by default if native QUIC support is not detected. > > + > > When configuring nginx, it's possible to enable QUIC and HTTP/3 > > using the following new configuration options: > > > > diff --git a/auto/lib/openssl/conf b/auto/lib/openssl/conf > > --- a/auto/lib/openssl/conf > > +++ b/auto/lib/openssl/conf > > @@ -10,6 +10,7 @@ if [ $OPENSSL != NONE ]; then > > > > if [ $USE_OPENSSL_QUIC = YES ]; then > > have=NGX_QUIC . auto/have > > + have=NGX_QUIC_OPENSSL_COMPAT . auto/have > > fi > > > > case "$CC" in > > @@ -124,6 +125,43 @@ else > > CORE_INCS="$CORE_INCS $ngx_feature_path" > > CORE_LIBS="$CORE_LIBS $ngx_feature_libs" > > OPENSSL=YES > > Given that you moved these tests under OpenSSL tests, some things > can be simplified, such as ngx_feature_run and ngx_feature_incs. Sure, thanks. > > + > > + if [ $USE_OPENSSL_QUIC = YES ]; then > > + > > + ngx_feature="OpenSSL QUIC support" > > + ngx_feature_name="NGX_OPENSSL_QUIC" > > This seems to revive NGX_OPENSSL_QUIC unused since 7603284f7af5. > It could be replaced with NGX_QUIC feature name, but this makes > hard to co-exist with NGX_QUIC_OPENSSL_COMPAT feature name below. > So the simplifiest is just to remove this line. FTR: discussed this with Sergey. We decided to use NGX_QUIC both here and in the next feature test. NGX_QUIC_OPENSSL_COMPAT now is defined explicitly, while explicit definition of NGX_QUIC is removed. > > + ngx_feature_run=no > > + ngx_feature_incs="#include " > > + ngx_feature_test="SSL_set_quic_method(NULL, NULL)" > > + . auto/feature > > + > > + if [ $ngx_found = no ]; then > > + > > + ngx_feature="OpenSSL QUIC compatibility" > > + ngx_feature_name="NGX_QUIC_OPENSSL_COMPAT" > > + ngx_feature_run=no > > + ngx_feature_incs="#include " > > + ngx_feature_test=" > > + (void) TLS1_3_VERSION; > > Note that SSL_CTX_add_custom_ext() seems to be enough there, because > this API was added in OpenSSL 1.1.1 only, which has TLSv1.3 support. > This makes testing TLS1_3_VERSION redundant. > Though, I don't insist against such explicit test. OK. > So this can be simplified to: > > diff --git a/auto/lib/openssl/conf b/auto/lib/openssl/conf > --- a/auto/lib/openssl/conf > +++ b/auto/lib/openssl/conf > @@ -129,9 +129,6 @@ else > if [ $USE_OPENSSL_QUIC = YES ]; then > > ngx_feature="OpenSSL QUIC support" > - ngx_feature_name="NGX_OPENSSL_QUIC" > - ngx_feature_run=no > - ngx_feature_incs="#include " > ngx_feature_test="SSL_set_quic_method(NULL, NULL)" > . auto/feature > > @@ -139,10 +136,7 @@ else > > ngx_feature="OpenSSL QUIC compatibility" > ngx_feature_name="NGX_QUIC_OPENSSL_COMPAT" > - ngx_feature_run=no > - ngx_feature_incs="#include " > ngx_feature_test=" > - (void) TLS1_3_VERSION; > SSL_CTX_add_custom_ext(NULL, 0, 0, NULL, NULL, > NULL, NULL, NULL)" > . auto/feature > > > + SSL_CTX_add_custom_ext(NULL, 0, 0, NULL, NULL, > > + NULL, NULL, NULL)" > > + . auto/feature > > + fi > > + > > + if [ $ngx_found = no ]; then > > +cat << END > > + > > +$0: error: certain modules require OpenSSL QUIC support. > > +You can either do not enable the modules, or install the OpenSSL library with > > +QUIC support into the system, or build the OpenSSL library with QUIC support > > +statically from the source with nginx by using --with-openssl= option. > > + > > +END > > + exit 1 > > + fi > > + > > + have=NGX_QUIC . auto/have > > + fi > > fi > > fi > > > > @@ -139,29 +177,4 @@ with nginx by using --with-openssl= > END > > exit 1 > > fi > > - > > - if [ $USE_OPENSSL_QUIC = YES ]; then > > - > > - ngx_feature="OpenSSL QUIC support" > > - ngx_feature_name="NGX_QUIC" > > - ngx_feature_run=no > > - ngx_feature_incs="#include " > > - ngx_feature_path= > > - ngx_feature_libs="-lssl -lcrypto $NGX_LIBDL $NGX_LIBPTHREAD" > > - ngx_feature_test="SSL_set_quic_method(NULL, NULL)" > > - . auto/feature > > - > > - if [ $ngx_found = no ]; then > > - > > -cat << END > > - > > -$0: error: certain modules require OpenSSL QUIC support. > > -You can either do not enable the modules, or install the OpenSSL library with > > -QUIC support into the system, or build the OpenSSL library with QUIC support > > -statically from the source with nginx by using --with-openssl= option. > > - > > -END > > - exit 1 > > - fi > > - fi > > N.B. there is an empty line in the default branch > > > fi > > diff --git a/auto/modules b/auto/modules > > --- a/auto/modules > > +++ b/auto/modules > > @@ -1342,7 +1342,8 @@ if [ $USE_OPENSSL_QUIC = YES ]; then > > src/event/quic/ngx_event_quic_tokens.h \ > > src/event/quic/ngx_event_quic_ack.h \ > > src/event/quic/ngx_event_quic_output.h \ > > - src/event/quic/ngx_event_quic_socket.h" > > + src/event/quic/ngx_event_quic_socket.h \ > > + src/event/quic/ngx_event_quic_openssl_compat.h" > > ngx_module_srcs="src/event/quic/ngx_event_quic.c \ > > src/event/quic/ngx_event_quic_udp.c \ > > src/event/quic/ngx_event_quic_transport.c \ > > @@ -1355,7 +1356,8 @@ if [ $USE_OPENSSL_QUIC = YES ]; then > > src/event/quic/ngx_event_quic_tokens.c \ > > src/event/quic/ngx_event_quic_ack.c \ > > src/event/quic/ngx_event_quic_output.c \ > > - src/event/quic/ngx_event_quic_socket.c" > > + src/event/quic/ngx_event_quic_socket.c \ > > + src/event/quic/ngx_event_quic_openssl_compat.c" > > > > ngx_module_libs= > > ngx_module_link=YES > > diff --git a/src/event/ngx_event_openssl.c b/src/event/ngx_event_openssl.c > > --- a/src/event/ngx_event_openssl.c > > +++ b/src/event/ngx_event_openssl.c > > @@ -9,6 +9,10 @@ > > #include > > #include > > > > +#if (NGX_QUIC_OPENSSL_COMPAT) > > +#include > > +#endif > > + > > This is unused now and can be removed. > > [..] Removed, thanks. > > The rest of the patch looks good. > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > https://mailman.nginx.org/mailman/listinfo/nginx-devel -- Roman Arutyunyan -------------- next part -------------- # HG changeset patch # User Roman Arutyunyan # Date 1675867049 -14400 # Wed Feb 08 18:37:29 2023 +0400 # Branch quic # Node ID a3142c8833f5bf1186599e7938141f5062fac4a2 # Parent 3c33d39a51d334d99fcc7d2b45e8d8190c431492 QUIC: OpenSSL compatibility layer. The change allows to compile QUIC with OpenSSL which lacks BoringSSL QUIC API. This implementation does not support 0-RTT. diff --git a/README b/README --- a/README +++ b/README @@ -62,12 +62,16 @@ 2. Building from sources --with-http_v3_module - enable QUIC and HTTP/3 --with-stream_quic_module - enable QUIC in Stream - A library that provides QUIC support is required to build nginx, there + A library that provides QUIC support is recommended to build nginx, there are several of those available on the market: + BoringSSL [4] + LibreSSL [5] + QuicTLS [6] + Alternatively, nginx can be configured with OpenSSL compatibility + layer, which emulates BoringSSL QUIC API for OpenSSL. This mode is + enabled by default if native QUIC support is not detected. + Clone the NGINX QUIC repository $ hg clone -b quic https://hg.nginx.org/nginx-quic diff --git a/auto/lib/openssl/conf b/auto/lib/openssl/conf --- a/auto/lib/openssl/conf +++ b/auto/lib/openssl/conf @@ -10,6 +10,7 @@ if [ $OPENSSL != NONE ]; then if [ $USE_OPENSSL_QUIC = YES ]; then have=NGX_QUIC . auto/have + have=NGX_QUIC_OPENSSL_COMPAT . auto/have fi case "$CC" in @@ -124,6 +125,35 @@ else CORE_INCS="$CORE_INCS $ngx_feature_path" CORE_LIBS="$CORE_LIBS $ngx_feature_libs" OPENSSL=YES + + if [ $USE_OPENSSL_QUIC = YES ]; then + + ngx_feature="OpenSSL QUIC support" + ngx_feature_name="NGX_QUIC" + ngx_feature_test="SSL_set_quic_method(NULL, NULL)" + . auto/feature + + if [ $ngx_found = no ]; then + have=NGX_QUIC_OPENSSL_COMPAT . auto/have + + ngx_feature="OpenSSL QUIC compatibility" + ngx_feature_test="SSL_CTX_add_custom_ext(NULL, 0, 0, + NULL, NULL, NULL, NULL, NULL)" + . auto/feature + fi + + if [ $ngx_found = no ]; then +cat << END + +$0: error: certain modules require OpenSSL QUIC support. +You can either do not enable the modules, or install the OpenSSL library with +QUIC support into the system, or build the OpenSSL library with QUIC support +statically from the source with nginx by using --with-openssl= option. + +END + exit 1 + fi + fi fi fi @@ -140,28 +170,4 @@ END exit 1 fi - if [ $USE_OPENSSL_QUIC = YES ]; then - - ngx_feature="OpenSSL QUIC support" - ngx_feature_name="NGX_QUIC" - ngx_feature_run=no - ngx_feature_incs="#include " - ngx_feature_path= - ngx_feature_libs="-lssl -lcrypto $NGX_LIBDL $NGX_LIBPTHREAD" - ngx_feature_test="SSL_set_quic_method(NULL, NULL)" - . auto/feature - - if [ $ngx_found = no ]; then - -cat << END - -$0: error: certain modules require OpenSSL QUIC support. -You can either do not enable the modules, or install the OpenSSL library with -QUIC support into the system, or build the OpenSSL library with QUIC support -statically from the source with nginx by using --with-openssl= option. - -END - exit 1 - fi - fi fi diff --git a/auto/modules b/auto/modules --- a/auto/modules +++ b/auto/modules @@ -1342,7 +1342,8 @@ if [ $USE_OPENSSL_QUIC = YES ]; then src/event/quic/ngx_event_quic_tokens.h \ src/event/quic/ngx_event_quic_ack.h \ src/event/quic/ngx_event_quic_output.h \ - src/event/quic/ngx_event_quic_socket.h" + src/event/quic/ngx_event_quic_socket.h \ + src/event/quic/ngx_event_quic_openssl_compat.h" ngx_module_srcs="src/event/quic/ngx_event_quic.c \ src/event/quic/ngx_event_quic_udp.c \ src/event/quic/ngx_event_quic_transport.c \ @@ -1355,7 +1356,8 @@ if [ $USE_OPENSSL_QUIC = YES ]; then src/event/quic/ngx_event_quic_tokens.c \ src/event/quic/ngx_event_quic_ack.c \ src/event/quic/ngx_event_quic_output.c \ - src/event/quic/ngx_event_quic_socket.c" + src/event/quic/ngx_event_quic_socket.c \ + src/event/quic/ngx_event_quic_openssl_compat.c" ngx_module_libs= ngx_module_link=YES diff --git a/src/event/quic/ngx_event_quic_connection.h b/src/event/quic/ngx_event_quic_connection.h --- a/src/event/quic/ngx_event_quic_connection.h +++ b/src/event/quic/ngx_event_quic_connection.h @@ -25,6 +25,9 @@ typedef struct ngx_quic_socket_s ng typedef struct ngx_quic_path_s ngx_quic_path_t; typedef struct ngx_quic_keys_s ngx_quic_keys_t; +#if (NGX_QUIC_OPENSSL_COMPAT) +#include +#endif #include #include #include @@ -236,6 +239,10 @@ struct ngx_quic_connection_s { ngx_uint_t nshadowbufs; #endif +#if (NGX_QUIC_OPENSSL_COMPAT) + ngx_quic_compat_t *compat; +#endif + ngx_quic_streams_t streams; ngx_quic_congestion_t congestion; diff --git a/src/event/quic/ngx_event_quic_openssl_compat.c b/src/event/quic/ngx_event_quic_openssl_compat.c new file mode 100644 --- /dev/null +++ b/src/event/quic/ngx_event_quic_openssl_compat.c @@ -0,0 +1,646 @@ + +/* + * Copyright (C) Nginx, Inc. + */ + + +#include +#include +#include +#include + + +#if (NGX_QUIC_OPENSSL_COMPAT) + +#define NGX_QUIC_COMPAT_RECORD_SIZE 1024 + +#define NGX_QUIC_COMPAT_SSL_TP_EXT 0x39 + +#define NGX_QUIC_COMPAT_CLIENT_HANDSHAKE "CLIENT_HANDSHAKE_TRAFFIC_SECRET" +#define NGX_QUIC_COMPAT_SERVER_HANDSHAKE "SERVER_HANDSHAKE_TRAFFIC_SECRET" +#define NGX_QUIC_COMPAT_CLIENT_APPLICATION "CLIENT_TRAFFIC_SECRET_0" +#define NGX_QUIC_COMPAT_SERVER_APPLICATION "SERVER_TRAFFIC_SECRET_0" + + +typedef struct { + ngx_quic_secret_t secret; + ngx_uint_t cipher; +} ngx_quic_compat_keys_t; + + +typedef struct { + ngx_log_t *log; + + u_char type; + ngx_str_t payload; + uint64_t number; + ngx_quic_compat_keys_t *keys; + + enum ssl_encryption_level_t level; +} ngx_quic_compat_record_t; + + +struct ngx_quic_compat_s { + const SSL_QUIC_METHOD *method; + + enum ssl_encryption_level_t write_level; + enum ssl_encryption_level_t read_level; + + uint64_t read_record; + ngx_quic_compat_keys_t keys; + + ngx_str_t tp; + ngx_str_t ctp; +}; + + +static void ngx_quic_compat_keylog_callback(const SSL *ssl, const char *line); +static ngx_int_t ngx_quic_compat_set_encryption_secret(ngx_log_t *log, + ngx_quic_compat_keys_t *keys, enum ssl_encryption_level_t level, + const SSL_CIPHER *cipher, const uint8_t *secret, size_t secret_len); +static int ngx_quic_compat_add_transport_params_callback(SSL *ssl, + unsigned int ext_type, unsigned int context, const unsigned char **out, + size_t *outlen, X509 *x, size_t chainidx, int *al, void *add_arg); +static int ngx_quic_compat_parse_transport_params_callback(SSL *ssl, + unsigned int ext_type, unsigned int context, const unsigned char *in, + size_t inlen, X509 *x, size_t chainidx, int *al, void *parse_arg); +static void ngx_quic_compat_message_callback(int write_p, int version, + int content_type, const void *buf, size_t len, SSL *ssl, void *arg); +static size_t ngx_quic_compat_create_header(ngx_quic_compat_record_t *rec, + u_char *out, ngx_uint_t plain); +static ngx_int_t ngx_quic_compat_create_record(ngx_quic_compat_record_t *rec, + ngx_str_t *res); + + +ngx_int_t +ngx_quic_compat_init(ngx_conf_t *cf, SSL_CTX *ctx) +{ + SSL_CTX_set_keylog_callback(ctx, ngx_quic_compat_keylog_callback); + + if (SSL_CTX_has_client_custom_ext(ctx, NGX_QUIC_COMPAT_SSL_TP_EXT)) { + return NGX_OK; + } + + if (SSL_CTX_add_custom_ext(ctx, NGX_QUIC_COMPAT_SSL_TP_EXT, + SSL_EXT_CLIENT_HELLO + |SSL_EXT_TLS1_3_ENCRYPTED_EXTENSIONS, + ngx_quic_compat_add_transport_params_callback, + NULL, + NULL, + ngx_quic_compat_parse_transport_params_callback, + NULL) + == 0) + { + ngx_log_error(NGX_LOG_EMERG, cf->log, 0, + "SSL_CTX_add_custom_ext() failed"); + return NGX_ERROR; + } + + return NGX_OK; +} + + +static void +ngx_quic_compat_keylog_callback(const SSL *ssl, const char *line) +{ + u_char ch, *p, *start, value; + size_t n; + ngx_uint_t write; + const SSL_CIPHER *cipher; + ngx_quic_compat_t *com; + ngx_connection_t *c; + ngx_quic_connection_t *qc; + enum ssl_encryption_level_t level; + u_char secret[EVP_MAX_MD_SIZE]; + + c = ngx_ssl_get_connection(ssl); + if (c->type != SOCK_DGRAM) { + return; + } + + p = (u_char *) line; + + for (start = p; *p && *p != ' '; p++); + + n = p - start; + + ngx_log_debug2(NGX_LOG_DEBUG_EVENT, c->log, 0, + "quic compat secret %*s", n, start); + + if (n == sizeof(NGX_QUIC_COMPAT_CLIENT_HANDSHAKE) - 1 + && ngx_strncmp(start, NGX_QUIC_COMPAT_CLIENT_HANDSHAKE, n) == 0) + { + level = ssl_encryption_handshake; + write = 0; + + } else if (n == sizeof(NGX_QUIC_COMPAT_SERVER_HANDSHAKE) - 1 + && ngx_strncmp(start, NGX_QUIC_COMPAT_SERVER_HANDSHAKE, n) == 0) + { + level = ssl_encryption_handshake; + write = 1; + + } else if (n == sizeof(NGX_QUIC_COMPAT_CLIENT_APPLICATION) - 1 + && ngx_strncmp(start, NGX_QUIC_COMPAT_CLIENT_APPLICATION, n) + == 0) + { + level = ssl_encryption_application; + write = 0; + + } else if (n == sizeof(NGX_QUIC_COMPAT_SERVER_APPLICATION) - 1 + && ngx_strncmp(start, NGX_QUIC_COMPAT_SERVER_APPLICATION, n) + == 0) + { + level = ssl_encryption_application; + write = 1; + + } else { + return; + } + + if (*p++ == '\0') { + return; + } + + for ( /* void */ ; *p && *p != ' '; p++); + + if (*p++ == '\0') { + return; + } + + for (n = 0, start = p; *p; p++) { + ch = *p; + + if (ch >= '0' && ch <= '9') { + value = ch - '0'; + goto next; + } + + ch = (u_char) (ch | 0x20); + + if (ch >= 'a' && ch <= 'f') { + value = ch - 'a' + 10; + goto next; + } + + ngx_log_error(NGX_LOG_EMERG, c->log, 0, + "invalid OpenSSL QUIC secret format"); + + return; + + next: + + if ((p - start) % 2) { + secret[n++] += value; + + } else { + if (n >= EVP_MAX_MD_SIZE) { + ngx_log_error(NGX_LOG_EMERG, c->log, 0, + "too big OpenSSL QUIC secret"); + return; + } + + secret[n] = (value << 4); + } + } + + qc = ngx_quic_get_connection(c); + com = qc->compat; + cipher = SSL_get_current_cipher(ssl); + + if (write) { + com->method->set_write_secret((SSL *) ssl, level, cipher, secret, n); + com->write_level = level; + + } else { + com->method->set_read_secret((SSL *) ssl, level, cipher, secret, n); + com->read_level = level; + com->read_record = 0; + + (void) ngx_quic_compat_set_encryption_secret(c->log, &com->keys, level, + cipher, secret, n); + } +} + + +static ngx_int_t +ngx_quic_compat_set_encryption_secret(ngx_log_t *log, + ngx_quic_compat_keys_t *keys, enum ssl_encryption_level_t level, + const SSL_CIPHER *cipher, const uint8_t *secret, size_t secret_len) +{ + ngx_int_t key_len; + ngx_str_t secret_str; + ngx_uint_t i; + ngx_quic_hkdf_t seq[2]; + ngx_quic_secret_t *peer_secret; + ngx_quic_ciphers_t ciphers; + + peer_secret = &keys->secret; + + keys->cipher = SSL_CIPHER_get_id(cipher); + + key_len = ngx_quic_ciphers(keys->cipher, &ciphers, level); + + if (key_len == NGX_ERROR) { + ngx_ssl_error(NGX_LOG_INFO, log, 0, "unexpected cipher"); + return NGX_ERROR; + } + + if (sizeof(peer_secret->secret.data) < secret_len) { + ngx_log_error(NGX_LOG_ALERT, log, 0, + "unexpected secret len: %uz", secret_len); + return NGX_ERROR; + } + + peer_secret->secret.len = secret_len; + ngx_memcpy(peer_secret->secret.data, secret, secret_len); + + peer_secret->key.len = key_len; + peer_secret->iv.len = NGX_QUIC_IV_LEN; + + secret_str.len = secret_len; + secret_str.data = (u_char *) secret; + + ngx_quic_hkdf_set(&seq[0], "tls13 key", &peer_secret->key, &secret_str); + ngx_quic_hkdf_set(&seq[1], "tls13 iv", &peer_secret->iv, &secret_str); + + for (i = 0; i < (sizeof(seq) / sizeof(seq[0])); i++) { + if (ngx_quic_hkdf_expand(&seq[i], ciphers.d, log) != NGX_OK) { + return NGX_ERROR; + } + } + + return NGX_OK; +} + + +static int +ngx_quic_compat_add_transport_params_callback(SSL *ssl, unsigned int ext_type, + unsigned int context, const unsigned char **out, size_t *outlen, X509 *x, + size_t chainidx, int *al, void *add_arg) +{ + ngx_connection_t *c; + ngx_quic_compat_t *com; + ngx_quic_connection_t *qc; + + c = ngx_ssl_get_connection(ssl); + if (c->type != SOCK_DGRAM) { + return 0; + } + + ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0, + "quic compat add transport params"); + + qc = ngx_quic_get_connection(c); + com = qc->compat; + + *out = com->tp.data; + *outlen = com->tp.len; + + return 1; +} + + +static int +ngx_quic_compat_parse_transport_params_callback(SSL *ssl, unsigned int ext_type, + unsigned int context, const unsigned char *in, size_t inlen, X509 *x, + size_t chainidx, int *al, void *parse_arg) +{ + u_char *p; + ngx_connection_t *c; + ngx_quic_compat_t *com; + ngx_quic_connection_t *qc; + + c = ngx_ssl_get_connection(ssl); + if (c->type != SOCK_DGRAM) { + return 0; + } + + ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0, + "quic compat parse transport params"); + + qc = ngx_quic_get_connection(c); + com = qc->compat; + + p = ngx_pnalloc(c->pool, inlen); + if (p == NULL) { + return 0; + } + + ngx_memcpy(p, in, inlen); + + com->ctp.data = p; + com->ctp.len = inlen; + + return 1; +} + + +int +SSL_set_quic_method(SSL *ssl, const SSL_QUIC_METHOD *quic_method) +{ + BIO *rbio, *wbio; + ngx_connection_t *c; + ngx_quic_compat_t *com; + ngx_quic_connection_t *qc; + + c = ngx_ssl_get_connection(ssl); + + ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0, "quic compat set method"); + + qc = ngx_quic_get_connection(c); + + qc->compat = ngx_pcalloc(c->pool, sizeof(ngx_quic_compat_t)); + if (qc->compat == NULL) { + return 0; + } + + com = qc->compat; + com->method = quic_method; + + rbio = BIO_new(BIO_s_mem()); + if (rbio == NULL) { + return 0; + } + + wbio = BIO_new(BIO_s_null()); + if (wbio == NULL) { + return 0; + } + + SSL_set_bio(ssl, rbio, wbio); + + SSL_set_msg_callback(ssl, ngx_quic_compat_message_callback); + + /* early data is not supported */ + SSL_set_max_early_data(ssl, 0); + + return 1; +} + + +static void +ngx_quic_compat_message_callback(int write_p, int version, int content_type, + const void *buf, size_t len, SSL *ssl, void *arg) +{ + ngx_uint_t alert; + ngx_connection_t *c; + ngx_quic_compat_t *com; + ngx_quic_connection_t *qc; + enum ssl_encryption_level_t level; + + if (!write_p) { + return; + } + + c = ngx_ssl_get_connection(ssl); + qc = ngx_quic_get_connection(c); + + if (qc == NULL) { + /* closing */ + return; + } + + com = qc->compat; + level = com->write_level; + + switch (content_type) { + + case SSL3_RT_HANDSHAKE: + ngx_log_debug2(NGX_LOG_DEBUG_EVENT, c->log, 0, + "quic compat tx %s len:%uz ", + ngx_quic_level_name(level), len); + + (void) com->method->add_handshake_data(ssl, level, buf, len); + + break; + + case SSL3_RT_ALERT: + if (len >= 2) { + alert = ((u_char *) buf)[1]; + + ngx_log_debug3(NGX_LOG_DEBUG_EVENT, c->log, 0, + "quic compat %s alert:%ui len:%uz ", + ngx_quic_level_name(level), alert, len); + + (void) com->method->send_alert(ssl, level, alert); + } + + break; + } +} + + +int +SSL_provide_quic_data(SSL *ssl, enum ssl_encryption_level_t level, + const uint8_t *data, size_t len) +{ + BIO *rbio; + size_t n; + u_char *p; + ngx_str_t res; + ngx_connection_t *c; + ngx_quic_compat_t *com; + ngx_quic_connection_t *qc; + ngx_quic_compat_record_t rec; + u_char in[NGX_QUIC_COMPAT_RECORD_SIZE + 1]; + u_char out[NGX_QUIC_COMPAT_RECORD_SIZE + 1 + + SSL3_RT_HEADER_LENGTH + + EVP_GCM_TLS_TAG_LEN]; + + c = ngx_ssl_get_connection(ssl); + + ngx_log_debug2(NGX_LOG_DEBUG_EVENT, c->log, 0, "quic compat rx %s len:%uz", + ngx_quic_level_name(level), len); + + qc = ngx_quic_get_connection(c); + com = qc->compat; + rbio = SSL_get_rbio(ssl); + + while (len) { + ngx_memzero(&rec, sizeof(ngx_quic_compat_record_t)); + + rec.type = SSL3_RT_HANDSHAKE; + rec.log = c->log; + rec.number = com->read_record++; + rec.keys = &com->keys; + + if (level == ssl_encryption_initial) { + n = ngx_min(len, 65535); + + rec.payload.len = n; + rec.payload.data = (u_char *) data; + + ngx_quic_compat_create_header(&rec, out, 1); + + BIO_write(rbio, out, SSL3_RT_HEADER_LENGTH); + BIO_write(rbio, data, n); + +#if defined(NGX_QUIC_DEBUG_CRYPTO) && defined(NGX_QUIC_DEBUG_PACKETS) + ngx_log_debug5(NGX_LOG_DEBUG_EVENT, c->log, 0, + "quic compat record len:%uz %*xs%*xs", + n + SSL3_RT_HEADER_LENGTH, + (size_t) SSL3_RT_HEADER_LENGTH, out, n, data); +#endif + + } else { + n = ngx_min(len, NGX_QUIC_COMPAT_RECORD_SIZE); + + p = ngx_cpymem(in, data, n); + *p++ = SSL3_RT_HANDSHAKE; + + rec.payload.len = p - in; + rec.payload.data = in; + + res.data = out; + + if (ngx_quic_compat_create_record(&rec, &res) != NGX_OK) { + return 0; + } + +#if defined(NGX_QUIC_DEBUG_CRYPTO) && defined(NGX_QUIC_DEBUG_PACKETS) + ngx_log_debug2(NGX_LOG_DEBUG_EVENT, c->log, 0, + "quic compat record len:%uz %xV", res.len, &res); +#endif + + BIO_write(rbio, res.data, res.len); + } + + data += n; + len -= n; + } + + return 1; +} + + +static size_t +ngx_quic_compat_create_header(ngx_quic_compat_record_t *rec, u_char *out, + ngx_uint_t plain) +{ + u_char type; + size_t len; + + len = rec->payload.len; + + if (plain) { + type = rec->type; + + } else { + type = SSL3_RT_APPLICATION_DATA; + len += EVP_GCM_TLS_TAG_LEN; + } + + out[0] = type; + out[1] = 0x03; + out[2] = 0x03; + out[3] = (len >> 8); + out[4] = len; + + return 5; +} + + +static ngx_int_t +ngx_quic_compat_create_record(ngx_quic_compat_record_t *rec, ngx_str_t *res) +{ + ngx_str_t ad, out; + ngx_quic_secret_t *secret; + ngx_quic_ciphers_t ciphers; + u_char nonce[NGX_QUIC_IV_LEN]; + + ad.data = res->data; + ad.len = ngx_quic_compat_create_header(rec, ad.data, 0); + + out.len = rec->payload.len + EVP_GCM_TLS_TAG_LEN; + out.data = res->data + ad.len; + +#ifdef NGX_QUIC_DEBUG_CRYPTO + ngx_log_debug2(NGX_LOG_DEBUG_EVENT, rec->log, 0, + "quic compat ad len:%uz %xV", ad.len, &ad); +#endif + + if (ngx_quic_ciphers(rec->keys->cipher, &ciphers, rec->level) == NGX_ERROR) + { + return NGX_ERROR; + } + + secret = &rec->keys->secret; + + ngx_memcpy(nonce, secret->iv.data, secret->iv.len); + ngx_quic_compute_nonce(nonce, sizeof(nonce), rec->number); + + if (ngx_quic_tls_seal(ciphers.c, secret, &out, + nonce, &rec->payload, &ad, rec->log) + != NGX_OK) + { + return NGX_ERROR; + } + + res->len = ad.len + out.len; + + return NGX_OK; +} + + +enum ssl_encryption_level_t +SSL_quic_read_level(const SSL *ssl) +{ + ngx_connection_t *c; + ngx_quic_connection_t *qc; + + c = ngx_ssl_get_connection(ssl); + qc = ngx_quic_get_connection(c); + + return qc->compat->read_level; +} + + +enum ssl_encryption_level_t +SSL_quic_write_level(const SSL *ssl) +{ + ngx_connection_t *c; + ngx_quic_connection_t *qc; + + c = ngx_ssl_get_connection(ssl); + qc = ngx_quic_get_connection(c); + + return qc->compat->write_level; +} + + +int +SSL_set_quic_transport_params(SSL *ssl, const uint8_t *params, + size_t params_len) +{ + ngx_connection_t *c; + ngx_quic_compat_t *com; + ngx_quic_connection_t *qc; + + c = ngx_ssl_get_connection(ssl); + qc = ngx_quic_get_connection(c); + com = qc->compat; + + com->tp.len = params_len; + com->tp.data = (u_char *) params; + + return 1; +} + + +void +SSL_get_peer_quic_transport_params(const SSL *ssl, const uint8_t **out_params, + size_t *out_params_len) +{ + ngx_connection_t *c; + ngx_quic_compat_t *com; + ngx_quic_connection_t *qc; + + c = ngx_ssl_get_connection(ssl); + qc = ngx_quic_get_connection(c); + com = qc->compat; + + *out_params = com->ctp.data; + *out_params_len = com->ctp.len; +} + +#endif /* NGX_QUIC_OPENSSL_COMPAT */ diff --git a/src/event/quic/ngx_event_quic_openssl_compat.h b/src/event/quic/ngx_event_quic_openssl_compat.h new file mode 100644 --- /dev/null +++ b/src/event/quic/ngx_event_quic_openssl_compat.h @@ -0,0 +1,60 @@ + +/* + * Copyright (C) Nginx, Inc. + */ + + +#ifndef _NGX_EVENT_QUIC_OPENSSL_COMPAT_H_INCLUDED_ +#define _NGX_EVENT_QUIC_OPENSSL_COMPAT_H_INCLUDED_ + +#ifdef TLSEXT_TYPE_quic_transport_parameters +#undef NGX_QUIC_OPENSSL_COMPAT +#else + + +#include +#include + + +typedef struct ngx_quic_compat_s ngx_quic_compat_t; + + +enum ssl_encryption_level_t { + ssl_encryption_initial = 0, + ssl_encryption_early_data, + ssl_encryption_handshake, + ssl_encryption_application +}; + + +typedef struct ssl_quic_method_st { + int (*set_read_secret)(SSL *ssl, enum ssl_encryption_level_t level, + const SSL_CIPHER *cipher, + const uint8_t *rsecret, size_t secret_len); + int (*set_write_secret)(SSL *ssl, enum ssl_encryption_level_t level, + const SSL_CIPHER *cipher, + const uint8_t *wsecret, size_t secret_len); + int (*add_handshake_data)(SSL *ssl, enum ssl_encryption_level_t level, + const uint8_t *data, size_t len); + int (*flush_flight)(SSL *ssl); + int (*send_alert)(SSL *ssl, enum ssl_encryption_level_t level, + uint8_t alert); +} SSL_QUIC_METHOD; + + +ngx_int_t ngx_quic_compat_init(ngx_conf_t *cf, SSL_CTX *ctx); + +int SSL_set_quic_method(SSL *ssl, const SSL_QUIC_METHOD *quic_method); +int SSL_provide_quic_data(SSL *ssl, enum ssl_encryption_level_t level, + const uint8_t *data, size_t len); +enum ssl_encryption_level_t SSL_quic_read_level(const SSL *ssl); +enum ssl_encryption_level_t SSL_quic_write_level(const SSL *ssl); +int SSL_set_quic_transport_params(SSL *ssl, const uint8_t *params, + size_t params_len); +void SSL_get_peer_quic_transport_params(const SSL *ssl, + const uint8_t **out_params, size_t *out_params_len); + + +#endif /* TLSEXT_TYPE_quic_transport_parameters */ + +#endif /* _NGX_EVENT_QUIC_OPENSSL_COMPAT_H_INCLUDED_ */ diff --git a/src/event/quic/ngx_event_quic_protection.c b/src/event/quic/ngx_event_quic_protection.c --- a/src/event/quic/ngx_event_quic_protection.c +++ b/src/event/quic/ngx_event_quic_protection.c @@ -23,37 +23,6 @@ #endif -#ifdef OPENSSL_IS_BORINGSSL -#define ngx_quic_cipher_t EVP_AEAD -#else -#define ngx_quic_cipher_t EVP_CIPHER -#endif - - -typedef struct { - const ngx_quic_cipher_t *c; - const EVP_CIPHER *hp; - const EVP_MD *d; -} ngx_quic_ciphers_t; - - -typedef struct { - size_t out_len; - u_char *out; - - size_t prk_len; - const uint8_t *prk; - - size_t label_len; - const u_char *label; -} ngx_quic_hkdf_t; - -#define ngx_quic_hkdf_set(seq, _label, _out, _prk) \ - (seq)->out_len = (_out)->len; (seq)->out = (_out)->data; \ - (seq)->prk_len = (_prk)->len, (seq)->prk = (_prk)->data, \ - (seq)->label_len = (sizeof(_label) - 1); (seq)->label = (u_char *)(_label); - - static ngx_int_t ngx_hkdf_expand(u_char *out_key, size_t out_len, const EVP_MD *digest, const u_char *prk, size_t prk_len, const u_char *info, size_t info_len); @@ -63,20 +32,12 @@ static ngx_int_t ngx_hkdf_extract(u_char static uint64_t ngx_quic_parse_pn(u_char **pos, ngx_int_t len, u_char *mask, uint64_t *largest_pn); -static void ngx_quic_compute_nonce(u_char *nonce, size_t len, uint64_t pn); -static ngx_int_t ngx_quic_ciphers(ngx_uint_t id, - ngx_quic_ciphers_t *ciphers, enum ssl_encryption_level_t level); static ngx_int_t ngx_quic_tls_open(const ngx_quic_cipher_t *cipher, ngx_quic_secret_t *s, ngx_str_t *out, u_char *nonce, ngx_str_t *in, ngx_str_t *ad, ngx_log_t *log); -static ngx_int_t ngx_quic_tls_seal(const ngx_quic_cipher_t *cipher, - ngx_quic_secret_t *s, ngx_str_t *out, u_char *nonce, ngx_str_t *in, - ngx_str_t *ad, ngx_log_t *log); static ngx_int_t ngx_quic_tls_hp(ngx_log_t *log, const EVP_CIPHER *cipher, ngx_quic_secret_t *s, u_char *out, u_char *in); -static ngx_int_t ngx_quic_hkdf_expand(ngx_quic_hkdf_t *hkdf, - const EVP_MD *digest, ngx_log_t *log); static ngx_int_t ngx_quic_create_packet(ngx_quic_header_t *pkt, ngx_str_t *res); @@ -84,7 +45,7 @@ static ngx_int_t ngx_quic_create_retry_p ngx_str_t *res); -static ngx_int_t +ngx_int_t ngx_quic_ciphers(ngx_uint_t id, ngx_quic_ciphers_t *ciphers, enum ssl_encryption_level_t level) { @@ -221,7 +182,7 @@ ngx_quic_keys_set_initial_secret(ngx_qui } -static ngx_int_t +ngx_int_t ngx_quic_hkdf_expand(ngx_quic_hkdf_t *h, const EVP_MD *digest, ngx_log_t *log) { size_t info_len; @@ -480,7 +441,7 @@ ngx_quic_tls_open(const ngx_quic_cipher_ } -static ngx_int_t +ngx_int_t ngx_quic_tls_seal(const ngx_quic_cipher_t *cipher, ngx_quic_secret_t *s, ngx_str_t *out, u_char *nonce, ngx_str_t *in, ngx_str_t *ad, ngx_log_t *log) { @@ -961,7 +922,7 @@ ngx_quic_parse_pn(u_char **pos, ngx_int_ } -static void +void ngx_quic_compute_nonce(u_char *nonce, size_t len, uint64_t pn) { nonce[len - 8] ^= (pn >> 56) & 0x3f; diff --git a/src/event/quic/ngx_event_quic_protection.h b/src/event/quic/ngx_event_quic_protection.h --- a/src/event/quic/ngx_event_quic_protection.h +++ b/src/event/quic/ngx_event_quic_protection.h @@ -23,6 +23,13 @@ #define NGX_QUIC_MAX_MD_SIZE 48 +#ifdef OPENSSL_IS_BORINGSSL +#define ngx_quic_cipher_t EVP_AEAD +#else +#define ngx_quic_cipher_t EVP_CIPHER +#endif + + typedef struct { size_t len; u_char data[NGX_QUIC_MAX_MD_SIZE]; @@ -56,6 +63,30 @@ struct ngx_quic_keys_s { }; +typedef struct { + const ngx_quic_cipher_t *c; + const EVP_CIPHER *hp; + const EVP_MD *d; +} ngx_quic_ciphers_t; + + +typedef struct { + size_t out_len; + u_char *out; + + size_t prk_len; + const uint8_t *prk; + + size_t label_len; + const u_char *label; +} ngx_quic_hkdf_t; + +#define ngx_quic_hkdf_set(seq, _label, _out, _prk) \ + (seq)->out_len = (_out)->len; (seq)->out = (_out)->data; \ + (seq)->prk_len = (_prk)->len, (seq)->prk = (_prk)->data, \ + (seq)->label_len = (sizeof(_label) - 1); (seq)->label = (u_char *)(_label); + + ngx_int_t ngx_quic_keys_set_initial_secret(ngx_quic_keys_t *keys, ngx_str_t *secret, ngx_log_t *log); ngx_int_t ngx_quic_keys_set_encryption_secret(ngx_log_t *log, @@ -70,6 +101,14 @@ void ngx_quic_keys_switch(ngx_connection ngx_int_t ngx_quic_keys_update(ngx_connection_t *c, ngx_quic_keys_t *keys); ngx_int_t ngx_quic_encrypt(ngx_quic_header_t *pkt, ngx_str_t *res); ngx_int_t ngx_quic_decrypt(ngx_quic_header_t *pkt, uint64_t *largest_pn); +void ngx_quic_compute_nonce(u_char *nonce, size_t len, uint64_t pn); +ngx_int_t ngx_quic_ciphers(ngx_uint_t id, ngx_quic_ciphers_t *ciphers, + enum ssl_encryption_level_t level); +ngx_int_t ngx_quic_tls_seal(const ngx_quic_cipher_t *cipher, + ngx_quic_secret_t *s, ngx_str_t *out, u_char *nonce, ngx_str_t *in, + ngx_str_t *ad, ngx_log_t *log); +ngx_int_t ngx_quic_hkdf_expand(ngx_quic_hkdf_t *hkdf, const EVP_MD *digest, + ngx_log_t *log); #endif /* _NGX_EVENT_QUIC_PROTECTION_H_INCLUDED_ */ diff --git a/src/event/quic/ngx_event_quic_ssl.c b/src/event/quic/ngx_event_quic_ssl.c --- a/src/event/quic/ngx_event_quic_ssl.c +++ b/src/event/quic/ngx_event_quic_ssl.c @@ -10,6 +10,13 @@ #include +#if defined OPENSSL_IS_BORINGSSL \ + || defined LIBRESSL_VERSION_NUMBER \ + || NGX_QUIC_OPENSSL_COMPAT +#define NGX_QUIC_BORINGSSL_API 1 +#endif + + /* * RFC 9000, 7.5. Cryptographic Message Buffering * @@ -18,7 +25,7 @@ #define NGX_QUIC_MAX_BUFFERED 65535 -#if defined OPENSSL_IS_BORINGSSL || defined LIBRESSL_VERSION_NUMBER +#if (NGX_QUIC_BORINGSSL_API) static int ngx_quic_set_read_secret(ngx_ssl_conn_t *ssl_conn, enum ssl_encryption_level_t level, const SSL_CIPHER *cipher, const uint8_t *secret, size_t secret_len); @@ -39,7 +46,7 @@ static int ngx_quic_send_alert(ngx_ssl_c static ngx_int_t ngx_quic_crypto_input(ngx_connection_t *c, ngx_chain_t *data); -#if defined OPENSSL_IS_BORINGSSL || defined LIBRESSL_VERSION_NUMBER +#if (NGX_QUIC_BORINGSSL_API) static int ngx_quic_set_read_secret(ngx_ssl_conn_t *ssl_conn, @@ -517,7 +524,7 @@ ngx_quic_init_connection(ngx_connection_ ssl_conn = c->ssl->connection; if (!quic_method.send_alert) { -#if defined OPENSSL_IS_BORINGSSL || defined LIBRESSL_VERSION_NUMBER +#if (NGX_QUIC_BORINGSSL_API) quic_method.set_read_secret = ngx_quic_set_read_secret; quic_method.set_write_secret = ngx_quic_set_write_secret; #else diff --git a/src/http/modules/ngx_http_ssl_module.c b/src/http/modules/ngx_http_ssl_module.c --- a/src/http/modules/ngx_http_ssl_module.c +++ b/src/http/modules/ngx_http_ssl_module.c @@ -9,6 +9,10 @@ #include #include +#if (NGX_QUIC_OPENSSL_COMPAT) +#include +#endif + typedef ngx_int_t (*ngx_ssl_variable_handler_pt)(ngx_connection_t *c, ngx_pool_t *pool, ngx_str_t *s); @@ -1317,16 +1321,22 @@ ngx_http_ssl_init(ngx_conf_t *cf) continue; } + cscf = addr[a].default_server; + sscf = cscf->ctx->srv_conf[ngx_http_ssl_module.ctx_index]; + if (addr[a].opt.http3) { name = "http3"; +#if (NGX_QUIC_OPENSSL_COMPAT) + if (ngx_quic_compat_init(cf, sscf->ssl.ctx) != NGX_OK) { + return NGX_ERROR; + } +#endif + } else { name = "ssl"; } - cscf = addr[a].default_server; - sscf = cscf->ctx->srv_conf[ngx_http_ssl_module.ctx_index]; - if (sscf->certificates) { if (addr[a].opt.http3 && !(sscf->protocols & NGX_SSL_TLSv1_3)) { diff --git a/src/stream/ngx_stream_ssl_module.c b/src/stream/ngx_stream_ssl_module.c --- a/src/stream/ngx_stream_ssl_module.c +++ b/src/stream/ngx_stream_ssl_module.c @@ -9,6 +9,10 @@ #include #include +#if (NGX_QUIC_OPENSSL_COMPAT) +#include +#endif + typedef ngx_int_t (*ngx_ssl_variable_handler_pt)(ngx_connection_t *c, ngx_pool_t *pool, ngx_str_t *s); @@ -1218,6 +1222,12 @@ ngx_stream_ssl_init(ngx_conf_t *cf) scf = listen[i].ctx->srv_conf[ngx_stream_ssl_module.ctx_index]; +#if (NGX_QUIC_OPENSSL_COMPAT) + if (ngx_quic_compat_init(cf, scf->ssl.ctx) != NGX_OK) { + return NGX_ERROR; + } +#endif + if (scf->certificates && !(scf->protocols & NGX_SSL_TLSv1_3)) { ngx_log_error(NGX_LOG_EMERG, cf->log, 0, "\"ssl_protocols\" must enable TLSv1.3 for " From pluknet at nginx.com Wed Feb 8 15:17:09 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Wed, 8 Feb 2023 19:17:09 +0400 Subject: [PATCH] QUIC: OpenSSL compatibility layer In-Reply-To: <20230208144138.u7pozeds34nqr6mi@N00W24XTQX> References: <64a365dcb52503e91d91.1671606452@arut-laptop> <20230109111316.yfqj24unoubibihz@N00W24XTQX> <20230202183522.o4w5xbjvegbchts4@Y9MQ9X2QVV> <20230206142701.pxzaq3irckqads7y@N00W24XTQX> <20230208122810.tsxynyhlqgt2ubsm@Y9MQ9X2QVV> <20230208144138.u7pozeds34nqr6mi@N00W24XTQX> Message-ID: <20230208151709.74ttczym3bgwfdli@Y9MQ9X2QVV> On Wed, Feb 08, 2023 at 06:41:38PM +0400, Roman Arutyunyan wrote: > # HG changeset patch > # User Roman Arutyunyan > # Date 1675867049 -14400 > # Wed Feb 08 18:37:29 2023 +0400 > # Branch quic > # Node ID a3142c8833f5bf1186599e7938141f5062fac4a2 > # Parent 3c33d39a51d334d99fcc7d2b45e8d8190c431492 > QUIC: OpenSSL compatibility layer. > > The change allows to compile QUIC with OpenSSL which lacks BoringSSL QUIC API. > > This implementation does not support 0-RTT. > Looks good. From pluknet at nginx.com Wed Feb 8 15:22:32 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Wed, 8 Feb 2023 19:22:32 +0400 Subject: [PATCH 0 of 3] Directives for enabling http2 and http3 In-Reply-To: References: Message-ID: > On 7 Feb 2023, at 18:50, Roman Arutyunyan wrote: > > Hi, > > Updated the patches based on latest comments. Looks good to me. -- Sergey Kandaurov From hafl at me.com Wed Feb 8 15:54:06 2023 From: hafl at me.com (Safar Safarly) Date: Wed, 08 Feb 2023 18:54:06 +0300 Subject: [PATCH 1 of 2] Core: connect() error log message made more verbose Message-ID: <55553146bd984be7e9e3.1675871646@ssafarly-nb> # HG changeset patch # User Safar Safarly # Date 1675779866 -10800 # Tue Feb 07 17:24:26 2023 +0300 # Node ID 55553146bd984be7e9e3bbfa851c282feda82d93 # Parent cffaf3f2eec8fd33605c2a37814f5ffc30371989 Core: connect() error log message made more verbose There was a major problem in logs: we could not identify to which servers connect() has failed. Previously log produced: ngx_log_error(NGX_LOG_ALERT, ngx_cycle->log, ngx_socket_errno, "connect() failed"); And now we'll have an address or unix socket in log: ngx_log_error(NGX_LOG_ALERT, ngx_cycle->log, ngx_socket_errno, "connect() to %V failed", &peer->server.name); Message format has chosen to be exact as it is used in ngx_event_connect.c:242 with similar error logging: ngx_log_error(level, c->log, err, "connect() to %V failed", pc->name); So everywhere connect() could fail we'd get a uniform and verbose error message in log. diff -r cffaf3f2eec8 -r 55553146bd98 src/core/ngx_resolver.c --- a/src/core/ngx_resolver.c Thu Feb 02 23:38:48 2023 +0300 +++ b/src/core/ngx_resolver.c Tue Feb 07 17:24:26 2023 +0300 @@ -4512,7 +4512,7 @@ if (rc == -1) { ngx_log_error(NGX_LOG_CRIT, &rec->log, ngx_socket_errno, - "connect() failed"); + "connect() to %V failed", &rec->server); goto failed; } diff -r cffaf3f2eec8 -r 55553146bd98 src/core/ngx_syslog.c --- a/src/core/ngx_syslog.c Thu Feb 02 23:38:48 2023 +0300 +++ b/src/core/ngx_syslog.c Tue Feb 07 17:24:26 2023 +0300 @@ -337,7 +337,7 @@ if (connect(fd, peer->server.sockaddr, peer->server.socklen) == -1) { ngx_log_error(NGX_LOG_ALERT, ngx_cycle->log, ngx_socket_errno, - "connect() failed"); + "connect() to %V failed", &peer->server.name); goto failed; } From hafl at me.com Wed Feb 8 15:54:07 2023 From: hafl at me.com (Safar Safarly) Date: Wed, 08 Feb 2023 18:54:07 +0300 Subject: [PATCH 2 of 2] Added config path when unknown variable error occurs In-Reply-To: <55553146bd984be7e9e3.1675871646@ssafarly-nb> References: <55553146bd984be7e9e3.1675871646@ssafarly-nb> Message-ID: <1396218aa1296ffc9776.1675871647@ssafarly-nb> # HG changeset patch # User Safar Safarly # Date 1675853769 -10800 # Wed Feb 08 13:56:09 2023 +0300 # Node ID 1396218aa1296ffc9776644534a1541b78a460a0 # Parent 55553146bd984be7e9e3bbfa851c282feda82d93 Added config path when unknown variable error occurs Previously, when unknown variable occured, error was logged with ngx_log_error(): nginx: [emerg] unknown "rate" variable instead of ngx_conf_log_error(): nginx: [emerg] unknown "rate" variable in /conf/nginx.conf:27 diff -r 55553146bd98 -r 1396218aa129 src/http/ngx_http_variables.c --- a/src/http/ngx_http_variables.c Tue Feb 07 17:24:26 2023 +0300 +++ b/src/http/ngx_http_variables.c Wed Feb 08 13:56:09 2023 +0300 @@ -2792,8 +2792,8 @@ } if (v[i].get_handler == NULL) { - ngx_log_error(NGX_LOG_EMERG, cf->log, 0, - "unknown \"%V\" variable", &v[i].name); + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, + "unknown \"%V\" variable", &v[i].name); return NGX_ERROR; } diff -r 55553146bd98 -r 1396218aa129 src/stream/ngx_stream_variables.c --- a/src/stream/ngx_stream_variables.c Tue Feb 07 17:24:26 2023 +0300 +++ b/src/stream/ngx_stream_variables.c Wed Feb 08 13:56:09 2023 +0300 @@ -1277,8 +1277,8 @@ } if (v[i].get_handler == NULL) { - ngx_log_error(NGX_LOG_EMERG, cf->log, 0, - "unknown \"%V\" variable", &v[i].name); + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, + "unknown \"%V\" variable", &v[i].name); return NGX_ERROR; } From P.Pautov at F5.com Thu Feb 9 06:55:22 2023 From: P.Pautov at F5.com (Pavel Pautov) Date: Thu, 9 Feb 2023 06:55:22 +0000 Subject: [PATCH 2 of 4] Win32: handling of localized MSVC cl output In-Reply-To: <43098cb134a87a404b70.1671543016@1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa> References: <43098cb134a87a404b70.1671543016@1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa> Message-ID: Hi, The 'for 80x86' part of cl version (in auto/configure output) always felt odd. Given it's used in subsequent patch, can we at least strip localized 'for' from NGX_MSVC_VER? Having it in otherwise non-localized output of "nginx -V" doesn't seem right either. From mdounin at mdounin.ru Thu Feb 9 08:11:57 2023 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 9 Feb 2023 11:11:57 +0300 Subject: [PATCH 1 of 2] Core: connect() error log message made more verbose In-Reply-To: <55553146bd984be7e9e3.1675871646@ssafarly-nb> References: <55553146bd984be7e9e3.1675871646@ssafarly-nb> Message-ID: Hello! On Wed, Feb 08, 2023 at 06:54:06PM +0300, Safar Safarly via nginx-devel wrote: > # HG changeset patch > # User Safar Safarly > # Date 1675779866 -10800 > # Tue Feb 07 17:24:26 2023 +0300 > # Node ID 55553146bd984be7e9e3bbfa851c282feda82d93 > # Parent cffaf3f2eec8fd33605c2a37814f5ffc30371989 > Core: connect() error log message made more verbose > > There was a major problem in logs: we could not identify to which servers > connect() has failed. Previously log produced: > > ngx_log_error(NGX_LOG_ALERT, ngx_cycle->log, ngx_socket_errno, > "connect() failed"); > > And now we'll have an address or unix socket in log: > > ngx_log_error(NGX_LOG_ALERT, ngx_cycle->log, ngx_socket_errno, > "connect() to %V failed", &peer->server.name); > > Message format has chosen to be exact as it is used in ngx_event_connect.c:242 > with similar error logging: > > ngx_log_error(level, c->log, err, "connect() to %V failed", > pc->name); > > So everywhere connect() could fail we'd get a uniform and verbose error message > in log. Thanks for the patch. Indeed, it might be non-trivial to find out what goes wrong when no context is provided in error messages, and logging to syslog fails to provide any. Please see comments below. > > diff -r cffaf3f2eec8 -r 55553146bd98 src/core/ngx_resolver.c > --- a/src/core/ngx_resolver.c Thu Feb 02 23:38:48 2023 +0300 > +++ b/src/core/ngx_resolver.c Tue Feb 07 17:24:26 2023 +0300 > @@ -4512,7 +4512,7 @@ > > if (rc == -1) { > ngx_log_error(NGX_LOG_CRIT, &rec->log, ngx_socket_errno, > - "connect() failed"); > + "connect() to %V failed", &rec->server); Resolver already provides details about the particular server it connects to, for example: 2023/02/08 19:39:53 [crit] 23077#100129: connect() failed (2: No such file or directory) while resolving, resolver: unix:/resolver.socket See ngx_resolver_log_error() for more information. > > goto failed; > } > diff -r cffaf3f2eec8 -r 55553146bd98 src/core/ngx_syslog.c > --- a/src/core/ngx_syslog.c Thu Feb 02 23:38:48 2023 +0300 > +++ b/src/core/ngx_syslog.c Tue Feb 07 17:24:26 2023 +0300 > @@ -337,7 +337,7 @@ > > if (connect(fd, peer->server.sockaddr, peer->server.socklen) == -1) { > ngx_log_error(NGX_LOG_ALERT, ngx_cycle->log, ngx_socket_errno, > - "connect() failed"); > + "connect() to %V failed", &peer->server.name); > goto failed; > } For syslog logging, I tend to think that this isn't really enough. For example, consider send() errors which also might happen when logging to syslog. Right now these are logged as: 2023/02/09 07:29:11 [alert] 23230#100162: send() failed (49: Can't assign requested address) 2023/02/09 07:29:11 [alert] 23230#100162: send() failed (49: Can't assign requested address) Clearly no better than connect() errors: 2023/02/09 07:29:11 [alert] 23230#100162: connect() failed (2: No such file or directory) A better fix would be to provide a log handler, similarly to ngx_resolver_log_error() mentioned above. With a simple log handler the above messages will instead look like: 2023/02/09 07:48:11 [alert] 23699#100145: send() failed (49: Can't assign requested address) while logging to syslog, server: 127.0.0.2:514 2023/02/09 07:48:11 [alert] 23699#100145: send() failed (49: Can't assign requested address) while logging to syslog, server: 127.0.0.2:514 2023/02/09 07:48:11 [alert] 23699#100145: connect() failed (2: No such file or directory) while logging to syslog, server: unix:/log.socket Patch: # HG changeset patch # User Maxim Dounin # Date 1675929813 -10800 # Thu Feb 09 11:03:33 2023 +0300 # Node ID 6b662855bf77c678a3954939aefe3fd4b4af4c70 # Parent cffaf3f2eec8fd33605c2a37814f5ffc30371989 Syslog: introduced error log handler. This ensures that errors which happen during logging to syslog are logged with proper context, such as "while logging to syslog" and the server name. Prodded by Safar Safarly. diff --git a/src/core/ngx_syslog.c b/src/core/ngx_syslog.c --- a/src/core/ngx_syslog.c +++ b/src/core/ngx_syslog.c @@ -18,6 +18,7 @@ static char *ngx_syslog_parse_args(ngx_conf_t *cf, ngx_syslog_peer_t *peer); static ngx_int_t ngx_syslog_init_peer(ngx_syslog_peer_t *peer); static void ngx_syslog_cleanup(void *data); +static u_char *ngx_syslog_log_error(ngx_log_t *log, u_char *buf, size_t len); static char *facilities[] = { @@ -66,6 +67,8 @@ ngx_syslog_process_conf(ngx_conf_t *cf, ngx_str_set(&peer->tag, "nginx"); } + peer->logp = &cf->cycle->new_log; + peer->conn.fd = (ngx_socket_t) -1; peer->conn.read = &ngx_syslog_dummy_event; @@ -286,15 +289,19 @@ ngx_syslog_send(ngx_syslog_peer_t *peer, { ssize_t n; + if (peer->log.handler == NULL) { + peer->log = *peer->logp; + peer->log.handler = ngx_syslog_log_error; + peer->log.data = peer; + peer->log.action = "logging to syslog"; + } + if (peer->conn.fd == (ngx_socket_t) -1) { if (ngx_syslog_init_peer(peer) != NGX_OK) { return NGX_ERROR; } } - /* log syslog socket events with valid log */ - peer->conn.log = ngx_cycle->log; - if (ngx_send) { n = ngx_send(&peer->conn, buf, len); @@ -324,24 +331,25 @@ ngx_syslog_init_peer(ngx_syslog_peer_t * fd = ngx_socket(peer->server.sockaddr->sa_family, SOCK_DGRAM, 0); if (fd == (ngx_socket_t) -1) { - ngx_log_error(NGX_LOG_ALERT, ngx_cycle->log, ngx_socket_errno, + ngx_log_error(NGX_LOG_ALERT, &peer->log, ngx_socket_errno, ngx_socket_n " failed"); return NGX_ERROR; } if (ngx_nonblocking(fd) == -1) { - ngx_log_error(NGX_LOG_ALERT, ngx_cycle->log, ngx_socket_errno, + ngx_log_error(NGX_LOG_ALERT, &peer->log, ngx_socket_errno, ngx_nonblocking_n " failed"); goto failed; } if (connect(fd, peer->server.sockaddr, peer->server.socklen) == -1) { - ngx_log_error(NGX_LOG_ALERT, ngx_cycle->log, ngx_socket_errno, + ngx_log_error(NGX_LOG_ALERT, &peer->log, ngx_socket_errno, "connect() failed"); goto failed; } peer->conn.fd = fd; + peer->conn.log = &peer->log; /* UDP sockets are always ready to write */ peer->conn.write->ready = 1; @@ -351,7 +359,7 @@ ngx_syslog_init_peer(ngx_syslog_peer_t * failed: if (ngx_close_socket(fd) == -1) { - ngx_log_error(NGX_LOG_ALERT, ngx_cycle->log, ngx_socket_errno, + ngx_log_error(NGX_LOG_ALERT, &peer->log, ngx_socket_errno, ngx_close_socket_n " failed"); } @@ -372,7 +380,30 @@ ngx_syslog_cleanup(void *data) } if (ngx_close_socket(peer->conn.fd) == -1) { - ngx_log_error(NGX_LOG_ALERT, ngx_cycle->log, ngx_socket_errno, + ngx_log_error(NGX_LOG_ALERT, &peer->log, ngx_socket_errno, ngx_close_socket_n " failed"); } } + + +static u_char * +ngx_syslog_log_error(ngx_log_t *log, u_char *buf, size_t len) +{ + u_char *p; + ngx_syslog_peer_t *peer; + + p = buf; + + if (log->action) { + p = ngx_snprintf(buf, len, " while %s", log->action); + len -= p - buf; + } + + peer = log->data; + + if (peer) { + p = ngx_snprintf(p, len, ", server: %V", &peer->server.name); + } + + return p; +} diff --git a/src/core/ngx_syslog.h b/src/core/ngx_syslog.h --- a/src/core/ngx_syslog.h +++ b/src/core/ngx_syslog.h @@ -9,14 +9,18 @@ typedef struct { - ngx_uint_t facility; - ngx_uint_t severity; - ngx_str_t tag; + ngx_uint_t facility; + ngx_uint_t severity; + ngx_str_t tag; - ngx_addr_t server; - ngx_connection_t conn; - unsigned busy:1; - unsigned nohostname:1; + ngx_addr_t server; + ngx_connection_t conn; + + ngx_log_t log; + ngx_log_t *logp; + + unsigned busy:1; + unsigned nohostname:1; } ngx_syslog_peer_t; -- Maxim Dounin http://mdounin.ru/ From mdounin at mdounin.ru Thu Feb 9 08:20:42 2023 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 9 Feb 2023 11:20:42 +0300 Subject: [PATCH 2 of 2] Added config path when unknown variable error occurs In-Reply-To: <1396218aa1296ffc9776.1675871647@ssafarly-nb> References: <55553146bd984be7e9e3.1675871646@ssafarly-nb> <1396218aa1296ffc9776.1675871647@ssafarly-nb> Message-ID: Hello! On Wed, Feb 08, 2023 at 06:54:07PM +0300, Safar Safarly via nginx-devel wrote: > # HG changeset patch > # User Safar Safarly > # Date 1675853769 -10800 > # Wed Feb 08 13:56:09 2023 +0300 > # Node ID 1396218aa1296ffc9776644534a1541b78a460a0 > # Parent 55553146bd984be7e9e3bbfa851c282feda82d93 > Added config path when unknown variable error occurs > > Previously, when unknown variable occured, error was logged with > ngx_log_error(): > > nginx: [emerg] unknown "rate" variable > > instead of ngx_conf_log_error(): > > nginx: [emerg] unknown "rate" variable in /conf/nginx.conf:27 > > diff -r 55553146bd98 -r 1396218aa129 src/http/ngx_http_variables.c > --- a/src/http/ngx_http_variables.c Tue Feb 07 17:24:26 2023 +0300 > +++ b/src/http/ngx_http_variables.c Wed Feb 08 13:56:09 2023 +0300 > @@ -2792,8 +2792,8 @@ > } > > if (v[i].get_handler == NULL) { > - ngx_log_error(NGX_LOG_EMERG, cf->log, 0, > - "unknown \"%V\" variable", &v[i].name); > + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, > + "unknown \"%V\" variable", &v[i].name); > > return NGX_ERROR; > } > diff -r 55553146bd98 -r 1396218aa129 src/stream/ngx_stream_variables.c > --- a/src/stream/ngx_stream_variables.c Tue Feb 07 17:24:26 2023 +0300 > +++ b/src/stream/ngx_stream_variables.c Wed Feb 08 13:56:09 2023 +0300 > @@ -1277,8 +1277,8 @@ > } > > if (v[i].get_handler == NULL) { > - ngx_log_error(NGX_LOG_EMERG, cf->log, 0, > - "unknown \"%V\" variable", &v[i].name); > + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, > + "unknown \"%V\" variable", &v[i].name); > return NGX_ERROR; > } > The ngx_http_variables_init_vars() function you are patching is called when the http block is fully parsed, and therefore ngx_conf_log_error() will always point to the end of the http block. This provides no usable information, so ngx_log_error() is used instead. -- Maxim Dounin http://mdounin.ru/ From pluknet at nginx.com Thu Feb 9 11:28:02 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Thu, 9 Feb 2023 15:28:02 +0400 Subject: [PATCH 2 of 3] HTTP/2: "http2" directive In-Reply-To: <735f9e501922e4b0a1b2.1675781431@arut-laptop> References: <735f9e501922e4b0a1b2.1675781431@arut-laptop> Message-ID: > On 7 Feb 2023, at 18:50, Roman Arutyunyan wrote: > > # HG changeset patch > # User Roman Arutyunyan > # Date 1675781276 -14400 > # Tue Feb 07 18:47:56 2023 +0400 > # Branch quic > # Node ID 735f9e501922e4b0a1b20730d62bac35ea398336 > # Parent 38eec3d9f2c0d2e6d041efe3ee6d9c1618d8f1e6 > HTTP/2: "http2" directive. > > The directive enables HTTP/2 in the current server. The previous way to > enable HTTP/2 via "listen ... http2" is now deprecated. The new approach > allows to share HTTP/2 and HTTP/0.9-1.1 on the same port. > > For SSL connections, HTTP/2 is now selected by ALPN callback based on whether > the protocol is enabled in the virtual server chosen by SNI. This however only > works since OpenSSL 1.0.2h, where ALPN callback is invoked after SNI callback. > For older versions of OpenSSL, HTTP/2 is enabled based on the default virtual > server configuration. > > For plain TCP connections, HTTP/2 is now auto-detected by HTTP/2 preface, if > HTTP/2 is enabled in the default virtual server. If preface is not matched, > HTTP/0.9-1.1 is assumed. > > diff --git a/src/http/modules/ngx_http_ssl_module.c b/src/http/modules/ngx_http_ssl_module.c > --- a/src/http/modules/ngx_http_ssl_module.c > +++ b/src/http/modules/ngx_http_ssl_module.c > @@ -427,6 +427,9 @@ ngx_http_ssl_alpn_select(ngx_ssl_conn_t > #if (NGX_HTTP_V2 || NGX_HTTP_V3) > ngx_http_connection_t *hc; > #endif > +#if (NGX_HTTP_V2) > + ngx_http_v2_srv_conf_t *h2scf; > +#endif > #if (NGX_HTTP_V3) > ngx_http_v3_srv_conf_t *h3scf; > #endif > @@ -448,12 +451,9 @@ ngx_http_ssl_alpn_select(ngx_ssl_conn_t > hc = c->data; > #endif > > -#if (NGX_HTTP_V2) > - if (hc->addr_conf->http2) { > - srv = (unsigned char *) NGX_HTTP_V2_ALPN_PROTO NGX_HTTP_ALPN_PROTOS; > - srvlen = sizeof(NGX_HTTP_V2_ALPN_PROTO NGX_HTTP_ALPN_PROTOS) - 1; > - } else > -#endif > + srv = (unsigned char *) NGX_HTTP_ALPN_PROTOS; > + srvlen = sizeof(NGX_HTTP_ALPN_PROTOS) - 1; > + > #if (NGX_HTTP_V3) > if (hc->addr_conf->quic) { > > @@ -479,10 +479,16 @@ ngx_http_ssl_alpn_select(ngx_ssl_conn_t > > } else > #endif > +#if (NGX_HTTP_V2) > { > - srv = (unsigned char *) NGX_HTTP_ALPN_PROTOS; > - srvlen = sizeof(NGX_HTTP_ALPN_PROTOS) - 1; > + h2scf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_v2_module); > + > + if (h2scf->enable || hc->addr_conf->http2) { > + srv = (unsigned char *) NGX_HTTP_V2_ALPN_PROTO NGX_HTTP_ALPN_PROTOS; > + srvlen = sizeof(NGX_HTTP_V2_ALPN_PROTO NGX_HTTP_ALPN_PROTOS) - 1; > + } > } > +#endif With NGX_HTTP_V3 defined but NGX_HTTP_V2 not, the else part will go to the SSL_select_next_proto() call. So, to fix this, NGX_HTTP_ALPN_PROTOS still has to be the last resort, for simplicity (and also reduces diff). My version: diff --git a/src/http/modules/ngx_http_ssl_module.c b/src/http/modules/ngx_http_ssl_module.c --- a/src/http/modules/ngx_http_ssl_module.c +++ b/src/http/modules/ngx_http_ssl_module.c @@ -427,6 +427,9 @@ ngx_http_ssl_alpn_select(ngx_ssl_conn_t #if (NGX_HTTP_V2 || NGX_HTTP_V3) ngx_http_connection_t *hc; #endif +#if (NGX_HTTP_V2) + ngx_http_v2_srv_conf_t *h2scf; +#endif #if (NGX_HTTP_V3) ngx_http_v3_srv_conf_t *h3scf; #endif @@ -449,7 +452,9 @@ ngx_http_ssl_alpn_select(ngx_ssl_conn_t #endif #if (NGX_HTTP_V2) - if (hc->addr_conf->http2) { + h2scf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_v2_module); + + if (h2scf->enable || hc->addr_conf->http2) { srv = (unsigned char *) NGX_HTTP_V2_ALPN_PROTO NGX_HTTP_ALPN_PROTOS; srvlen = sizeof(NGX_HTTP_V2_ALPN_PROTO NGX_HTTP_ALPN_PROTOS) - 1; } else > > if (SSL_select_next_proto((unsigned char **) out, outlen, srv, srvlen, > in, inlen) [..] -- Sergey Kandaurov From arut at nginx.com Thu Feb 9 12:02:34 2023 From: arut at nginx.com (Roman Arutyunyan) Date: Thu, 9 Feb 2023 16:02:34 +0400 Subject: [PATCH 2 of 3] HTTP/2: "http2" directive In-Reply-To: References: <735f9e501922e4b0a1b2.1675781431@arut-laptop> Message-ID: <20230209120234.rttyjviwir4sneed@N00W24XTQX> Hi, On Thu, Feb 09, 2023 at 03:28:02PM +0400, Sergey Kandaurov wrote: > > > On 7 Feb 2023, at 18:50, Roman Arutyunyan wrote: > > > > # HG changeset patch > > # User Roman Arutyunyan > > # Date 1675781276 -14400 > > # Tue Feb 07 18:47:56 2023 +0400 > > # Branch quic > > # Node ID 735f9e501922e4b0a1b20730d62bac35ea398336 > > # Parent 38eec3d9f2c0d2e6d041efe3ee6d9c1618d8f1e6 > > HTTP/2: "http2" directive. > > > > The directive enables HTTP/2 in the current server. The previous way to > > enable HTTP/2 via "listen ... http2" is now deprecated. The new approach > > allows to share HTTP/2 and HTTP/0.9-1.1 on the same port. > > > > For SSL connections, HTTP/2 is now selected by ALPN callback based on whether > > the protocol is enabled in the virtual server chosen by SNI. This however only > > works since OpenSSL 1.0.2h, where ALPN callback is invoked after SNI callback. > > For older versions of OpenSSL, HTTP/2 is enabled based on the default virtual > > server configuration. > > > > For plain TCP connections, HTTP/2 is now auto-detected by HTTP/2 preface, if > > HTTP/2 is enabled in the default virtual server. If preface is not matched, > > HTTP/0.9-1.1 is assumed. > > > > diff --git a/src/http/modules/ngx_http_ssl_module.c b/src/http/modules/ngx_http_ssl_module.c > > --- a/src/http/modules/ngx_http_ssl_module.c > > +++ b/src/http/modules/ngx_http_ssl_module.c > > @@ -427,6 +427,9 @@ ngx_http_ssl_alpn_select(ngx_ssl_conn_t > > #if (NGX_HTTP_V2 || NGX_HTTP_V3) > > ngx_http_connection_t *hc; > > #endif > > +#if (NGX_HTTP_V2) > > + ngx_http_v2_srv_conf_t *h2scf; > > +#endif > > #if (NGX_HTTP_V3) > > ngx_http_v3_srv_conf_t *h3scf; > > #endif > > @@ -448,12 +451,9 @@ ngx_http_ssl_alpn_select(ngx_ssl_conn_t > > hc = c->data; > > #endif > > > > -#if (NGX_HTTP_V2) > > - if (hc->addr_conf->http2) { > > - srv = (unsigned char *) NGX_HTTP_V2_ALPN_PROTO NGX_HTTP_ALPN_PROTOS; > > - srvlen = sizeof(NGX_HTTP_V2_ALPN_PROTO NGX_HTTP_ALPN_PROTOS) - 1; > > - } else > > -#endif > > + srv = (unsigned char *) NGX_HTTP_ALPN_PROTOS; > > + srvlen = sizeof(NGX_HTTP_ALPN_PROTOS) - 1; > > + > > #if (NGX_HTTP_V3) > > if (hc->addr_conf->quic) { > > > > @@ -479,10 +479,16 @@ ngx_http_ssl_alpn_select(ngx_ssl_conn_t > > > > } else > > #endif > > +#if (NGX_HTTP_V2) > > { > > - srv = (unsigned char *) NGX_HTTP_ALPN_PROTOS; > > - srvlen = sizeof(NGX_HTTP_ALPN_PROTOS) - 1; > > + h2scf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_v2_module); > > + > > + if (h2scf->enable || hc->addr_conf->http2) { > > + srv = (unsigned char *) NGX_HTTP_V2_ALPN_PROTO NGX_HTTP_ALPN_PROTOS; > > + srvlen = sizeof(NGX_HTTP_V2_ALPN_PROTO NGX_HTTP_ALPN_PROTOS) - 1; > > + } > > } > > +#endif > > With NGX_HTTP_V3 defined but NGX_HTTP_V2 not, > the else part will go to the SSL_select_next_proto() call. > So, to fix this, NGX_HTTP_ALPN_PROTOS still has to be the last resort, > for simplicity (and also reduces diff). > > My version: > > diff --git a/src/http/modules/ngx_http_ssl_module.c b/src/http/modules/ngx_http_ssl_module.c > --- a/src/http/modules/ngx_http_ssl_module.c > +++ b/src/http/modules/ngx_http_ssl_module.c > @@ -427,6 +427,9 @@ ngx_http_ssl_alpn_select(ngx_ssl_conn_t > #if (NGX_HTTP_V2 || NGX_HTTP_V3) > ngx_http_connection_t *hc; > #endif > +#if (NGX_HTTP_V2) > + ngx_http_v2_srv_conf_t *h2scf; > +#endif > #if (NGX_HTTP_V3) > ngx_http_v3_srv_conf_t *h3scf; > #endif > @@ -449,7 +452,9 @@ ngx_http_ssl_alpn_select(ngx_ssl_conn_t > #endif > > #if (NGX_HTTP_V2) > - if (hc->addr_conf->http2) { > + h2scf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_v2_module); > + > + if (h2scf->enable || hc->addr_conf->http2) { > srv = (unsigned char *) NGX_HTTP_V2_ALPN_PROTO NGX_HTTP_ALPN_PROTOS; > srvlen = sizeof(NGX_HTTP_V2_ALPN_PROTO NGX_HTTP_ALPN_PROTOS) - 1; > } else > > > > > if (SSL_select_next_proto((unsigned char **) out, outlen, srv, srvlen, > > in, inlen) > Thanks for noticing this. If fixed this way, nginx will be able to negotiate HTTP/2 over QUIC, which is probably not what we want. Following a discussion with Sergey, here's an updated version of the patch. -- Roman Arutyunyan -------------- next part -------------- # HG changeset patch # User Roman Arutyunyan # Date 1675781276 -14400 # Tue Feb 07 18:47:56 2023 +0400 # Branch quic # Node ID 25c6c44600f6f0e2aa947e184790e04cedf4ff6d # Parent 36550cf579bc032586b5a1695c41b43c1f7e34d3 HTTP/2: "http2" directive. The directive enables HTTP/2 in the current server. The previous way to enable HTTP/2 via "listen ... http2" is now deprecated. The new approach allows to share HTTP/2 and HTTP/0.9-1.1 on the same port. For SSL connections, HTTP/2 is now selected by ALPN callback based on whether the protocol is enabled in the virtual server chosen by SNI. This however only works since OpenSSL 1.0.2h, where ALPN callback is invoked after SNI callback. For older versions of OpenSSL, HTTP/2 is enabled based on the default virtual server configuration. For plain TCP connections, HTTP/2 is now auto-detected by HTTP/2 preface, if HTTP/2 is enabled in the default virtual server. If preface is not matched, HTTP/0.9-1.1 is assumed. diff --git a/src/http/modules/ngx_http_ssl_module.c b/src/http/modules/ngx_http_ssl_module.c --- a/src/http/modules/ngx_http_ssl_module.c +++ b/src/http/modules/ngx_http_ssl_module.c @@ -427,6 +427,9 @@ ngx_http_ssl_alpn_select(ngx_ssl_conn_t #if (NGX_HTTP_V2 || NGX_HTTP_V3) ngx_http_connection_t *hc; #endif +#if (NGX_HTTP_V2) + ngx_http_v2_srv_conf_t *h2scf; +#endif #if (NGX_HTTP_V3) ngx_http_v3_srv_conf_t *h3scf; #endif @@ -448,12 +451,6 @@ ngx_http_ssl_alpn_select(ngx_ssl_conn_t hc = c->data; #endif -#if (NGX_HTTP_V2) - if (hc->addr_conf->http2) { - srv = (unsigned char *) NGX_HTTP_V2_ALPN_PROTO NGX_HTTP_ALPN_PROTOS; - srvlen = sizeof(NGX_HTTP_V2_ALPN_PROTO NGX_HTTP_ALPN_PROTOS) - 1; - } else -#endif #if (NGX_HTTP_V3) if (hc->addr_conf->quic) { @@ -480,8 +477,19 @@ ngx_http_ssl_alpn_select(ngx_ssl_conn_t } else #endif { - srv = (unsigned char *) NGX_HTTP_ALPN_PROTOS; - srvlen = sizeof(NGX_HTTP_ALPN_PROTOS) - 1; +#if (NGX_HTTP_V2) + h2scf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_v2_module); + + if (h2scf->enable || hc->addr_conf->http2) { + srv = (unsigned char *) NGX_HTTP_V2_ALPN_PROTO NGX_HTTP_ALPN_PROTOS; + srvlen = sizeof(NGX_HTTP_V2_ALPN_PROTO NGX_HTTP_ALPN_PROTOS) - 1; + + } else +#endif + { + srv = (unsigned char *) NGX_HTTP_ALPN_PROTOS; + srvlen = sizeof(NGX_HTTP_ALPN_PROTOS) - 1; + } } if (SSL_select_next_proto((unsigned char **) out, outlen, srv, srvlen, diff --git a/src/http/ngx_http_core_module.c b/src/http/ngx_http_core_module.c --- a/src/http/ngx_http_core_module.c +++ b/src/http/ngx_http_core_module.c @@ -4179,6 +4179,11 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx if (ngx_strcmp(value[n].data, "http2") == 0) { #if (NGX_HTTP_V2) + ngx_conf_log_error(NGX_LOG_WARN, cf, 0, + "the \"listen ... http2\" directive " + "is deprecated, use " + "the \"http2\" directive instead"); + lsopt.http2 = 1; continue; #else diff --git a/src/http/ngx_http_request.c b/src/http/ngx_http_request.c --- a/src/http/ngx_http_request.c +++ b/src/http/ngx_http_request.c @@ -318,12 +318,6 @@ ngx_http_init_connection(ngx_connection_ rev->handler = ngx_http_wait_request_handler; c->write->handler = ngx_http_empty_handler; -#if (NGX_HTTP_V2) - if (hc->addr_conf->http2) { - rev->handler = ngx_http_v2_init; - } -#endif - #if (NGX_HTTP_V3) if (hc->addr_conf->quic) { ngx_http_v3_init_stream(c); @@ -383,6 +377,9 @@ ngx_http_wait_request_handler(ngx_event_ ngx_buf_t *b; ngx_connection_t *c; ngx_http_connection_t *hc; +#if (NGX_HTTP_V2) + ngx_http_v2_srv_conf_t *h2scf; +#endif ngx_http_core_srv_conf_t *cscf; c = rev->data; @@ -429,6 +426,8 @@ ngx_http_wait_request_handler(ngx_event_ b->end = b->last + size; } + size = b->end - b->last; + n = c->recv(c, b->last, size); if (n == NGX_AGAIN) { @@ -443,12 +442,16 @@ ngx_http_wait_request_handler(ngx_event_ return; } - /* - * We are trying to not hold c->buffer's memory for an idle connection. - */ - - if (ngx_pfree(c->pool, b->start) == NGX_OK) { - b->start = NULL; + if (b->pos == b->last) { + + /* + * We are trying to not hold c->buffer's memory for an + * idle connection. + */ + + if (ngx_pfree(c->pool, b->start) == NGX_OK) { + b->start = NULL; + } } return; @@ -489,10 +492,34 @@ ngx_http_wait_request_handler(ngx_event_ } } + ngx_reusable_connection(c, 0); + +#if (NGX_HTTP_V2) + + h2scf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_v2_module); + + if (!c->ssl && (h2scf->enable || hc->addr_conf->http2)) { + + size = ngx_min(sizeof(NGX_HTTP_V2_PREFACE) - 1, + (size_t) (b->last - b->pos)); + + if (ngx_memcmp(b->pos, NGX_HTTP_V2_PREFACE, size) == 0) { + + if (size == sizeof(NGX_HTTP_V2_PREFACE) - 1) { + ngx_http_v2_init(rev); + return; + } + + c->log->action = "waiting for request"; + ngx_post_event(rev, &ngx_posted_events); + return; + } + } + +#endif + c->log->action = "reading client request line"; - ngx_reusable_connection(c, 0); - c->data = ngx_http_create_request(c); if (c->data == NULL) { ngx_http_close_connection(c); @@ -808,13 +835,16 @@ ngx_http_ssl_handshake_handler(ngx_conne #if (NGX_HTTP_V2 \ && defined TLSEXT_TYPE_application_layer_protocol_negotiation) { - unsigned int len; - const unsigned char *data; - ngx_http_connection_t *hc; + unsigned int len; + const unsigned char *data; + ngx_http_connection_t *hc; + ngx_http_v2_srv_conf_t *h2scf; hc = c->data; - if (hc->addr_conf->http2) { + h2scf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_v2_module); + + if (h2scf->enable || hc->addr_conf->http2) { SSL_get0_alpn_selected(c->ssl->connection, &data, &len); diff --git a/src/http/v2/ngx_http_v2.c b/src/http/v2/ngx_http_v2.c --- a/src/http/v2/ngx_http_v2.c +++ b/src/http/v2/ngx_http_v2.c @@ -63,8 +63,6 @@ static void ngx_http_v2_handle_connectio static void ngx_http_v2_lingering_close(ngx_connection_t *c); static void ngx_http_v2_lingering_close_handler(ngx_event_t *rev); -static u_char *ngx_http_v2_state_proxy_protocol(ngx_http_v2_connection_t *h2c, - u_char *pos, u_char *end); static u_char *ngx_http_v2_state_preface(ngx_http_v2_connection_t *h2c, u_char *pos, u_char *end); static u_char *ngx_http_v2_state_preface_end(ngx_http_v2_connection_t *h2c, @@ -232,6 +230,7 @@ static ngx_http_v2_parse_header_t ngx_h void ngx_http_v2_init(ngx_event_t *rev) { + u_char *p, *end; ngx_connection_t *c; ngx_pool_cleanup_t *cln; ngx_http_connection_t *hc; @@ -314,8 +313,7 @@ ngx_http_v2_init(ngx_event_t *rev) return; } - h2c->state.handler = hc->proxy_protocol ? ngx_http_v2_state_proxy_protocol - : ngx_http_v2_state_preface; + h2c->state.handler = ngx_http_v2_state_preface; ngx_queue_init(&h2c->waiting); ngx_queue_init(&h2c->dependencies); @@ -333,7 +331,23 @@ ngx_http_v2_init(ngx_event_t *rev) } c->idle = 1; - ngx_reusable_connection(c, 0); + + if (c->buffer) { + p = c->buffer->pos; + end = c->buffer->last; + + do { + p = h2c->state.handler(h2c, p, end); + + if (p == NULL) { + return; + } + + } while (p != end); + + h2c->total_bytes += p - c->buffer->pos; + c->buffer->pos = p; + } ngx_http_v2_read_handler(rev); } @@ -847,31 +861,10 @@ ngx_http_v2_lingering_close_handler(ngx_ static u_char * -ngx_http_v2_state_proxy_protocol(ngx_http_v2_connection_t *h2c, u_char *pos, - u_char *end) -{ - ngx_log_t *log; - - log = h2c->connection->log; - log->action = "reading PROXY protocol"; - - pos = ngx_proxy_protocol_read(h2c->connection, pos, end); - - log->action = "processing HTTP/2 connection"; - - if (pos == NULL) { - return ngx_http_v2_connection_error(h2c, NGX_HTTP_V2_PROTOCOL_ERROR); - } - - return ngx_http_v2_state_preface(h2c, pos, end); -} - - -static u_char * ngx_http_v2_state_preface(ngx_http_v2_connection_t *h2c, u_char *pos, u_char *end) { - static const u_char preface[] = "PRI * HTTP/2.0\r\n"; + static const u_char preface[] = NGX_HTTP_V2_PREFACE_START; if ((size_t) (end - pos) < sizeof(preface) - 1) { return ngx_http_v2_state_save(h2c, pos, end, ngx_http_v2_state_preface); @@ -892,7 +885,7 @@ static u_char * ngx_http_v2_state_preface_end(ngx_http_v2_connection_t *h2c, u_char *pos, u_char *end) { - static const u_char preface[] = "\r\nSM\r\n\r\n"; + static const u_char preface[] = NGX_HTTP_V2_PREFACE_END; if ((size_t) (end - pos) < sizeof(preface) - 1) { return ngx_http_v2_state_save(h2c, pos, end, @@ -3946,10 +3939,22 @@ static void ngx_http_v2_run_request(ngx_http_request_t *r) { ngx_connection_t *fc; + ngx_http_v2_srv_conf_t *h2scf; ngx_http_v2_connection_t *h2c; fc = r->connection; + h2scf = ngx_http_get_module_srv_conf(r, ngx_http_v2_module); + + if (!h2scf->enable && !r->http_connection->addr_conf->http2) { + ngx_log_error(NGX_LOG_INFO, fc->log, 0, + "client attempted to request the server name " + "for which the negotiated protocol is disabled"); + + ngx_http_finalize_request(r, NGX_HTTP_MISDIRECTED_REQUEST); + goto failed; + } + if (ngx_http_v2_construct_request_line(r) != NGX_OK) { goto failed; } diff --git a/src/http/v2/ngx_http_v2.h b/src/http/v2/ngx_http_v2.h --- a/src/http/v2/ngx_http_v2.h +++ b/src/http/v2/ngx_http_v2.h @@ -64,6 +64,16 @@ typedef u_char *(*ngx_http_v2_handler_pt typedef struct { + ngx_flag_t enable; + size_t pool_size; + ngx_uint_t concurrent_streams; + ngx_uint_t concurrent_pushes; + size_t preread_size; + ngx_uint_t streams_index_mask; +} ngx_http_v2_srv_conf_t; + + +typedef struct { ngx_str_t name; ngx_str_t value; } ngx_http_v2_header_t; @@ -408,9 +418,17 @@ ngx_int_t ngx_http_v2_table_size(ngx_htt #define NGX_HTTP_V2_USER_AGENT_INDEX 58 #define NGX_HTTP_V2_VARY_INDEX 59 +#define NGX_HTTP_V2_PREFACE_START "PRI * HTTP/2.0\r\n" +#define NGX_HTTP_V2_PREFACE_END "\r\nSM\r\n\r\n" +#define NGX_HTTP_V2_PREFACE NGX_HTTP_V2_PREFACE_START \ + NGX_HTTP_V2_PREFACE_END + u_char *ngx_http_v2_string_encode(u_char *dst, u_char *src, size_t len, u_char *tmp, ngx_uint_t lower); +extern ngx_module_t ngx_http_v2_module; + + #endif /* _NGX_HTTP_V2_H_INCLUDED_ */ diff --git a/src/http/v2/ngx_http_v2_module.c b/src/http/v2/ngx_http_v2_module.c --- a/src/http/v2/ngx_http_v2_module.c +++ b/src/http/v2/ngx_http_v2_module.c @@ -75,6 +75,13 @@ static ngx_conf_post_t ngx_http_v2_chun static ngx_command_t ngx_http_v2_commands[] = { + { ngx_string("http2"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_CONF_FLAG, + ngx_conf_set_flag_slot, + NGX_HTTP_SRV_CONF_OFFSET, + offsetof(ngx_http_v2_srv_conf_t, enable), + NULL }, + { ngx_string("http2_recv_buffer_size"), NGX_HTTP_MAIN_CONF|NGX_CONF_TAKE1, ngx_conf_set_size_slot, @@ -314,6 +321,8 @@ ngx_http_v2_create_srv_conf(ngx_conf_t * return NULL; } + h2scf->enable = NGX_CONF_UNSET; + h2scf->pool_size = NGX_CONF_UNSET_SIZE; h2scf->concurrent_streams = NGX_CONF_UNSET_UINT; @@ -333,6 +342,8 @@ ngx_http_v2_merge_srv_conf(ngx_conf_t *c ngx_http_v2_srv_conf_t *prev = parent; ngx_http_v2_srv_conf_t *conf = child; + ngx_conf_merge_value(conf->enable, prev->enable, 0); + ngx_conf_merge_size_value(conf->pool_size, prev->pool_size, 4096); ngx_conf_merge_uint_value(conf->concurrent_streams, diff --git a/src/http/v2/ngx_http_v2_module.h b/src/http/v2/ngx_http_v2_module.h --- a/src/http/v2/ngx_http_v2_module.h +++ b/src/http/v2/ngx_http_v2_module.h @@ -21,15 +21,6 @@ typedef struct { typedef struct { - size_t pool_size; - ngx_uint_t concurrent_streams; - ngx_uint_t concurrent_pushes; - size_t preread_size; - ngx_uint_t streams_index_mask; -} ngx_http_v2_srv_conf_t; - - -typedef struct { size_t chunk_size; ngx_flag_t push_preload; @@ -39,7 +30,4 @@ typedef struct { } ngx_http_v2_loc_conf_t; -extern ngx_module_t ngx_http_v2_module; - - #endif /* _NGX_HTTP_V2_MODULE_H_INCLUDED_ */ From arut at nginx.com Thu Feb 9 12:33:59 2023 From: arut at nginx.com (Roman Arutyunyan) Date: Thu, 9 Feb 2023 16:33:59 +0400 Subject: [PATCH 2 of 3] HTTP/2: "http2" directive In-Reply-To: <20230209120234.rttyjviwir4sneed@N00W24XTQX> References: <735f9e501922e4b0a1b2.1675781431@arut-laptop> <20230209120234.rttyjviwir4sneed@N00W24XTQX> Message-ID: <20230209123359.i6fosl5r6tdhpvhv@N00W24XTQX> Hi, On Thu, Feb 09, 2023 at 04:02:34PM +0400, Roman Arutyunyan wrote: [..] > diff --git a/src/http/ngx_http_request.c b/src/http/ngx_http_request.c > --- a/src/http/ngx_http_request.c > +++ b/src/http/ngx_http_request.c > @@ -318,12 +318,6 @@ ngx_http_init_connection(ngx_connection_ > rev->handler = ngx_http_wait_request_handler; > c->write->handler = ngx_http_empty_handler; > > -#if (NGX_HTTP_V2) > - if (hc->addr_conf->http2) { > - rev->handler = ngx_http_v2_init; > - } > -#endif > - > #if (NGX_HTTP_V3) > if (hc->addr_conf->quic) { > ngx_http_v3_init_stream(c); > @@ -383,6 +377,9 @@ ngx_http_wait_request_handler(ngx_event_ > ngx_buf_t *b; > ngx_connection_t *c; > ngx_http_connection_t *hc; > +#if (NGX_HTTP_V2) > + ngx_http_v2_srv_conf_t *h2scf; > +#endif > ngx_http_core_srv_conf_t *cscf; > > c = rev->data; > @@ -429,6 +426,8 @@ ngx_http_wait_request_handler(ngx_event_ > b->end = b->last + size; > } > > + size = b->end - b->last; > + > n = c->recv(c, b->last, size); > > if (n == NGX_AGAIN) { > @@ -443,12 +442,16 @@ ngx_http_wait_request_handler(ngx_event_ > return; > } > > - /* > - * We are trying to not hold c->buffer's memory for an idle connection. > - */ > - > - if (ngx_pfree(c->pool, b->start) == NGX_OK) { > - b->start = NULL; > + if (b->pos == b->last) { > + > + /* > + * We are trying to not hold c->buffer's memory for an > + * idle connection. > + */ > + > + if (ngx_pfree(c->pool, b->start) == NGX_OK) { > + b->start = NULL; > + } > } > > return; > @@ -489,10 +492,34 @@ ngx_http_wait_request_handler(ngx_event_ > } > } > > + ngx_reusable_connection(c, 0); > + > +#if (NGX_HTTP_V2) > + > + h2scf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_v2_module); > + > + if (!c->ssl && (h2scf->enable || hc->addr_conf->http2)) { And one more fix for compilation with HTTP/2, but without SSL: diff --git a/src/http/ngx_http_request.c b/src/http/ngx_http_request.c --- a/src/http/ngx_http_request.c +++ b/src/http/ngx_http_request.c @@ -498,8 +498,12 @@ ngx_http_wait_request_handler(ngx_event_ h2scf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_v2_module); - if (!c->ssl && (h2scf->enable || hc->addr_conf->http2)) { - + if ((h2scf->enable || hc->addr_conf->http2) +#if (NGX_HTTP_SSL) + && !c->ssl +#endif + ) + { size = ngx_min(sizeof(NGX_HTTP_V2_PREFACE) - 1, (size_t) (b->last - b->pos)); > + > + size = ngx_min(sizeof(NGX_HTTP_V2_PREFACE) - 1, > + (size_t) (b->last - b->pos)); > + > + if (ngx_memcmp(b->pos, NGX_HTTP_V2_PREFACE, size) == 0) { > + > + if (size == sizeof(NGX_HTTP_V2_PREFACE) - 1) { > + ngx_http_v2_init(rev); > + return; > + } > + > + c->log->action = "waiting for request"; > + ngx_post_event(rev, &ngx_posted_events); > + return; > + } > + } > + > +#endif > + > c->log->action = "reading client request line"; > > - ngx_reusable_connection(c, 0); > - > c->data = ngx_http_create_request(c); > if (c->data == NULL) { > ngx_http_close_connection(c); [..] -- Roman Arutyunyan From pluknet at nginx.com Thu Feb 9 15:56:55 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Thu, 9 Feb 2023 19:56:55 +0400 Subject: [PATCH 2 of 3] HTTP/2: "http2" directive In-Reply-To: <20230209123359.i6fosl5r6tdhpvhv@N00W24XTQX> References: <735f9e501922e4b0a1b2.1675781431@arut-laptop> <20230209120234.rttyjviwir4sneed@N00W24XTQX> <20230209123359.i6fosl5r6tdhpvhv@N00W24XTQX> Message-ID: <71CADCEB-5BD1-4F67-AE85-581943918D5B@nginx.com> > On 9 Feb 2023, at 16:33, Roman Arutyunyan wrote: > > Hi, > > On Thu, Feb 09, 2023 at 04:02:34PM +0400, Roman Arutyunyan wrote: > > [..] > >> diff --git a/src/http/ngx_http_request.c b/src/http/ngx_http_request.c >> --- a/src/http/ngx_http_request.c >> +++ b/src/http/ngx_http_request.c >> @@ -318,12 +318,6 @@ ngx_http_init_connection(ngx_connection_ >> rev->handler = ngx_http_wait_request_handler; >> c->write->handler = ngx_http_empty_handler; >> >> -#if (NGX_HTTP_V2) >> - if (hc->addr_conf->http2) { >> - rev->handler = ngx_http_v2_init; >> - } >> -#endif >> - >> #if (NGX_HTTP_V3) >> if (hc->addr_conf->quic) { >> ngx_http_v3_init_stream(c); >> @@ -383,6 +377,9 @@ ngx_http_wait_request_handler(ngx_event_ >> ngx_buf_t *b; >> ngx_connection_t *c; >> ngx_http_connection_t *hc; >> +#if (NGX_HTTP_V2) >> + ngx_http_v2_srv_conf_t *h2scf; >> +#endif >> ngx_http_core_srv_conf_t *cscf; >> >> c = rev->data; >> @@ -429,6 +426,8 @@ ngx_http_wait_request_handler(ngx_event_ >> b->end = b->last + size; >> } >> >> + size = b->end - b->last; >> + >> n = c->recv(c, b->last, size); >> >> if (n == NGX_AGAIN) { >> @@ -443,12 +442,16 @@ ngx_http_wait_request_handler(ngx_event_ >> return; >> } >> >> - /* >> - * We are trying to not hold c->buffer's memory for an idle connection. >> - */ >> - >> - if (ngx_pfree(c->pool, b->start) == NGX_OK) { >> - b->start = NULL; >> + if (b->pos == b->last) { >> + >> + /* >> + * We are trying to not hold c->buffer's memory for an >> + * idle connection. >> + */ >> + >> + if (ngx_pfree(c->pool, b->start) == NGX_OK) { >> + b->start = NULL; >> + } >> } >> >> return; >> @@ -489,10 +492,34 @@ ngx_http_wait_request_handler(ngx_event_ >> } >> } >> >> + ngx_reusable_connection(c, 0); >> + >> +#if (NGX_HTTP_V2) >> + >> + h2scf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_v2_module); >> + >> + if (!c->ssl && (h2scf->enable || hc->addr_conf->http2)) { > > And one more fix for compilation with HTTP/2, but without SSL: > > diff --git a/src/http/ngx_http_request.c b/src/http/ngx_http_request.c > --- a/src/http/ngx_http_request.c > +++ b/src/http/ngx_http_request.c > @@ -498,8 +498,12 @@ ngx_http_wait_request_handler(ngx_event_ > > h2scf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_v2_module); > > - if (!c->ssl && (h2scf->enable || hc->addr_conf->http2)) { > - > + if ((h2scf->enable || hc->addr_conf->http2) > +#if (NGX_HTTP_SSL) > + && !c->ssl > +#endif > + ) > + { > size = ngx_min(sizeof(NGX_HTTP_V2_PREFACE) - 1, > (size_t) (b->last - b->pos)); > I think this test needs to be replaced with !hc->ssl. Otherwise, it would allow to establish (and keep) h2c on ssl-enabled sockets, which we likely do not want to allow. > >> + >> + size = ngx_min(sizeof(NGX_HTTP_V2_PREFACE) - 1, >> + (size_t) (b->last - b->pos)); >> + >> + if (ngx_memcmp(b->pos, NGX_HTTP_V2_PREFACE, size) == 0) { >> + >> + if (size == sizeof(NGX_HTTP_V2_PREFACE) - 1) { >> + ngx_http_v2_init(rev); >> + return; >> + } >> + >> + c->log->action = "waiting for request"; >> + ngx_post_event(rev, &ngx_posted_events); >> + return; >> + } >> + } >> + >> +#endif >> + >> c->log->action = "reading client request line"; >> >> - ngx_reusable_connection(c, 0); >> - >> c->data = ngx_http_create_request(c); >> if (c->data == NULL) { >> ngx_http_close_connection(c); > > [..] > -- Sergey Kandaurov From a.bavshin at nginx.com Thu Feb 9 16:45:11 2023 From: a.bavshin at nginx.com (Aleksei Bavshin) Date: Thu, 9 Feb 2023 08:45:11 -0800 Subject: [PATCH 5 of 6] Upstream: allow any worker to resolve upstream servers In-Reply-To: References: Message-ID: <887bbc2c-9d0f-0414-88e5-b04ca70218ab@nginx.com> On 2/5/2023 7:01 PM, J Carter wrote: > Hi Aleksei, > > Why not permanently assign the task of resolving a given upstream server > group (all servers/peers within it) to a single worker? > > It seems that this approach would resolve the SRV issues, and remove the > need for the shared queue of tasks. > > The load would still be spread evenly for the most realistic scenarios - > which is where there are many upstream server groups of few servers, as > opposed to few upstream server groups of many servers. The intent of the change was exactly opposite, to avoid any permanent assignment of periodic tasks to a worker and allow another processes to resume resolving if the original assignee exits, no matter if normally or abnormally. I'm not even doing enough for that -- I should've kept in-progress tasks at the end of the queue with expires = resolver timeout + a small constant, and retry from another process when the timeout is reached, but the idea was abandoned for a minuscule improvement of insertion time. I expect to be asked to reconsider, as patch 6/6 does not cover all the possible situations where we want to recover a stale task. A permanent assignment of a whole upstream would also require notifying another processes that the upstream is no longer assigned if the worker exits or consistently recovering that assignment over a restart of single worker (e.g. after a crash - not a regular situation, but one we should take into account nonetheless). And the benefit is not quite obvious - I mentioned that resolving SRVs with a lot of records may take longer to update the list of peers, but the situation with contention is not expected to change significantly* if we pin these tasks to a single worker as another worker may be doing the same for another upstream. Most importantly, this isn't even a bottleneck. It only slightly exacerbates an existing problem with certain balancers that already suffer from the overuse of locks, in a configuration that was specifically crafted to amplify and highlight the difference and is far from these most realistic scenarios. * Pending verification on a performance test stand. From jordanc.carter at outlook.com Fri Feb 10 00:04:59 2023 From: jordanc.carter at outlook.com (J Carter) Date: Fri, 10 Feb 2023 00:04:59 +0000 Subject: [PATCH 5 of 6] Upstream: allow any worker to resolve upstream servers In-Reply-To: <887bbc2c-9d0f-0414-88e5-b04ca70218ab@nginx.com> References: <887bbc2c-9d0f-0414-88e5-b04ca70218ab@nginx.com> Message-ID: On 09/02/2023 16:45, Aleksei Bavshin wrote: > On 2/5/2023 7:01 PM, J Carter wrote: >> Hi Aleksei, >> >> Why not permanently assign the task of resolving a given upstream >> server group (all servers/peers within it) to a single worker? >> >> It seems that this approach would resolve the SRV issues, and remove >> the need for the shared queue of tasks. >> >> The load would still be spread evenly for the most realistic >> scenarios - which is where there are many upstream server groups of >> few servers, as opposed to few upstream server groups of many servers. > > The intent of the change was exactly opposite, to avoid any permanent > assignment of periodic tasks to a worker and allow another processes > to resume resolving if the original assignee exits, no matter if > normally or abnormally. I'm not even doing enough for that -- I > should've kept in-progress tasks at the end of the queue with expires > = resolver timeout + a small constant, and retry from another process > when the timeout is reached, but the idea was abandoned for a > minuscule improvement of insertion time. I expect to be asked to > reconsider, as patch 6/6 does not cover all the possible situations > where we want to recover a stale task. Makes sense. > A permanent assignment of a whole upstream would also require > notifying another processes that the upstream is no longer assigned if > the worker exits or consistently recovering that assignment over a > restart of single worker (e.g. after a crash - not a regular > situation, but one we should take into account nonetheless). It's a good point, in my mind I had rendezvous hashing + a notification sent to all workers when a fellow worker dies - the next worker in the rendezvous 'list' would simply assume the dead worker's upstreams while the new one inits, and share it back once the replacement worker is up (would still use some locks). Or to keep it simple, just wait for the dead worker's replacement to reinit, and pick up the former's stale upstreams. > And the benefit is not quite obvious - I mentioned that resolving SRVs > with a lot of records may take longer to update the list of peers, but > the situation with contention is not expected to change significantly* > if we pin these tasks to a single worker as another worker may be > doing the same for another upstream. Most importantly, this isn't even > a bottleneck. It only slightly exacerbates an existing problem with > certain balancers that already suffer from the overuse of locks, in a > configuration that was specifically crafted to amplify and highlight > the difference and is far from these most realistic scenarios. > * Pending verification on a performance test stand. Well the benefit is that it would prevent the disadvantage you listed, and remove at least one other contended lock throughout normal operations (the priority queue). But fair enough, yes it makes sense to profile it in a wide range of scenarios to see if it's any of those are legitimate worries first. > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > https://mailman.nginx.org/mailman/listinfo/nginx-devel From mdounin at mdounin.ru Fri Feb 10 04:33:01 2023 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 10 Feb 2023 07:33:01 +0300 Subject: [PATCH 2 of 4] Win32: handling of localized MSVC cl output In-Reply-To: References: <43098cb134a87a404b70.1671543016@1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa> Message-ID: Hello! On Thu, Feb 09, 2023 at 06:55:22AM +0000, Pavel Pautov via nginx-devel wrote: > The 'for 80x86' part of cl version (in auto/configure output) > always felt odd. Given it's used in subsequent patch, can we at > least strip localized 'for' from NGX_MSVC_VER? Having it in > otherwise non-localized output of "nginx -V" doesn't seem right > either. The "nginx -V" output simply shows how the compiler identifies itself. The patch has no intention to change this, it just ensures that output of localized MSVC cl versions is recognized by nginx. And, actually, I would rather object stripping the details, since any aspects of compiler identification might be important. -- Maxim Dounin http://mdounin.ru/ From xeioex at nginx.com Fri Feb 10 05:07:27 2023 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Fri, 10 Feb 2023 05:07:27 +0000 Subject: [njs] Version bump. Message-ID: details: https://hg.nginx.org/njs/rev/4da29a050b6e branches: changeset: 2044:4da29a050b6e user: Dmitry Volyntsev date: Thu Feb 09 18:34:46 2023 -0800 description: Version bump. diffstat: src/njs.h | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diffs (14 lines): diff -r 083feee041d0 -r 4da29a050b6e src/njs.h --- a/src/njs.h Tue Feb 07 09:03:04 2023 -0800 +++ b/src/njs.h Thu Feb 09 18:34:46 2023 -0800 @@ -11,8 +11,8 @@ #include -#define NJS_VERSION "0.7.10" -#define NJS_VERSION_NUMBER 0x00070a +#define NJS_VERSION "0.7.11" +#define NJS_VERSION_NUMBER 0x00070b #include /* STDOUT_FILENO, STDERR_FILENO */ From xeioex at nginx.com Fri Feb 10 05:07:29 2023 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Fri, 10 Feb 2023 05:07:29 +0000 Subject: [njs] Types: fixed Fetch API types. Message-ID: details: https://hg.nginx.org/njs/rev/12a6d0517ae8 branches: changeset: 2045:12a6d0517ae8 user: Dmitry Volyntsev date: Thu Feb 09 18:34:51 2023 -0800 description: Types: fixed Fetch API types. diffstat: test/ts/test.ts | 3 +++ ts/ngx_core.d.ts | 4 ++-- 2 files changed, 5 insertions(+), 2 deletions(-) diffs (32 lines): diff -r 4da29a050b6e -r 12a6d0517ae8 test/ts/test.ts --- a/test/ts/test.ts Thu Feb 09 18:34:46 2023 -0800 +++ b/test/ts/test.ts Thu Feb 09 18:34:51 2023 -0800 @@ -91,6 +91,9 @@ async function http_module(r: NginxHTTPR throw 'oops' }; + let out: Array = reply.headers.getAll("foo"); + let has: boolean = reply.headers.has("foo"); + return reply.text() }) .then(body => r.return(200, body)) diff -r 4da29a050b6e -r 12a6d0517ae8 ts/ngx_core.d.ts --- a/ts/ngx_core.d.ts Thu Feb 09 18:34:46 2023 -0800 +++ b/ts/ngx_core.d.ts Thu Feb 09 18:34:51 2023 -0800 @@ -10,13 +10,13 @@ interface NgxResponseHeaders { * with the specified name. * @param name A name of the header. */ - getAll(name:NjsStringLike): NjsByteString; + getAll(name:NjsStringLike): Array; /** * Returns a boolean value indicating whether a header with * the specified name exists. * @param name A name of the header. */ - has(name:NjsStringLike): NjsByteString; + has(name:NjsStringLike): boolean; } interface NgxResponse { From xeioex at nginx.com Fri Feb 10 05:07:31 2023 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Fri, 10 Feb 2023 05:07:31 +0000 Subject: [njs] Types: added forgotten r.rawHeadersIn and r.rawHeadersOut description. Message-ID: details: https://hg.nginx.org/njs/rev/8b0350c4b19e branches: changeset: 2046:8b0350c4b19e user: Dmitry Volyntsev date: Thu Feb 09 18:34:51 2023 -0800 description: Types: added forgotten r.rawHeadersIn and r.rawHeadersOut description. diffstat: test/ts/test.ts | 2 ++ ts/ngx_http_js_module.d.ts | 11 +++++++++++ ts/njs_core.d.ts | 6 ++++++ 3 files changed, 19 insertions(+), 0 deletions(-) diffs (48 lines): diff -r 12a6d0517ae8 -r 8b0350c4b19e test/ts/test.ts --- a/test/ts/test.ts Thu Feb 09 18:34:51 2023 -0800 +++ b/test/ts/test.ts Thu Feb 09 18:34:51 2023 -0800 @@ -45,6 +45,8 @@ async function http_module(r: NginxHTTPR r.headersOut['Set-Cookie'] = ['aaa', 'bbb']; r.headersOut['Foo'] = ['aaa', 'bbb']; + let values: Array = r.rawHeadersIn.filter(v=>v[0].toLowerCase() == 'foo').map(v=>v[1]); + // r.log r.log(bs); diff -r 12a6d0517ae8 -r 8b0350c4b19e ts/ngx_http_js_module.d.ts --- a/ts/ngx_http_js_module.d.ts Thu Feb 09 18:34:51 2023 -0800 +++ b/ts/ngx_http_js_module.d.ts Thu Feb 09 18:34:51 2023 -0800 @@ -339,6 +339,17 @@ interface NginxHTTPRequest { */ readonly parent?: NginxHTTPRequest; /** + * An array of key-value pairs exactly as they were received from the client. + * @since 0.4.1 + */ + readonly rawHeadersIn: [NjsFixedSizeArray<2, NjsStringLike>]; + /** + * An array of key-value pairs of response headers. + * Header field names are not converted to lower case, duplicate field values are not merged. + * @since 0.4.1 + */ + readonly rawHeadersOut: [NjsFixedSizeArray<2, NjsStringLike>]; + /** * Client address. */ readonly remoteAddress: NjsByteString; diff -r 12a6d0517ae8 -r 8b0350c4b19e ts/njs_core.d.ts --- a/ts/njs_core.d.ts Thu Feb 09 18:34:51 2023 -0800 +++ b/ts/njs_core.d.ts Thu Feb 09 18:34:51 2023 -0800 @@ -1,5 +1,11 @@ type BufferEncoding = "utf8" | "hex" | "base64" | "base64url"; +type NjsFixedSizeArray = N extends 0 ? never[] : { + 0: T; + length: N; +} & ReadonlyArray; + + interface StringConstructor { /** * Creates a byte string from an encoded string. From xeioex at nginx.com Fri Feb 10 05:07:33 2023 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Fri, 10 Feb 2023 05:07:33 +0000 Subject: [njs] Types: updated Fetch API types. Message-ID: details: https://hg.nginx.org/njs/rev/29fbf8f85c09 branches: changeset: 2047:29fbf8f85c09 user: Dmitry Volyntsev date: Thu Feb 09 18:34:51 2023 -0800 description: Types: updated Fetch API types. diffstat: test/ts/test.ts | 9 +++ ts/ngx_core.d.ts | 153 ++++++++++++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 154 insertions(+), 8 deletions(-) diffs (231 lines): diff -r 8b0350c4b19e -r 29fbf8f85c09 test/ts/test.ts --- a/test/ts/test.ts Thu Feb 09 18:34:51 2023 -0800 +++ b/test/ts/test.ts Thu Feb 09 18:34:51 2023 -0800 @@ -96,12 +96,21 @@ async function http_module(r: NginxHTTPR let out: Array = reply.headers.getAll("foo"); let has: boolean = reply.headers.has("foo"); + reply.headers.append("foo", "xxx"); + reply.headers.delete("xxx"); + reply.headers.forEach((name, value) => { /* do something. */ }); + return reply.text() }) .then(body => r.return(200, body)) .catch(e => r.return(501, e.message)) + let response = await ngx.fetch('http://nginx.org/'); + let response2 = new Response("xxx", {headers: {"Content-Type": "text/plain"}, status: 404}); + + let req = new Request("http://nginx.org", {method: "POST", headers: new Headers(["Foo", "bar"])}); + let response3 = await ngx.fetch(req); // js_body_filter r.sendBuffer(Buffer.from("xxx"), {last:true}); diff -r 8b0350c4b19e -r 29fbf8f85c09 ts/ngx_core.d.ts --- a/ts/ngx_core.d.ts Thu Feb 09 18:34:51 2023 -0800 +++ b/ts/ngx_core.d.ts Thu Feb 09 18:34:51 2023 -0800 @@ -1,4 +1,28 @@ -interface NgxResponseHeaders { +type NgxHeaders = Headers | Object | [NjsFixedSizeArray<2, NjsStringLike>]; + +declare class Headers { + /** + * Appends a new value into an existing header in the Headers object, + * or adds the header if it does not already exist. + * @param name A name of the header. + * @param value A value of the header. + * @since 0.7.10 + */ + append(name:NjsStringLike, value: NjsStringLike): void; + /** + * Headers constructors. + * + * @param init is an optional initialization object. + * @returns returns Headers object. + * @since 0.7.10 + */ + constructor(init?: Object | [NjsFixedSizeArray<2, NjsStringLike>]); + /** + * Deletes a header from the Headers object. + * @param name A name of the header to be deleted. + * @since 0.7.10 + */ + delete(name:NjsStringLike): void; /** * Returns a string containing the values of all headers * with the specified name separated by a comma and a space. @@ -12,14 +36,119 @@ interface NgxResponseHeaders { */ getAll(name:NjsStringLike): Array; /** + * Executes a provided function once for each key/value + * pair in the Headers object. + * @param fn the function to be envoked. + * @since 0.7.10 + */ + forEach(fn:(name: NjsStringLike, value: NjsStringLike) => void): void; + /** * Returns a boolean value indicating whether a header with * the specified name exists. * @param name A name of the header. */ has(name:NjsStringLike): boolean; + /** + * Sets a new value for an existing header inside the Headers object, + * or adds the header if it does not already exist. + * @param name A name of the header. + * @param value A value of the header. + * @since 0.7.10 + */ + set(name:NjsStringLike, value: NjsStringLike): void; +} + +interface NgxRequestOptions { + /** + * Request body, by default is empty. + */ + body?: NjsStringLike; + /** + * Cache mode, by default is "default". + */ + cache?: "default" | "no-store" | "reload" | "no-cache" | "force-cache" | "only-if-cached"; + /** + * Credentials, by default is "same-origin". + */ + credentials?: "omit" | "same-origin" | "include"; + /** + * Request headers. + */ + headers?: NgxHeaders; + /** + * Request method, by default the GET method is used. + */ + method?: NjsStringLike; + /** + * Mode, by default is "no-cors". + */ + mode?: "same-origin" | "no-cors" | "cors"; } -interface NgxResponse { +declare class Request { + /** + * Returns a Promise that resolves with an body as ArrayBuffer. + */ + arrayBuffer(): Promise; + /** + * A boolean value, true if the body has been used. + */ + readonly bodyUsed: boolean; + /** + * Cache mode. + */ + readonly cache: NjsByteString; + /** + * Request constructors. + * + * @param init is an optional initialization object. + * @returns returns Headers object. + * @since 0.7.10 + */ + constructor(input: NjsStringLike | Request, options?: NgxRequestOptions); + /** + * Credentials. + */ + readonly credentials: NjsByteString; + /** + * Returns a Promise that resolves with an result of applying of + * JSON.parse() to a body. + */ + json(): Promise; + /** + * The Headers object associated with the request. + */ + headers: Headers; + /** + * Request mode. + */ + readonly mode: NjsByteString; + /** + * Returns a Promise that resolves with an body as String. + */ + text(): Promise; + /** + * Request url. + */ + readonly url: NjsByteString; +} + +interface NgxResponseOptions { + /** + * Request headers. + */ + headers?: NgxHeaders; + /** + * Response status, 200 by default. + */ + status?: number; + /** + * Response status test, '' by default. + */ + statusText?: NjsStringLike; +} + +declare class Response { /** * Takes a Response stream and reads it to completion. * Returns a Promise that resolves with an ArrayBuffer. @@ -30,15 +159,23 @@ interface NgxResponse { */ readonly bodyUsed: boolean; /** + * Response constructors. + * + * @param init is an optional initialization object. + * @returns returns Headers object. + * @since 0.7.10 + */ + constructor(body?: NjsStringLike, options?: NgxResponseOptions); + /** * Takes a Response stream and reads it to completion. * Returns a Promise that resolves with the result of * parsing the body text as JSON. */ json(): Promise; /** - * The NgxResponseHeaders object associated with the response. + * The Headers object associated with the response. */ - headers: NgxResponseHeaders; + headers: Headers; /** * A boolean value, true if the response was successful * (status in the range 200-299). @@ -86,7 +223,7 @@ interface NgxFetchOptions { /** * Request headers object. */ - headers?: Object, + headers?: NgxHeaders; /** * The maximum size of the response body in bytes, by default is 1048576 (32768 before 0.7.4). * Nginx specific. @@ -118,13 +255,13 @@ interface NgxObject { log(level: number, message: NjsStringOrBuffer): void; /** * Makes a request to fetch an URL. - * Returns a Promise that resolves with the NgxResponse object. + * Returns a Promise that resolves with the Response object. * Since 0.7.0 HTTPS is supported, redirects are not handled. - * @param url URL of a resource to fetch. + * @param init URL of a resource to fetch or a Request object. * @param options An object containing additional settings. * @since 0.5.1 */ - fetch(url: NjsStringOrBuffer, options?: NgxFetchOptions): Promise; + fetch(init: NjsStringOrBuffer | Request, options?: NgxFetchOptions): Promise; } declare const ngx: NgxObject; From jerome at loyet.net Fri Feb 10 09:04:35 2023 From: jerome at loyet.net (=?UTF-8?B?SsOpcsO0bWUgTG95ZXQ=?=) Date: Fri, 10 Feb 2023 10:04:35 +0100 Subject: switch to ms resolution for rate limiting In-Reply-To: References: Message-ID: Le mer. 30 nov. 2022 à 04:42, Maxim Dounin a écrit : > > Hello! > > On Mon, Nov 28, 2022 at 06:40:33AM +0100, Jérôme Loyet wrote: > > > Hello, > > > > I'm using rate limiting within the stream module. While it works great > > for long connections it does not work on request smaller than the rate > > limite size for 1 second. I set up a 1gbps rate limit (limit_rate > > 125m) and request smaller than 125M or not limited. This is a normal > > behavioir as the rate limiting is done with a second precision. This > > patch change second precision to millisecond precision. From my first > > tests (still on going) it seems to works better. > > > > What guys do you think about this patch ? > > A while ago a similar patch was considered for both stream and > http, though there is an open question on how to handle > long-running requests on 32-bit platforms (where ngx_msec_t will > overflow after ~48 days). With the naive approach, which is also > seen in your patch, all traffic on a such connection is likely to > completely stop after the overflow, which does not seem to be a > good outcome. Thanks you maxime for your feedbacks. here is another proposal. To address this issue, let's use the the millisecond for the first 10 seconds and switch back to seconds for the rest of the transfer. would it be acceptable ? Regards ++ Jerome # HG changeset patch # User Jerome Loyet # Date 1676019332 -3600 # Fri Feb 10 09:55:32 2023 +0100 # Node ID f381512e0f116b0d63751a48996cacd3963954ca # Parent cffaf3f2eec8fd33605c2a37814f5ffc30371989 switch to ms resolution for rate limiting rate limiting use a second resolution which can be bypassed on requests/response smaller than the configured rate for 1 second. This patch switches to millisecond resolution to ensure better precision of the rate limiting. Millisecond resolution is only used the first 10 seconds and switches back to seconds resolution after to prevent overflowing the variable. diff -r cffaf3f2eec8 -r f381512e0f11 src/event/ngx_event_pipe.c --- a/src/event/ngx_event_pipe.c Thu Feb 02 23:38:48 2023 +0300 +++ b/src/event/ngx_event_pipe.c Fri Feb 10 09:55:32 2023 +0100 @@ -104,6 +104,7 @@ off_t limit; ssize_t n, size; ngx_int_t rc; + ngx_uint_t time_diff; ngx_buf_t *b; ngx_msec_t delay; ngx_chain_t *chain, *cl, *ln; @@ -203,8 +204,14 @@ break; } - limit = (off_t) p->limit_rate * (ngx_time() - p->start_sec + 1) - - p->read_length; + // use millisecond resolution for the first 10 seconds + // then switch to second resolution to prevent variable overflow + if (ngx_time() - u-> start_sec < 10) { + time_diff = (ngx_current_msec - u->start_msec + 1) / 1000; + } else { + time_diff = ngx_time() - u->start_sec + 1; + } + limit = (off_t) p->limit_rate * time_diff - p->read_length; if (limit <= 0) { p->upstream->read->delayed = 1; diff -r cffaf3f2eec8 -r f381512e0f11 src/event/ngx_event_pipe.h --- a/src/event/ngx_event_pipe.h Thu Feb 02 23:38:48 2023 +0300 +++ b/src/event/ngx_event_pipe.h Fri Feb 10 09:55:32 2023 +0100 @@ -92,6 +92,7 @@ size_t limit_rate; time_t start_sec; + ngx_msec_t start_msec; ngx_temp_file_t *temp_file; diff -r cffaf3f2eec8 -r f381512e0f11 src/http/ngx_http_upstream.c --- a/src/http/ngx_http_upstream.c Thu Feb 02 23:38:48 2023 +0300 +++ b/src/http/ngx_http_upstream.c Fri Feb 10 09:55:32 2023 +0100 @@ -3218,6 +3218,7 @@ p->log = c->log; p->limit_rate = u->conf->limit_rate; p->start_sec = ngx_time(); + p->start_msec = ngx_current_msec; p->cacheable = u->cacheable || u->store; diff -r cffaf3f2eec8 -r f381512e0f11 src/stream/ngx_stream_proxy_module.c --- a/src/stream/ngx_stream_proxy_module.c Thu Feb 02 23:38:48 2023 +0300 +++ b/src/stream/ngx_stream_proxy_module.c Fri Feb 10 09:55:32 2023 +0100 @@ -436,6 +436,7 @@ u->peer.type = c->type; u->start_sec = ngx_time(); + u->start_msec = ngx_current_msec; c->write->handler = ngx_stream_proxy_downstream_handler; c->read->handler = ngx_stream_proxy_downstream_handler; @@ -1594,7 +1595,7 @@ ssize_t n; ngx_buf_t *b; ngx_int_t rc; - ngx_uint_t flags, *packets; + ngx_uint_t flags, *packets, time_diff; ngx_msec_t delay; ngx_chain_t *cl, **ll, **out, **busy; ngx_connection_t *c, *pc, *src, *dst; @@ -1678,8 +1679,14 @@ if (size && src->read->ready && !src->read->delayed) { if (limit_rate) { - limit = (off_t) limit_rate * (ngx_time() - u->start_sec + 1) - - *received; + // use millisecond resolution for the first 10 seconds + // then switch to second resolution to prevent variable overflow + if (ngx_time() - u-> start_sec < 10) { + time_diff = (ngx_current_msec - u->start_msec + 1) / 1000; + } else { + time_diff = ngx_time() - u->start_sec + 1; + } + limit = (off_t) limit_rate * time_diff - *received; if (limit <= 0) { src->read->delayed = 1; diff -r cffaf3f2eec8 -r f381512e0f11 src/stream/ngx_stream_upstream.h --- a/src/stream/ngx_stream_upstream.h Thu Feb 02 23:38:48 2023 +0300 +++ b/src/stream/ngx_stream_upstream.h Fri Feb 10 09:55:32 2023 +0100 @@ -128,6 +128,7 @@ off_t received; time_t start_sec; + ngx_msec_t start_msec; ngx_uint_t requests; ngx_uint_t responses; ngx_msec_t start_time; From pluknet at nginx.com Fri Feb 10 11:21:05 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Fri, 10 Feb 2023 15:21:05 +0400 Subject: [PATCH 2 of 4] Win32: handling of localized MSVC cl output In-Reply-To: <43098cb134a87a404b70.1671543016@1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa> References: <43098cb134a87a404b70.1671543016@1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa> Message-ID: <014BEFAD-B793-4E50-A432-71406DE50D11@nginx.com> > On 20 Dec 2022, at 17:30, Maxim Dounin wrote: > > # HG changeset patch > # User Maxim Dounin > # Date 1671541078 -10800 > # Tue Dec 20 15:57:58 2022 +0300 > # Node ID 43098cb134a87a404b70fcc77ad01ca343cba969 > # Parent f5d9c24fb4ac2a6b82b9d842b88978a329690138 > Win32: handling of localized MSVC cl output. > > Output examples in English, Russian, and Spanish: > > Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 16.00.30319.01 for 80x86 > Оптимизирующий 32-разрядный компилятор Microsoft (R) C/C++ версии 16.00.30319.01 для 80x86 > Compilador de optimización de C/C++ de Microsoft (R) versión 16.00.30319.01 para x64 > The transaction to import this change with mercurial on a non-localized win2003 aborts with the error: abort: decoding near '....': 'charmap' codec can't decode byte 0x8f in position 218: character maps to ! The position matches UTF-8 U+044F (0xd18f), Cyrillic small letter ya: $ hg ex --template {desc} | hexdump -C | grep `printf "%08x\n" $((218/16*16))` 000000d0 2d d1 80 d0 b0 d0 b7 d1 80 d1 8f d0 b4 d0 bd d1 |-?.аз?.?.дн?| Although the error can be suppressed using HGENCODING, "hg log" produces garbled output in place of Cyrillic and umlaut symbols. The safest solution can be to mangle such localized examples in ascii, to allow the change to apply and still have a sketchy knowledge how cl output can be different. Another way is to skip the examples. > Since most of the words are translated, instead of looking for the words > "Compiler Version" we now search for "C/C++" and the version number. > > diff -r f5d9c24fb4ac -r 43098cb134a8 auto/cc/msvc > --- a/auto/cc/msvc Tue Dec 20 15:57:51 2022 +0300 > +++ b/auto/cc/msvc Tue Dec 20 15:57:58 2022 +0300 > @@ -11,8 +11,8 @@ > # MSVC 2015 (14.0) cl 19.00 > > > -NGX_MSVC_VER=`$NGX_WINE $CC 2>&1 | grep 'Compiler Version' 2>&1 \ > - | sed -e 's/^.* Version \(.*\)/\1/'` > +NGX_MSVC_VER=`$NGX_WINE $CC 2>&1 | grep 'C/C++.* [0-9][0-9]*\.[0-9]' 2>&1 \ > + | sed -e 's/^.* \([0-9][0-9]*\.[0-9].*\)/\1/'` > > echo " + cl version: $NGX_MSVC_VER" > I recall there were discussions whether we can avoid using the grep command, or if it should search other words for better matching. Personally, I think the proposed change is good enough. -- Sergey Kandaurov From pluknet at nginx.com Fri Feb 10 12:13:18 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Fri, 10 Feb 2023 16:13:18 +0400 Subject: [PATCH 3 of 4] Win32: i386 now assumed when crossbuilding (ticket #2416) In-Reply-To: <6606ed21a7091b060ebe.1671543017@1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa> References: <6606ed21a7091b060ebe.1671543017@1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa> Message-ID: <5CA9260F-88B6-40DC-AB74-21162C80F113@nginx.com> > On 20 Dec 2022, at 17:30, Maxim Dounin wrote: > > # HG changeset patch > # User Maxim Dounin > # Date 1671542876 -10800 > # Tue Dec 20 16:27:56 2022 +0300 > # Node ID 6606ed21a7091b060ebec0d082876ddbbbe0ea79 > # Parent 43098cb134a87a404b70fcc77ad01ca343cba969 > Win32: i386 now assumed when crossbuilding (ticket #2416). > > Previously, NGX_MACHINE was not set when crossbuilding, resulting in > NGX_ALIGNMENT=16 being used in 32-bit builds (if not explicitly set to a > correct value). This in turn might result in memory corruption in > ngx_palloc() (as there are no usable aligned allocator on Windows, and > normal malloc() is used instead, which provides 8 byte alignment on > 32-bit platforms). > > To fix this, now i386 machine is set when crossbuilding, so nginx won't > assume strict alignment requirements. > > diff -r 43098cb134a8 -r 6606ed21a709 auto/configure > --- a/auto/configure Tue Dec 20 15:57:58 2022 +0300 > +++ b/auto/configure Tue Dec 20 16:27:56 2022 +0300 > @@ -44,6 +44,7 @@ if test -z "$NGX_PLATFORM"; then > else > echo "building for $NGX_PLATFORM" > NGX_SYSTEM=$NGX_PLATFORM > + NGX_MACHINE=i386 > fi > > . auto/cc/conf Don't you want to put the assignment under this test? [ "$NGX_PLATFORM" = win32 ] There are numerous examples in the wild of using the --crossbuild option with other values, despite statements that crossbuilds aren't really supported. If we'd like to insist crossbuilding is win32 only, then it makes sense to have an explicit restriction, e.g. by rejecting --crossbuild values other than win32. -- Sergey Kandaurov From pluknet at nginx.com Fri Feb 10 14:11:48 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Fri, 10 Feb 2023 18:11:48 +0400 Subject: [PATCH 4 of 4] Win32: OpenSSL compilation for x64 targets with MSVC In-Reply-To: References: Message-ID: <2AD9CC4B-5466-4DC4-AEAF-136F0512085E@nginx.com> > On 20 Dec 2022, at 17:30, Maxim Dounin wrote: > > # HG changeset patch > # User Maxim Dounin > # Date 1671542914 -10800 > # Tue Dec 20 16:28:34 2022 +0300 > # Node ID e5a75718823d5ec365703275f3efa87d0b63f8c4 > # Parent 6606ed21a7091b060ebec0d082876ddbbbe0ea79 > Win32: OpenSSL compilation for x64 targets with MSVC. > > To ensure proper target selection the NGX_MACHINE variable is now set > based on the MSVC compiler output, and the OpenSSL target is set based > on it. > > This is not important as long as "no-asm" is used (as in misc/GNUmakefile > and win32 build instructions), but might be beneficial if someone is trying > to build OpenSSL with assembler code. I was not able to test x64 case, but looking at the change it's all good. > > diff -r 6606ed21a709 -r e5a75718823d auto/cc/msvc > --- a/auto/cc/msvc Tue Dec 20 16:27:56 2022 +0300 > +++ b/auto/cc/msvc Tue Dec 20 16:28:34 2022 +0300 > @@ -22,6 +22,21 @@ have=NGX_COMPILER value="\"cl $NGX_MSVC_ > ngx_msvc_ver=`echo $NGX_MSVC_VER | sed -e 's/^\([0-9]*\).*/\1/'` > > > +# detect x64 builds > + > +case "$NGX_MSVC_VER" in > + > + *x64) > + NGX_MACHINE=amd64 > + ;; > + > + *) > + NGX_MACHINE=i386 > + ;; > + > +esac > + > + > # optimizations > > # maximize speed, equivalent to -Og -Oi -Ot -Oy -Ob2 -Gs -GF -Gy > diff -r 6606ed21a709 -r e5a75718823d auto/lib/openssl/make > --- a/auto/lib/openssl/make Tue Dec 20 16:27:56 2022 +0300 > +++ b/auto/lib/openssl/make Tue Dec 20 16:28:34 2022 +0300 > @@ -7,11 +7,24 @@ case "$CC" in > > cl) > > + case "$NGX_MACHINE" in > + > + amd64) > + OPENSSL_TARGET=VC-WIN64A > + ;; > + > + *) > + OPENSSL_TARGET=VC-WIN32 > + ;; > + > + esac > + > cat << END >> $NGX_MAKEFILE > > $OPENSSL/openssl/include/openssl/ssl.h: $NGX_MAKEFILE > \$(MAKE) -f auto/lib/openssl/makefile.msvc \ > - OPENSSL="$OPENSSL" OPENSSL_OPT="$OPENSSL_OPT" > + OPENSSL="$OPENSSL" OPENSSL_OPT="$OPENSSL_OPT" \ > + OPENSSL_TARGET="$OPENSSL_TARGET" > > END > > diff -r 6606ed21a709 -r e5a75718823d auto/lib/openssl/makefile.msvc > --- a/auto/lib/openssl/makefile.msvc Tue Dec 20 16:27:56 2022 +0300 > +++ b/auto/lib/openssl/makefile.msvc Tue Dec 20 16:28:34 2022 +0300 > @@ -6,7 +6,7 @@ > all: > cd $(OPENSSL) > > - perl Configure VC-WIN32 no-shared no-threads \ > + perl Configure $(OPENSSL_TARGET) no-shared no-threads \ > --prefix="%cd%/openssl" \ > --openssldir="%cd%/openssl/ssl" \ > $(OPENSSL_OPT) -- Sergey Kandaurov From mdounin at mdounin.ru Fri Feb 10 19:15:08 2023 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 10 Feb 2023 22:15:08 +0300 Subject: [PATCH 2 of 4] Win32: handling of localized MSVC cl output In-Reply-To: <014BEFAD-B793-4E50-A432-71406DE50D11@nginx.com> References: <43098cb134a87a404b70.1671543016@1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa> <014BEFAD-B793-4E50-A432-71406DE50D11@nginx.com> Message-ID: Hello! On Fri, Feb 10, 2023 at 03:21:05PM +0400, Sergey Kandaurov wrote: > > > On 20 Dec 2022, at 17:30, Maxim Dounin wrote: > > > > # HG changeset patch > > # User Maxim Dounin > > # Date 1671541078 -10800 > > # Tue Dec 20 15:57:58 2022 +0300 > > # Node ID 43098cb134a87a404b70fcc77ad01ca343cba969 > > # Parent f5d9c24fb4ac2a6b82b9d842b88978a329690138 > > Win32: handling of localized MSVC cl output. > > > > Output examples in English, Russian, and Spanish: > > > > Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 16.00.30319.01 for 80x86 > > Оптимизирующий 32-разрядный компилятор Microsoft (R) C/C++ версии 16.00.30319.01 для 80x86 > > Compilador de optimización de C/C++ de Microsoft (R) versión 16.00.30319.01 para x64 > > > > The transaction to import this change with mercurial on a non-localized > win2003 aborts with the error: > > abort: decoding near '....': 'charmap' codec can't decode byte 0x8f in position 218: character maps to ! > > The position matches UTF-8 U+044F (0xd18f), Cyrillic small letter ya: > $ hg ex --template {desc} | hexdump -C | grep `printf "%08x\n" $((218/16*16))` > 000000d0 2d d1 80 d0 b0 d0 b7 d1 80 d1 8f d0 b4 d0 bd d1 |-?.аз?.?.дн?| > > Although the error can be suppressed using HGENCODING, "hg log" > produces garbled output in place of Cyrillic and umlaut symbols. > > The safest solution can be to mangle such localized examples in ascii, > to allow the change to apply and still have a sketchy knowledge how cl > output can be different. Another way is to skip the examples. I don't think that mangling examples is a good idea - these are provided specifically to make it possible to review and test the code. On the other hand, it's quite normal that importing a patch with UTF-8 characters on a system not configured to use UTF-8 fails. Just pulling from another repo will work fine though. That is, this only affects developers specifically involved in the patch development. Further, it is quite normal that displaying such a patch produces garbled output on a system without UTF-8 console support. On systems with Cyrillic and/or UTF-8 console support it will be displayed correctly (provided that HGENCODING is set or properly autodetected). Overall, I tend to think that using UTF-8 in commit logs, at least in various quotes, shouldn't be a problem. Note well that we already have UTF-8 in various commit logs and author names. > > > Since most of the words are translated, instead of looking for the words > > "Compiler Version" we now search for "C/C++" and the version number. > > > > diff -r f5d9c24fb4ac -r 43098cb134a8 auto/cc/msvc > > --- a/auto/cc/msvc Tue Dec 20 15:57:51 2022 +0300 > > +++ b/auto/cc/msvc Tue Dec 20 15:57:58 2022 +0300 > > @@ -11,8 +11,8 @@ > > # MSVC 2015 (14.0) cl 19.00 > > > > > > -NGX_MSVC_VER=`$NGX_WINE $CC 2>&1 | grep 'Compiler Version' 2>&1 \ > > - | sed -e 's/^.* Version \(.*\)/\1/'` > > +NGX_MSVC_VER=`$NGX_WINE $CC 2>&1 | grep 'C/C++.* [0-9][0-9]*\.[0-9]' 2>&1 \ > > + | sed -e 's/^.* \([0-9][0-9]*\.[0-9].*\)/\1/'` > > > > echo " + cl version: $NGX_MSVC_VER" > > > > I recall there were discussions whether we can avoid using the grep command, > or if it should search other words for better matching. > Personally, I think the proposed change is good enough. The change which was previously discussed was slightly different (and grep wasn't adding anything there). In this change, usage of grep is perfectly in line with other uses (and selects just one line, which is then modified by sed). -- Maxim Dounin http://mdounin.ru/ From mdounin at mdounin.ru Fri Feb 10 21:15:39 2023 From: mdounin at mdounin.ru (Maxim Dounin) Date: Sat, 11 Feb 2023 00:15:39 +0300 Subject: [PATCH 3 of 4] Win32: i386 now assumed when crossbuilding (ticket #2416) In-Reply-To: <5CA9260F-88B6-40DC-AB74-21162C80F113@nginx.com> References: <6606ed21a7091b060ebe.1671543017@1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa> <5CA9260F-88B6-40DC-AB74-21162C80F113@nginx.com> Message-ID: Hello! On Fri, Feb 10, 2023 at 04:13:18PM +0400, Sergey Kandaurov wrote: > > > On 20 Dec 2022, at 17:30, Maxim Dounin wrote: > > > > # HG changeset patch > > # User Maxim Dounin > > # Date 1671542876 -10800 > > # Tue Dec 20 16:27:56 2022 +0300 > > # Node ID 6606ed21a7091b060ebec0d082876ddbbbe0ea79 > > # Parent 43098cb134a87a404b70fcc77ad01ca343cba969 > > Win32: i386 now assumed when crossbuilding (ticket #2416). > > > > Previously, NGX_MACHINE was not set when crossbuilding, resulting in > > NGX_ALIGNMENT=16 being used in 32-bit builds (if not explicitly set to a > > correct value). This in turn might result in memory corruption in > > ngx_palloc() (as there are no usable aligned allocator on Windows, and > > normal malloc() is used instead, which provides 8 byte alignment on > > 32-bit platforms). > > > > To fix this, now i386 machine is set when crossbuilding, so nginx won't > > assume strict alignment requirements. > > > > diff -r 43098cb134a8 -r 6606ed21a709 auto/configure > > --- a/auto/configure Tue Dec 20 15:57:58 2022 +0300 > > +++ b/auto/configure Tue Dec 20 16:27:56 2022 +0300 > > @@ -44,6 +44,7 @@ if test -z "$NGX_PLATFORM"; then > > else > > echo "building for $NGX_PLATFORM" > > NGX_SYSTEM=$NGX_PLATFORM > > + NGX_MACHINE=i386 > > fi > > > > . auto/cc/conf > > Don't you want to put the assignment under this test? > [ "$NGX_PLATFORM" = win32 ] > > There are numerous examples in the wild of using > the --crossbuild option with other values, despite > statements that crossbuilds aren't really supported. > > If we'd like to insist crossbuilding is win32 only, > then it makes sense to have an explicit restriction, > e.g. by rejecting --crossbuild values other than win32. Using --crossbuild for anything isn't really suppported at all, even for win32, see comments to the ticket mentioned. It mostly works for win32 though, but does not work at all for other platforms. Examples you've seen are likely about using 3rd party crossbuild patches, and it's up to the patches to set appropriate NGX_MACHINE. -- Maxim Dounin http://mdounin.ru/ From P.Pautov at F5.com Sat Feb 11 05:30:39 2023 From: P.Pautov at F5.com (Pavel Pautov) Date: Sat, 11 Feb 2023 05:30:39 +0000 Subject: [PATCH 2 of 4] Win32: handling of localized MSVC cl output In-Reply-To: References: <43098cb134a87a404b70.1671543016@1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa> Message-ID: > -----Original Message----- > From: nginx-devel On Behalf Of Maxim > Dounin > Sent: Thursday, February 9, 2023 20:33 > > Hello! > > On Thu, Feb 09, 2023 at 06:55:22AM +0000, Pavel Pautov via nginx-devel wrote: > > > The 'for 80x86' part of cl version (in auto/configure output) > > always felt odd. Given it's used in subsequent patch, can we at > > least strip localized 'for' from NGX_MSVC_VER? Having it in > > otherwise non-localized output of "nginx -V" doesn't seem right > > either. > > The "nginx -V" output simply shows how the compiler identifies > itself. The patch has no intention to change this, it just > ensures that output of localized MSVC cl versions is recognized by > nginx. And, actually, I would rather object stripping the > details, since any aspects of compiler identification might be > important. The patch effectively introduces localized strings into "nginx -V" output and, I guess, I just don't see how that is useful. The binary architecture can be deduced from binary itself and random peace of localized string might not be enough for build machine locale identification (and why bother with it, anyway). And here are some other localizations to consider: Microsoft (R) C/C++ 최적화 컴파일러 버전 19.29.30147(x64) 用于 x64 的 Microsoft (R) C/C++ 优化编译器 19.29.30147 版 So, I'd suggest to introduce something like NGX_MSVC_VER_LINE and then derive both NGX_MSVC_VER (which would contain only dotted version number) and NGX_MACHINE from it. From P.Pautov at F5.com Sat Feb 11 07:27:36 2023 From: P.Pautov at F5.com (Pavel Pautov) Date: Sat, 11 Feb 2023 07:27:36 +0000 Subject: [PATCH 4 of 4] Win32: OpenSSL compilation for x64 targets with MSVC In-Reply-To: <2AD9CC4B-5466-4DC4-AEAF-136F0512085E@nginx.com> References: <2AD9CC4B-5466-4DC4-AEAF-136F0512085E@nginx.com> Message-ID: > -----Original Message----- > From: nginx-devel On Behalf Of Sergey > Kandaurov > Sent: Friday, February 10, 2023 06:12 > > > On 20 Dec 2022, at 17:30, Maxim Dounin wrote: > > > > # HG changeset patch > > # User Maxim Dounin > > # Date 1671542914 -10800 > > # Tue Dec 20 16:28:34 2022 +0300 > > # Node ID e5a75718823d5ec365703275f3efa87d0b63f8c4 > > # Parent 6606ed21a7091b060ebec0d082876ddbbbe0ea79 > > Win32: OpenSSL compilation for x64 targets with MSVC. > > > > To ensure proper target selection the NGX_MACHINE variable is now set > > based on the MSVC compiler output, and the OpenSSL target is set based > > on it. > > > > This is not important as long as "no-asm" is used (as in misc/GNUmakefile > > and win32 build instructions), but might be beneficial if someone is trying > > to build OpenSSL with assembler code. > > I was not able to test x64 case, but looking at the change it's all good. x64 build did work for me. From mdounin at mdounin.ru Sat Feb 11 20:28:56 2023 From: mdounin at mdounin.ru (Maxim Dounin) Date: Sat, 11 Feb 2023 23:28:56 +0300 Subject: [PATCH 2 of 4] Win32: handling of localized MSVC cl output In-Reply-To: References: <43098cb134a87a404b70.1671543016@1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa> Message-ID: Hello! On Sat, Feb 11, 2023 at 05:30:39AM +0000, Pavel Pautov via nginx-devel wrote: > > -----Original Message----- > > From: nginx-devel On Behalf Of Maxim > > Dounin > > Sent: Thursday, February 9, 2023 20:33 > > > > Hello! > > > > On Thu, Feb 09, 2023 at 06:55:22AM +0000, Pavel Pautov via nginx-devel wrote: > > > > > The 'for 80x86' part of cl version (in auto/configure output) > > > always felt odd. Given it's used in subsequent patch, can we at > > > least strip localized 'for' from NGX_MSVC_VER? Having it in > > > otherwise non-localized output of "nginx -V" doesn't seem right > > > either. > > > > The "nginx -V" output simply shows how the compiler identifies > > itself. The patch has no intention to change this, it just > > ensures that output of localized MSVC cl versions is recognized by > > nginx. And, actually, I would rather object stripping the > > details, since any aspects of compiler identification might be > > important. > > The patch effectively introduces localized strings into "nginx > -V" output and, I guess, I just don't see how that is useful. > The binary architecture can be deduced from binary itself and > random peace of localized string might not be enough for build > machine locale identification (and why bother with it, anyway). The details are usually needed in case of compilation issues (so there will be no binary at all) or miscompilation (so you might need all the details to find out why it happened). Further, deducing architecture from binary itself is not something readily available in most support cases. Note well that we do the same for other compilers: any additional details following the version number are preserved by nginx and shown in the "nginx -V" output. This is proven to be helpful in many practical cases. > And here are some other localizations to consider: > Microsoft (R) C/C++ 최적화 컴파일러 버전 19.29.30147(x64) > 用于 x64 的 Microsoft (R) C/C++ 优化编译器 19.29.30147 版 Thanks for additional samples. That's unfortunate that Microsoft uses so many variants of the compiler identification, especially given that nginx explicitly requests non-localized output as per POSIX, with LC_ALL=C. > So, I'd suggest to introduce something like NGX_MSVC_VER_LINE > and then derive both NGX_MSVC_VER (which would contain only > dotted version number) and NGX_MACHINE from it. See above for arguments against only the version number. As for the additional ways to identify NGX_MACHINE based on the full cl output, I don't think it worth the effort. It is not really needed except for x64 builds with OpenSSL assembler optimizations, which is not something nginx uses for standard builds. Further, the simple approach as suggested in this patch series covers most of the practical cases. If a more complete solution would be needed, a better approach might be to provide a way to manually specify NGX_MACHINE, or instead detect it by compile-only feature tests from predefined macros. This is, however, outside of the scope of this patch series. -- Maxim Dounin http://mdounin.ru/ From michael.kourlas at solace.com Wed Feb 15 16:50:13 2023 From: michael.kourlas at solace.com (Michael Kourlas) Date: Wed, 15 Feb 2023 11:50:13 -0500 Subject: [PATCH] HTTP: Add new uri_normalization_percent_decode option Message-ID: <129437ade41b14a584fb.1676479813@dev2-97.sol-local> # HG changeset patch # User Michael Kourlas # Date 1676408746 18000 # Tue Feb 14 16:05:46 2023 -0500 # Node ID 129437ade41b14a584fb4b7558accc1b8dee7f45 # Parent cffaf3f2eec8fd33605c2a37814f5ffc30371989 HTTP: Add new uri_normalization_percent_decode option This patch addresses ticket #2225 by adding a new uri_normalization_percent_decode configuration option that controls which characters are percent-decoded by nginx as part of its URI normalization. The option has two values: "all" and "all-except-reserved". "all" is the default value and is the current behaviour. When the option is set to "all-except-reserved", nginx percent-decodes all characters except those in the reserved set defined by RFC 3986: reserved = gen-delims / sub-delims gen-delims = ":" / "/" / "?" / "#" / "[" / "]" / "@" sub-delims = "!" / "$" / "&" / "'" / "(" / ")" / "*" / "+" / "," / ";" / "=" In addition, when "all-except-reserved" is used, nginx will not re-encode "%" from the request URI when it observes that it is part of a percent-encoded reserved character. When nginx percent-decodes reserved characters, this can often change the request URI's semantics, making it impossible to use a normalized URI for certain use cases. "uri_normalization_percent_decode" gives the configuration author the freedom to determine which reserved characters are semantically relevant and which are not. For example, consider the following location block, which handles part of a hypothetical API: location ~ ^/api/objects/[^/]+/subobjects(/.*)?$ { ... } Because nginx always normalizes "%2F" to "/", this location block will not match a path of /api/objects/sample%2Fname/subobjects, even if the API permits "/" to appear percent-encoded in the URI as part of object names. nginx will instead interpret this as /api/objects/sample/name/subobjects, a completely different path. Setting "uri_normalization_percent_decode" to "all-except-reserved" will leave "%2F" encoded, resulting in the expected behaviour. diff -r cffaf3f2eec8 -r 129437ade41b src/core/ngx_string.c --- a/src/core/ngx_string.c Thu Feb 02 23:38:48 2023 +0300 +++ b/src/core/ngx_string.c Tue Feb 14 16:05:46 2023 -0500 @@ -1487,7 +1487,8 @@ uintptr_t -ngx_escape_uri(u_char *dst, u_char *src, size_t size, ngx_uint_t type) +ngx_escape_uri(u_char *dst, u_char *src, size_t size, ngx_uint_t type, + ngx_uint_t skip_preencoded_type) { ngx_uint_t n; uint32_t *escape; @@ -1641,7 +1642,11 @@ n = 0; while (size) { - if (escape[*src >> 5] & (1U << (*src & 0x1f))) { + if ((escape[*src >> 5] & (1U << (*src & 0x1f))) + && !(*src == '%' && size >= 3 + && ngx_escape_uri_skip_preencoded_character( + src + 1, skip_preencoded_type))) + { n++; } src++; @@ -1652,7 +1657,11 @@ } while (size) { - if (escape[*src >> 5] & (1U << (*src & 0x1f))) { + if ((escape[*src >> 5] & (1U << (*src & 0x1f))) + && !(*src == '%' && size >= 3 + && ngx_escape_uri_skip_preencoded_character( + src + 1, skip_preencoded_type))) + { *dst++ = '%'; *dst++ = hex[*src >> 4]; *dst++ = hex[*src & 0xf]; @@ -1668,6 +1677,87 @@ } +ngx_uint_t +ngx_escape_uri_skip_preencoded_character(u_char *hex_component, + ngx_uint_t skip_preencoded_type) +{ + u_char ch, decoded_ch; + uint32_t *skip; + + static uint32_t none[] = { + 0x00000000, /* 0000 0000 0000 0000 0000 0000 0000 0000 */ + + /* ?>=< ;:98 7654 3210 /.-, +*)( '&%$ #"! */ + 0x00000000, /* 0000 0000 0000 0000 0000 0000 0000 0000 */ + + /* _^]\ [ZYX WVUT SRQP ONML KJIH GFED CBA@ */ + 0x00000000, /* 0000 0000 0000 0000 0000 0000 0000 0000 */ + + /* ~}| {zyx wvut srqp onml kjih gfed cba` */ + 0x00000000, /* 0000 0000 0000 0000 0000 0000 0000 0000 */ + + 0x00000000, /* 0000 0000 0000 0000 0000 0000 0000 0000 */ + 0x00000000, /* 0000 0000 0000 0000 0000 0000 0000 0000 */ + 0x00000000, /* 0000 0000 0000 0000 0000 0000 0000 0000 */ + 0x00000000 /* 0000 0000 0000 0000 0000 0000 0000 0000 */ + }; + + static uint32_t reserved_only[] = { + 0x00000000, /* 0000 0000 0000 0000 0000 0000 0000 0000 */ + + /* ?>=< ;:98 7654 3210 /.-, +*)( '&%$ #"! */ + 0xac009fda, /* 1010 1100 0000 0000 1001 1111 1101 1010 */ + + /* _^]\ [ZYX WVUT SRQP ONML KJIH GFED CBA@ */ + 0x28000001, /* 0010 1000 0000 0000 0000 0000 0000 0001 */ + + /* ~}| {zyx wvut srqp onml kjih gfed cba` */ + 0x00000000, /* 0000 0000 0000 0000 0000 0000 0000 0000 */ + + 0x00000000, /* 0000 0000 0000 0000 0000 0000 0000 0000 */ + 0x00000000, /* 0000 0000 0000 0000 0000 0000 0000 0000 */ + 0x00000000, /* 0000 0000 0000 0000 0000 0000 0000 0000 */ + 0x00000000 /* 0000 0000 0000 0000 0000 0000 0000 0000 */ + }; + + static uint32_t *skip_map[] = { none, reserved_only }; + + skip = skip_map[skip_preencoded_type]; + + ch = *hex_component; + if (ch >= '0' && ch <= '9') { + decoded_ch = (u_char) (ch - '0'); + } else { + ch = (u_char) (ch | 0x20); + if (ch >= 'a' && ch <= 'f') { + decoded_ch = (u_char) (ch - 'a' + 10); + } else { + /* not part of a percent-encoded character */ + return 0; + } + } + + ch = *(hex_component + 1); + if (ch >= '0' && ch <= '9') { + decoded_ch = (u_char) ((decoded_ch << 4) + (ch - '0')); + } else { + ch = (u_char) (ch | 0x20); + if (ch >= 'a' && ch <= 'f') { + decoded_ch = (u_char) ((decoded_ch << 4) + (ch - 'a') + 10); + } else { + /* not part of a percent-encoded character */ + return 0; + } + } + + if (skip[decoded_ch >> 5] & (1U << (decoded_ch & 0x1f))) { + return 1; + } else { + return 0; + } +} + + void ngx_unescape_uri(u_char **dst, u_char **src, size_t size, ngx_uint_t type) { diff -r cffaf3f2eec8 -r 129437ade41b src/core/ngx_string.h --- a/src/core/ngx_string.h Thu Feb 02 23:38:48 2023 +0300 +++ b/src/core/ngx_string.h Tue Feb 14 16:05:46 2023 -0500 @@ -204,11 +204,20 @@ #define NGX_ESCAPE_MEMCACHED 5 #define NGX_ESCAPE_MAIL_AUTH 6 +/* + * these enumeration values must correspond to the enumeration values for + * NGX_HTTP_URI_NORMALIZATION_PERCENT_DECODE + */ +#define NGX_ESCAPE_SKIP_PREENCODED_NONE 0 +#define NGX_ESCAPE_SKIP_PREENCODED_RESERVED 1 + #define NGX_UNESCAPE_URI 1 #define NGX_UNESCAPE_REDIRECT 2 uintptr_t ngx_escape_uri(u_char *dst, u_char *src, size_t size, - ngx_uint_t type); + ngx_uint_t type, ngx_uint_t skip_preencoded_type); +ngx_uint_t ngx_escape_uri_skip_preencoded_character(u_char *seq, + ngx_uint_t skip_preencoded_type); void ngx_unescape_uri(u_char **dst, u_char **src, size_t size, ngx_uint_t type); uintptr_t ngx_escape_html(u_char *dst, u_char *src, size_t size); uintptr_t ngx_escape_json(u_char *dst, u_char *src, size_t size); diff -r cffaf3f2eec8 -r 129437ade41b src/event/ngx_event_openssl.c --- a/src/event/ngx_event_openssl.c Thu Feb 02 23:38:48 2023 +0300 +++ b/src/event/ngx_event_openssl.c Tue Feb 14 16:05:46 2023 -0500 @@ -5366,7 +5366,8 @@ return NGX_OK; } - n = ngx_escape_uri(NULL, cert.data, cert.len, NGX_ESCAPE_URI_COMPONENT); + n = ngx_escape_uri(NULL, cert.data, cert.len, NGX_ESCAPE_URI_COMPONENT, + NGX_ESCAPE_SKIP_PREENCODED_NONE); s->len = cert.len + n * 2; s->data = ngx_pnalloc(pool, s->len); @@ -5374,7 +5375,8 @@ return NGX_ERROR; } - ngx_escape_uri(s->data, cert.data, cert.len, NGX_ESCAPE_URI_COMPONENT); + ngx_escape_uri(s->data, cert.data, cert.len, NGX_ESCAPE_URI_COMPONENT, + NGX_ESCAPE_SKIP_PREENCODED_NONE); return NGX_OK; } diff -r cffaf3f2eec8 -r 129437ade41b src/event/ngx_event_openssl_stapling.c --- a/src/event/ngx_event_openssl_stapling.c Thu Feb 02 23:38:48 2023 +0300 +++ b/src/event/ngx_event_openssl_stapling.c Tue Feb 14 16:05:46 2023 -0500 @@ -1747,7 +1747,8 @@ ngx_encode_base64(&base64, &binary); escape = ngx_escape_uri(NULL, base64.data, base64.len, - NGX_ESCAPE_URI_COMPONENT); + NGX_ESCAPE_URI_COMPONENT, + NGX_ESCAPE_SKIP_PREENCODED_NONE); ngx_log_debug2(NGX_LOG_DEBUG_EVENT, ctx->log, 0, "ssl ocsp request length %z, escape %d", @@ -1777,7 +1778,8 @@ } else { p = (u_char *) ngx_escape_uri(p, base64.data, base64.len, - NGX_ESCAPE_URI_COMPONENT); + NGX_ESCAPE_URI_COMPONENT, + NGX_ESCAPE_SKIP_PREENCODED_NONE); } p = ngx_cpymem(p, " HTTP/1.0" CRLF, sizeof(" HTTP/1.0" CRLF) - 1); diff -r cffaf3f2eec8 -r 129437ade41b src/http/modules/ngx_http_autoindex_module.c --- a/src/http/modules/ngx_http_autoindex_module.c Thu Feb 02 23:38:48 2023 +0300 +++ b/src/http/modules/ngx_http_autoindex_module.c Tue Feb 14 16:05:46 2023 -0500 @@ -487,7 +487,8 @@ for (i = 0; i < entries->nelts; i++) { entry[i].escape = 2 * ngx_escape_uri(NULL, entry[i].name.data, entry[i].name.len, - NGX_ESCAPE_URI_COMPONENT); + NGX_ESCAPE_URI_COMPONENT, + NGX_ESCAPE_SKIP_PREENCODED_NONE); entry[i].escape_html = ngx_escape_html(NULL, entry[i].name.data, entry[i].name.len); @@ -549,7 +550,8 @@ if (entry[i].escape) { ngx_escape_uri(b->last, entry[i].name.data, entry[i].name.len, - NGX_ESCAPE_URI_COMPONENT); + NGX_ESCAPE_URI_COMPONENT, + NGX_ESCAPE_SKIP_PREENCODED_NONE); b->last += entry[i].name.len + entry[i].escape; diff -r cffaf3f2eec8 -r 129437ade41b src/http/modules/ngx_http_dav_module.c --- a/src/http/modules/ngx_http_dav_module.c Thu Feb 02 23:38:48 2023 +0300 +++ b/src/http/modules/ngx_http_dav_module.c Tue Feb 14 16:05:46 2023 -0500 @@ -1072,9 +1072,12 @@ static ngx_int_t ngx_http_dav_location(ngx_http_request_t *r) { - u_char *p; - size_t len; - uintptr_t escape; + u_char *p; + size_t len; + uintptr_t escape; + ngx_http_core_srv_conf_t *cscf; + + cscf = ngx_http_get_module_srv_conf(r, ngx_http_core_module); r->headers_out.location = ngx_list_push(&r->headers_out.headers); if (r->headers_out.location == NULL) { @@ -1085,7 +1088,8 @@ r->headers_out.location->next = NULL; ngx_str_set(&r->headers_out.location->key, "Location"); - escape = 2 * ngx_escape_uri(NULL, r->uri.data, r->uri.len, NGX_ESCAPE_URI); + escape = 2 * ngx_escape_uri(NULL, r->uri.data, r->uri.len, NGX_ESCAPE_URI, + cscf->uri_normalization_percent_decode); if (escape) { len = r->uri.len + escape; @@ -1099,7 +1103,8 @@ r->headers_out.location->value.len = len; r->headers_out.location->value.data = p; - ngx_escape_uri(p, r->uri.data, r->uri.len, NGX_ESCAPE_URI); + ngx_escape_uri(p, r->uri.data, r->uri.len, NGX_ESCAPE_URI, + cscf->uri_normalization_percent_decode); } else { r->headers_out.location->value = r->uri; diff -r cffaf3f2eec8 -r 129437ade41b src/http/modules/ngx_http_grpc_module.c --- a/src/http/modules/ngx_http_grpc_module.c Thu Feb 02 23:38:48 2023 +0300 +++ b/src/http/modules/ngx_http_grpc_module.c Tue Feb 14 16:05:46 2023 -0500 @@ -720,12 +720,15 @@ ngx_http_upstream_t *u; ngx_http_grpc_frame_t *f; ngx_http_script_code_pt code; + ngx_http_core_srv_conf_t *cscf; ngx_http_grpc_loc_conf_t *glcf; ngx_http_script_engine_t e, le; ngx_http_script_len_code_pt lcode; u = r->upstream; + cscf = ngx_http_get_module_srv_conf(r, ngx_http_core_module); + glcf = ngx_http_get_module_loc_conf(r, ngx_http_grpc_module); ctx = ngx_http_get_module_ctx(r, ngx_http_grpc_module); @@ -756,7 +759,8 @@ } else { escape = 2 * ngx_escape_uri(NULL, r->uri.data, r->uri.len, - NGX_ESCAPE_URI); + NGX_ESCAPE_URI, + cscf->uri_normalization_percent_decode); uri_len = r->uri.len + escape + sizeof("?") - 1 + r->args.len; } @@ -950,7 +954,7 @@ if (escape) { p = (u_char *) ngx_escape_uri(p, r->uri.data, r->uri.len, - NGX_ESCAPE_URI); + NGX_ESCAPE_URI, cscf->uri_normalization_percent_decode); } else { p = ngx_copy(p, r->uri.data, r->uri.len); diff -r cffaf3f2eec8 -r 129437ade41b src/http/modules/ngx_http_memcached_module.c --- a/src/http/modules/ngx_http_memcached_module.c Thu Feb 02 23:38:48 2023 +0300 +++ b/src/http/modules/ngx_http_memcached_module.c Tue Feb 14 16:05:46 2023 -0500 @@ -255,7 +255,8 @@ return NGX_ERROR; } - escape = 2 * ngx_escape_uri(NULL, vv->data, vv->len, NGX_ESCAPE_MEMCACHED); + escape = 2 * ngx_escape_uri(NULL, vv->data, vv->len, NGX_ESCAPE_MEMCACHED, + NGX_ESCAPE_SKIP_PREENCODED_NONE); len = sizeof("get ") - 1 + vv->len + escape + sizeof(CRLF) - 1; @@ -285,7 +286,7 @@ } else { b->last = (u_char *) ngx_escape_uri(b->last, vv->data, vv->len, - NGX_ESCAPE_MEMCACHED); + NGX_ESCAPE_MEMCACHED, NGX_ESCAPE_SKIP_PREENCODED_NONE); } ctx->key.len = b->last - ctx->key.data; diff -r cffaf3f2eec8 -r 129437ade41b src/http/modules/ngx_http_proxy_module.c --- a/src/http/modules/ngx_http_proxy_module.c Thu Feb 02 23:38:48 2023 +0300 +++ b/src/http/modules/ngx_http_proxy_module.c Tue Feb 14 16:05:46 2023 -0500 @@ -1143,10 +1143,13 @@ ngx_str_t *key; ngx_http_upstream_t *u; ngx_http_proxy_ctx_t *ctx; + ngx_http_core_srv_conf_t *cscf; ngx_http_proxy_loc_conf_t *plcf; u = r->upstream; + cscf = ngx_http_get_module_srv_conf(r, ngx_http_core_module); + plcf = ngx_http_get_module_loc_conf(r, ngx_http_proxy_module); ctx = ngx_http_get_module_ctx(r, ngx_http_proxy_module); @@ -1190,7 +1193,8 @@ if (r->quoted_uri || r->internal) { escape = 2 * ngx_escape_uri(NULL, r->uri.data + loc_len, - r->uri.len - loc_len, NGX_ESCAPE_URI); + r->uri.len - loc_len, NGX_ESCAPE_URI, + cscf->uri_normalization_percent_decode); } else { escape = 0; } @@ -1211,7 +1215,8 @@ if (escape) { ngx_escape_uri(p, r->uri.data + loc_len, - r->uri.len - loc_len, NGX_ESCAPE_URI); + r->uri.len - loc_len, NGX_ESCAPE_URI, + cscf->uri_normalization_percent_decode); p += r->uri.len - loc_len + escape; } else { @@ -1249,11 +1254,14 @@ ngx_http_script_code_pt code; ngx_http_proxy_headers_t *headers; ngx_http_script_engine_t e, le; + ngx_http_core_srv_conf_t *cscf; ngx_http_proxy_loc_conf_t *plcf; ngx_http_script_len_code_pt lcode; u = r->upstream; + cscf = ngx_http_get_module_srv_conf(r, ngx_http_core_module); + plcf = ngx_http_get_module_loc_conf(r, ngx_http_proxy_module); #if (NGX_HTTP_CACHE) @@ -1303,7 +1311,7 @@ if (r->quoted_uri || r->internal) { escape = 2 * ngx_escape_uri(NULL, r->uri.data + loc_len, - r->uri.len - loc_len, NGX_ESCAPE_URI); + r->uri.len - loc_len, NGX_ESCAPE_URI, cscf->uri_normalization_percent_decode); } uri_len = ctx->vars.uri.len + r->uri.len - loc_len + escape @@ -1428,7 +1436,8 @@ if (escape) { ngx_escape_uri(b->last, r->uri.data + loc_len, - r->uri.len - loc_len, NGX_ESCAPE_URI); + r->uri.len - loc_len, NGX_ESCAPE_URI, + cscf->uri_normalization_percent_decode); b->last += r->uri.len - loc_len + escape; } else { diff -r cffaf3f2eec8 -r 129437ade41b src/http/modules/ngx_http_ssi_filter_module.c --- a/src/http/modules/ngx_http_ssi_filter_module.c Thu Feb 02 23:38:48 2023 +0300 +++ b/src/http/modules/ngx_http_ssi_filter_module.c Tue Feb 14 16:05:46 2023 -0500 @@ -2348,7 +2348,8 @@ case NGX_HTTP_SSI_URL_ENCODING: len = 2 * ngx_escape_uri(NULL, value->data, value->len, - NGX_ESCAPE_HTML); + NGX_ESCAPE_HTML, + NGX_ESCAPE_SKIP_PREENCODED_NONE); if (len) { p = ngx_pnalloc(r->pool, value->len + len); @@ -2356,7 +2357,8 @@ return NGX_HTTP_SSI_ERROR; } - (void) ngx_escape_uri(p, value->data, value->len, NGX_ESCAPE_HTML); + (void) ngx_escape_uri(p, value->data, value->len, NGX_ESCAPE_HTML, + NGX_ESCAPE_SKIP_PREENCODED_NONE); } len += value->len; diff -r cffaf3f2eec8 -r 129437ade41b src/http/modules/ngx_http_static_module.c --- a/src/http/modules/ngx_http_static_module.c Thu Feb 02 23:38:48 2023 +0300 +++ b/src/http/modules/ngx_http_static_module.c Tue Feb 14 16:05:46 2023 -0500 @@ -58,6 +58,7 @@ ngx_buf_t *b; ngx_chain_t out; ngx_open_file_info_t of; + ngx_http_core_srv_conf_t *cscf; ngx_http_core_loc_conf_t *clcf; if (!(r->method & (NGX_HTTP_GET|NGX_HTTP_HEAD|NGX_HTTP_POST))) { @@ -85,6 +86,8 @@ ngx_log_debug1(NGX_LOG_DEBUG_HTTP, log, 0, "http filename: \"%s\"", path.data); + cscf = ngx_http_get_module_srv_conf(r, ngx_http_core_module); + clcf = ngx_http_get_module_loc_conf(r, ngx_http_core_module); ngx_memzero(&of, sizeof(ngx_open_file_info_t)); @@ -157,7 +160,8 @@ } escape = 2 * ngx_escape_uri(NULL, r->uri.data, r->uri.len, - NGX_ESCAPE_URI); + NGX_ESCAPE_URI, + cscf->uri_normalization_percent_decode); if (!clcf->alias && r->args.len == 0 && escape == 0) { len = r->uri.len + 1; @@ -180,7 +184,7 @@ if (escape) { last = (u_char *) ngx_escape_uri(location, r->uri.data, - r->uri.len, NGX_ESCAPE_URI); + r->uri.len, NGX_ESCAPE_URI, cscf->uri_normalization_percent_decode); } else { last = ngx_copy(location, r->uri.data, r->uri.len); diff -r cffaf3f2eec8 -r 129437ade41b src/http/ngx_http.c --- a/src/http/ngx_http.c Thu Feb 02 23:38:48 2023 +0300 +++ b/src/http/ngx_http.c Tue Feb 14 16:05:46 2023 -0500 @@ -900,7 +900,8 @@ uintptr_t escape; escape = 2 * ngx_escape_uri(NULL, clcf->name.data, clcf->name.len, - NGX_ESCAPE_URI); + NGX_ESCAPE_URI, + NGX_ESCAPE_SKIP_PREENCODED_NONE); if (escape) { len = clcf->name.len + escape; @@ -913,7 +914,8 @@ clcf->escaped_name.len = len; clcf->escaped_name.data = p; - ngx_escape_uri(p, clcf->name.data, clcf->name.len, NGX_ESCAPE_URI); + ngx_escape_uri(p, clcf->name.data, clcf->name.len, NGX_ESCAPE_URI, + NGX_ESCAPE_SKIP_PREENCODED_NONE); } else { clcf->escaped_name = clcf->name; diff -r cffaf3f2eec8 -r 129437ade41b src/http/ngx_http.h --- a/src/http/ngx_http.h Thu Feb 02 23:38:48 2023 +0300 +++ b/src/http/ngx_http.h Tue Feb 14 16:05:46 2023 -0500 @@ -96,7 +96,7 @@ ngx_int_t ngx_http_parse_request_line(ngx_http_request_t *r, ngx_buf_t *b); ngx_int_t ngx_http_parse_uri(ngx_http_request_t *r); ngx_int_t ngx_http_parse_complex_uri(ngx_http_request_t *r, - ngx_uint_t merge_slashes); + ngx_uint_t merge_slashes, ngx_uint_t uri_normalization_percent_decode); ngx_int_t ngx_http_parse_status_line(ngx_http_request_t *r, ngx_buf_t *b, ngx_http_status_t *status); ngx_int_t ngx_http_parse_unsafe_uri(ngx_http_request_t *r, ngx_str_t *uri, diff -r cffaf3f2eec8 -r 129437ade41b src/http/ngx_http_core_module.c --- a/src/http/ngx_http_core_module.c Thu Feb 02 23:38:48 2023 +0300 +++ b/src/http/ngx_http_core_module.c Tue Feb 14 16:05:46 2023 -0500 @@ -180,6 +180,14 @@ #endif +static ngx_conf_enum_t ngx_http_core_uri_normalization_percent_decode[] = { + { ngx_string("all"), NGX_HTTP_URI_NORMALIZATION_PERCENT_DECODE_ALL }, + { ngx_string("all-except-reserved"), + NGX_HTTP_URI_NORMALIZATION_PERCENT_DECODE_ALL_EXCEPT_RESERVED }, + { ngx_null_string, 0 } +}; + + static ngx_command_t ngx_http_core_commands[] = { { ngx_string("variables_hash_max_size"), @@ -778,6 +786,13 @@ #endif + { ngx_string("uri_normalization_percent_decode"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_CONF_TAKE1, + ngx_conf_set_enum_slot, + NGX_HTTP_SRV_CONF_OFFSET, + offsetof(ngx_http_core_srv_conf_t, uri_normalization_percent_decode), + &ngx_http_core_uri_normalization_percent_decode }, + ngx_null_command }; @@ -3462,6 +3477,7 @@ cscf->ignore_invalid_headers = NGX_CONF_UNSET; cscf->merge_slashes = NGX_CONF_UNSET; cscf->underscores_in_headers = NGX_CONF_UNSET; + cscf->uri_normalization_percent_decode = NGX_CONF_UNSET; cscf->file_name = cf->conf_file->file.name.data; cscf->line = cf->conf_file->line; @@ -3539,6 +3555,10 @@ return NGX_CONF_ERROR; } + ngx_conf_merge_uint_value(conf->uri_normalization_percent_decode, + prev->uri_normalization_percent_decode, + NGX_HTTP_URI_NORMALIZATION_PERCENT_DECODE_ALL); + return NGX_CONF_OK; } diff -r cffaf3f2eec8 -r 129437ade41b src/http/ngx_http_core_module.h --- a/src/http/ngx_http_core_module.h Thu Feb 02 23:38:48 2023 +0300 +++ b/src/http/ngx_http_core_module.h Tue Feb 14 16:05:46 2023 -0500 @@ -60,6 +60,14 @@ #define NGX_HTTP_SERVER_TOKENS_BUILD 2 +/* + * these enumeration values must correspond to the enumeration values for + * NGX_ESCAPE_SKIP_PREENCODED + */ +#define NGX_HTTP_URI_NORMALIZATION_PERCENT_DECODE_ALL 0 +#define NGX_HTTP_URI_NORMALIZATION_PERCENT_DECODE_ALL_EXCEPT_RESERVED 1 + + typedef struct ngx_http_location_tree_node_s ngx_http_location_tree_node_t; typedef struct ngx_http_core_loc_conf_s ngx_http_core_loc_conf_t; @@ -200,6 +208,8 @@ ngx_flag_t merge_slashes; ngx_flag_t underscores_in_headers; + ngx_uint_t uri_normalization_percent_decode; + unsigned listen:1; #if (NGX_PCRE) unsigned captures:1; diff -r cffaf3f2eec8 -r 129437ade41b src/http/ngx_http_parse.c --- a/src/http/ngx_http_parse.c Thu Feb 02 23:38:48 2023 +0300 +++ b/src/http/ngx_http_parse.c Tue Feb 14 16:05:46 2023 -0500 @@ -1245,9 +1245,11 @@ ngx_int_t -ngx_http_parse_complex_uri(ngx_http_request_t *r, ngx_uint_t merge_slashes) +ngx_http_parse_complex_uri(ngx_http_request_t *r, ngx_uint_t merge_slashes, + ngx_uint_t uri_normalization_percent_decode) { u_char c, ch, decoded, *p, *u; + uint32_t* decode; enum { sw_usual = 0, sw_slash, @@ -1257,6 +1259,44 @@ sw_quoted_second } state, quoted_state; + static uint32_t all[] = { + 0xffffffff, /* 1111 1111 1111 1111 1111 1111 1111 1111 */ + + /* ?>=< ;:98 7654 3210 /.-, +*)( '&%$ #"! */ + 0xffffffff, /* 1111 1111 1111 1111 1111 1111 1111 1111 */ + + /* _^]\ [ZYX WVUT SRQP ONML KJIH GFED CBA@ */ + 0xffffffff, /* 1111 1111 1111 1111 1111 1111 1111 1111 */ + + /* ~}| {zyx wvut srqp onml kjih gfed cba` */ + 0xffffffff, /* 1111 1111 1111 1111 1111 1111 1111 1111 */ + + 0xffffffff, /* 1111 1111 1111 1111 1111 1111 1111 1111 */ + 0xffffffff, /* 1111 1111 1111 1111 1111 1111 1111 1111 */ + 0xffffffff, /* 1111 1111 1111 1111 1111 1111 1111 1111 */ + 0xffffffff /* 1111 1111 1111 1111 1111 1111 1111 1111 */ + }; + + static uint32_t all_except_reserved[] = { + 0xffffffff, /* 1111 1111 1111 1111 1111 1111 1111 1111 */ + + /* ?>=< ;:98 7654 3210 /.-, +*)( '&%$ #"! */ + 0x53ff6025, /* 0101 0011 1111 1111 0110 0000 0010 0101 */ + + /* _^]\ [ZYX WVUT SRQP ONML KJIH GFED CBA@ */ + 0xd7fffffe, /* 1101 0111 1111 1111 1111 1111 1111 1110 */ + + /* ~}| {zyx wvut srqp onml kjih gfed cba` */ + 0xffffffff, /* 1111 1111 1111 1111 1111 1111 1111 1111 */ + + 0xffffffff, /* 1111 1111 1111 1111 1111 1111 1111 1111 */ + 0xffffffff, /* 1111 1111 1111 1111 1111 1111 1111 1111 */ + 0xffffffff, /* 1111 1111 1111 1111 1111 1111 1111 1111 */ + 0xffffffff /* 1111 1111 1111 1111 1111 1111 1111 1111 */ + }; + + static uint32_t *decode_map[] = { all, all_except_reserved }; + #if (NGX_SUPPRESS_WARN) decoded = '\0'; quoted_state = sw_usual; @@ -1267,6 +1307,7 @@ u = r->uri.data; r->uri_ext = NULL; r->args_start = NULL; + decode = decode_map[uri_normalization_percent_decode]; if (r->empty_path_in_uri) { *u++ = '/'; @@ -1520,6 +1561,14 @@ if (ch >= '0' && ch <= '9') { ch = (u_char) ((decoded << 4) + (ch - '0')); + if (!(decode[ch >> 5] & (1U << (ch & 0x1f)))) { + state = sw_usual; + ngx_memcpy(u, p - 3, 3); + u += 3; + ch = *p++; + break; + } + if (ch == '%' || ch == '#') { state = sw_usual; *u++ = ch; @@ -1538,6 +1587,14 @@ if (c >= 'a' && c <= 'f') { ch = (u_char) ((decoded << 4) + (c - 'a') + 10); + if (!(decode[ch >> 5] & (1U << (ch & 0x1f)))) { + state = sw_usual; + ngx_memcpy(u, p - 3, 3); + u += 3; + ch = *p++; + break; + } + if (ch == '?') { state = sw_usual; *u++ = ch; diff -r cffaf3f2eec8 -r 129437ade41b src/http/ngx_http_request.c --- a/src/http/ngx_http_request.c Thu Feb 02 23:38:48 2023 +0300 +++ b/src/http/ngx_http_request.c Tue Feb 14 16:05:46 2023 -0500 @@ -1234,7 +1234,9 @@ cscf = ngx_http_get_module_srv_conf(r, ngx_http_core_module); - if (ngx_http_parse_complex_uri(r, cscf->merge_slashes) != NGX_OK) { + if (ngx_http_parse_complex_uri(r, cscf->merge_slashes, + cscf->uri_normalization_percent_decode) != NGX_OK) + { r->uri.len = 0; ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, diff -r cffaf3f2eec8 -r 129437ade41b src/http/ngx_http_script.c --- a/src/http/ngx_http_script.c Thu Feb 02 23:38:48 2023 +0300 +++ b/src/http/ngx_http_script.c Tue Feb 14 16:05:46 2023 -0500 @@ -1044,11 +1044,14 @@ ngx_http_script_engine_t le; ngx_http_script_len_code_pt lcode; ngx_http_script_regex_code_t *code; + ngx_http_core_srv_conf_t *cscf; code = (ngx_http_script_regex_code_t *) e->ip; r = e->request; + cscf = ngx_http_get_module_srv_conf(r, ngx_http_core_module); + ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, "http script regex: \"%V\"", &code->name); @@ -1146,7 +1149,7 @@ if (code->uri) { if (r->ncaptures && (r->quoted_uri || r->plus_in_uri)) { e->buf.len += 2 * ngx_escape_uri(NULL, r->uri.data, r->uri.len, - NGX_ESCAPE_ARGS); + NGX_ESCAPE_ARGS, cscf->uri_normalization_percent_decode); } } @@ -1339,9 +1342,12 @@ ngx_uint_t n; ngx_http_request_t *r; ngx_http_script_copy_capture_code_t *code; + ngx_http_core_srv_conf_t *cscf; r = e->request; + cscf = ngx_http_get_module_srv_conf(r, ngx_http_core_module); + code = (ngx_http_script_copy_capture_code_t *) e->ip; e->ip += sizeof(ngx_http_script_copy_capture_code_t); @@ -1359,7 +1365,7 @@ return cap[n + 1] - cap[n] + 2 * ngx_escape_uri(NULL, &p[cap[n]], cap[n + 1] - cap[n], - NGX_ESCAPE_ARGS); + NGX_ESCAPE_ARGS, cscf->uri_normalization_percent_decode); } else { return cap[n + 1] - cap[n]; } @@ -1377,9 +1383,12 @@ ngx_uint_t n; ngx_http_request_t *r; ngx_http_script_copy_capture_code_t *code; + ngx_http_core_srv_conf_t *cscf; r = e->request; + cscf = ngx_http_get_module_srv_conf(r, ngx_http_core_module); + code = (ngx_http_script_copy_capture_code_t *) e->ip; e->ip += sizeof(ngx_http_script_copy_capture_code_t); @@ -1397,8 +1406,7 @@ && (e->request->quoted_uri || e->request->plus_in_uri)) { e->pos = (u_char *) ngx_escape_uri(pos, &p[cap[n]], - cap[n + 1] - cap[n], - NGX_ESCAPE_ARGS); + cap[n + 1] - cap[n], NGX_ESCAPE_ARGS, cscf->uri_normalization_percent_decode); } else { e->pos = ngx_copy(pos, &p[cap[n]], cap[n + 1] - cap[n]); } diff -r cffaf3f2eec8 -r 129437ade41b src/http/ngx_http_special_response.c --- a/src/http/ngx_http_special_response.c Thu Feb 02 23:38:48 2023 +0300 +++ b/src/http/ngx_http_special_response.c Tue Feb 14 16:05:46 2023 -0500 @@ -797,7 +797,8 @@ len = r->headers_out.location->value.len; location = r->headers_out.location->value.data; - escape = 2 * ngx_escape_uri(NULL, location, len, NGX_ESCAPE_REFRESH); + escape = 2 * ngx_escape_uri(NULL, location, len, NGX_ESCAPE_REFRESH, + NGX_ESCAPE_SKIP_PREENCODED_NONE); size = sizeof(ngx_http_msie_refresh_head) - 1 + escape + len @@ -841,7 +842,8 @@ p = ngx_cpymem(p, location, len); } else { - p = (u_char *) ngx_escape_uri(p, location, len, NGX_ESCAPE_REFRESH); + p = (u_char *) ngx_escape_uri(p, location, len, NGX_ESCAPE_REFRESH, + NGX_ESCAPE_SKIP_PREENCODED_NONE); } b->last = ngx_cpymem(p, ngx_http_msie_refresh_tail, diff -r cffaf3f2eec8 -r 129437ade41b src/mail/ngx_mail_auth_http_module.c --- a/src/mail/ngx_mail_auth_http_module.c Thu Feb 02 23:38:48 2023 +0300 +++ b/src/mail/ngx_mail_auth_http_module.c Tue Feb 14 16:05:46 2023 -0500 @@ -1478,7 +1478,8 @@ u_char *p; uintptr_t n; - n = ngx_escape_uri(NULL, text->data, text->len, NGX_ESCAPE_MAIL_AUTH); + n = ngx_escape_uri(NULL, text->data, text->len, NGX_ESCAPE_MAIL_AUTH, + NGX_ESCAPE_SKIP_PREENCODED_NONE); if (n == 0) { *escaped = *text; @@ -1492,7 +1493,8 @@ return NGX_ERROR; } - (void) ngx_escape_uri(p, text->data, text->len, NGX_ESCAPE_MAIL_AUTH); + (void) ngx_escape_uri(p, text->data, text->len, NGX_ESCAPE_MAIL_AUTH, + NGX_ESCAPE_SKIP_PREENCODED_NONE); escaped->data = p; ________________________________ Confidentiality notice This e-mail message and any attachment hereto contain confidential information which may be privileged and which is intended for the exclusive use of its addressee(s). If you receive this message in error, please inform sender immediately and destroy any copy thereof. Furthermore, any disclosure, distribution or copying of this message and/or any attachment hereto without the consent of the sender is strictly prohibited. Thank you. From jonas at huenig.name Wed Feb 15 17:50:32 2023 From: jonas at huenig.name (=?UTF-8?Q?Jonas_H=c3=bcnig?=) Date: Wed, 15 Feb 2023 18:50:32 +0100 Subject: [quic] can't define more than one server Message-ID: <79c55e27-b64f-a948-128f-fc1dfdb44112@huenig.name> Hi, I installed the new prebuild nginx package with http/3 support. This works very well for one server block, but when I add the listen directives to a second server block I get this error on configtest: nginx: [emerg] duplicate listen options for X.X.X.X:443 in /etc/nginx/sites-available/quic2.c-X.maxcluster.net/userdefined.conf.init:1 nginx: configuration file /etc/nginx/nginx.conf test failed anyone has an idea about this? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jiuzhoucui at 163.com Thu Feb 16 13:36:52 2023 From: jiuzhoucui at 163.com (Jiuzhou Cui) Date: Thu, 16 Feb 2023 21:36:52 +0800 (CST) Subject: QUIC: add error code for handshake failed. Message-ID: <5b0eee35.4de7.1865a7130cb.Coremail.jiuzhoucui@163.com> Hello! # HG changeset patch # User Jiuzhou Cui # Date 1676554419 -28800 # Thu Feb 16 21:33:39 2023 +0800 # Branch quic # Node ID 13396c3ad10bdc8c1ac6969e965ceac510dc162f # Parent b87a0dbc1150f415def5bc1e1f00d02b33519026 QUIC: add error code for handshake failed. diff -r b87a0dbc1150 -r 13396c3ad10b src/event/quic/ngx_event_quic_ssl.c --- a/src/event/quic/ngx_event_quic_ssl.c Tue Oct 25 12:52:09 2022 +0400 +++ b/src/event/quic/ngx_event_quic_ssl.c Thu Feb 16 21:33:39 2023 +0800 @@ -202,7 +202,7 @@ SSL_get0_alpn_selected(ssl_conn, &alpn_data, &alpn_len); if (alpn_len == 0) { - qc->error = 0x100 + SSL_AD_NO_APPLICATION_PROTOCOL; + qc->error = NGX_QUIC_ERR_CRYPTO(SSL_AD_NO_APPLICATION_PROTOCOL); qc->error_reason = "unsupported protocol in ALPN extension"; ngx_log_error(NGX_LOG_INFO, c->log, 0, @@ -452,6 +452,7 @@ if (sslerr != SSL_ERROR_WANT_READ) { ngx_ssl_error(NGX_LOG_ERR, c->log, 0, "SSL_do_handshake() failed"); + qc->error = NGX_QUIC_ERR_CRYPTO(sslerr); qc->error_reason = "handshake failed"; return NGX_ERROR; } -------------- next part -------------- An HTML attachment was scrubbed... URL: From pluknet at nginx.com Thu Feb 16 14:42:27 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Thu, 16 Feb 2023 18:42:27 +0400 Subject: QUIC: add error code for handshake failed. In-Reply-To: <5b0eee35.4de7.1865a7130cb.Coremail.jiuzhoucui@163.com> References: <5b0eee35.4de7.1865a7130cb.Coremail.jiuzhoucui@163.com> Message-ID: <9CFEFC7C-19E1-4AFD-8CE5-2D55AAC02AA8@nginx.com> > On 16 Feb 2023, at 17:36, Jiuzhou Cui wrote: > > Hello! > > # HG changeset patch > # User Jiuzhou Cui > # Date 1676554419 -28800 > # Thu Feb 16 21:33:39 2023 +0800 > # Branch quic > # Node ID 13396c3ad10bdc8c1ac6969e965ceac510dc162f > # Parent b87a0dbc1150f415def5bc1e1f00d02b33519026 > QUIC: add error code for handshake failed. > > diff -r b87a0dbc1150 -r 13396c3ad10b src/event/quic/ngx_event_quic_ssl.c > --- a/src/event/quic/ngx_event_quic_ssl.c Tue Oct 25 12:52:09 2022 +0400 > +++ b/src/event/quic/ngx_event_quic_ssl.c Thu Feb 16 21:33:39 2023 +0800 > @@ -202,7 +202,7 @@ > SSL_get0_alpn_selected(ssl_conn, &alpn_data, &alpn_len); > > if (alpn_len == 0) { > - qc->error = 0x100 + SSL_AD_NO_APPLICATION_PROTOCOL; > + qc->error = NGX_QUIC_ERR_CRYPTO(SSL_AD_NO_APPLICATION_PROTOCOL); > qc->error_reason = "unsupported protocol in ALPN extension"; > > ngx_log_error(NGX_LOG_INFO, c->log, 0, > @@ -452,6 +452,7 @@ > > if (sslerr != SSL_ERROR_WANT_READ) { > ngx_ssl_error(NGX_LOG_ERR, c->log, 0, "SSL_do_handshake() failed"); > + qc->error = NGX_QUIC_ERR_CRYPTO(sslerr); > qc->error_reason = "handshake failed"; > return NGX_ERROR; > } Thank you for the patch. Applying to TLS handshake, qc->error is used to keep CRYPTO_ERROR, a value based on the TLS alert. You are trying to set there something different, this is not going to work. More, qc->error is usually set in the send_alert callback as passed from TLS, so no need to deal with it here. Other places are QUIC protocol- specific additions to negotiate ALPN and to carry transport parameters, they are managed elsewhere. The ALPN part is ok. Looks like it was missed from the 97adb87f149b change, which went soon after ALPN checks added in a2c34e77cfc1. # HG changeset patch # User Sergey Kandaurov # Date 1676558213 -14400 # Thu Feb 16 18:36:53 2023 +0400 # Branch quic # Node ID 2fcd590d85da9c3a0205a18cb295ec316c03f18e # Parent 12b756caaf167d2239fd3bd7a75b270ca89ca26b QUIC: using NGX_QUIC_ERR_CRYPTO macro in ALPN checks. Patch by Jiuzhou Cui. diff --git a/src/event/quic/ngx_event_quic_ssl.c b/src/event/quic/ngx_event_quic_ssl.c --- a/src/event/quic/ngx_event_quic_ssl.c +++ b/src/event/quic/ngx_event_quic_ssl.c @@ -190,7 +190,7 @@ ngx_quic_add_handshake_data(ngx_ssl_conn SSL_get0_alpn_selected(ssl_conn, &alpn_data, &alpn_len); if (alpn_len == 0) { - qc->error = 0x100 + SSL_AD_NO_APPLICATION_PROTOCOL; + qc->error = NGX_QUIC_ERR_CRYPTO(SSL_AD_NO_APPLICATION_PROTOCOL); qc->error_reason = "unsupported protocol in ALPN extension"; ngx_log_error(NGX_LOG_INFO, c->log, 0, -- Sergey Kandaurov From jiuzhoucui at 163.com Fri Feb 17 05:17:13 2023 From: jiuzhoucui at 163.com (Jiuzhou Cui) Date: Fri, 17 Feb 2023 13:17:13 +0800 (CST) Subject: QUIC: add error code for handshake failed. In-Reply-To: <9CFEFC7C-19E1-4AFD-8CE5-2D55AAC02AA8@nginx.com> References: <5b0eee35.4de7.1865a7130cb.Coremail.jiuzhoucui@163.com> <9CFEFC7C-19E1-4AFD-8CE5-2D55AAC02AA8@nginx.com> Message-ID: <170ee7d0.234e.1865dce18a2.Coremail.jiuzhoucui@163.com> Maybe NGX_QUIC_ERR_CRYPTO(sslerr) is not suitable here. I think error and error_reason should be set together. My scenes is error_reason is "handshake failed", but error is 8 set by parse TRANSPORT_PARAMETER failed before. My actual problem is parse transport parameter failed, so maybe error_reason "failed to process transport parameters" is useful for me to positioning problem. What do you think about this? At 2023-02-16 22:42:27, "Sergey Kandaurov" wrote: > >> On 16 Feb 2023, at 17:36, Jiuzhou Cui wrote: >> >> Hello! >> >> # HG changeset patch >> # User Jiuzhou Cui >> # Date 1676554419 -28800 >> # Thu Feb 16 21:33:39 2023 +0800 >> # Branch quic >> # Node ID 13396c3ad10bdc8c1ac6969e965ceac510dc162f >> # Parent b87a0dbc1150f415def5bc1e1f00d02b33519026 >> QUIC: add error code for handshake failed. >> >> diff -r b87a0dbc1150 -r 13396c3ad10b src/event/quic/ngx_event_quic_ssl.c >> --- a/src/event/quic/ngx_event_quic_ssl.c Tue Oct 25 12:52:09 2022 +0400 >> +++ b/src/event/quic/ngx_event_quic_ssl.c Thu Feb 16 21:33:39 2023 +0800 >> @@ -202,7 +202,7 @@ >> SSL_get0_alpn_selected(ssl_conn, &alpn_data, &alpn_len); >> >> if (alpn_len == 0) { >> - qc->error = 0x100 + SSL_AD_NO_APPLICATION_PROTOCOL; >> + qc->error = NGX_QUIC_ERR_CRYPTO(SSL_AD_NO_APPLICATION_PROTOCOL); >> qc->error_reason = "unsupported protocol in ALPN extension"; >> >> ngx_log_error(NGX_LOG_INFO, c->log, 0, >> @@ -452,6 +452,7 @@ >> >> if (sslerr != SSL_ERROR_WANT_READ) { >> ngx_ssl_error(NGX_LOG_ERR, c->log, 0, "SSL_do_handshake() failed"); >> + qc->error = NGX_QUIC_ERR_CRYPTO(sslerr); >> qc->error_reason = "handshake failed"; >> return NGX_ERROR; >> } > >Thank you for the patch. > >Applying to TLS handshake, qc->error is used to keep CRYPTO_ERROR, >a value based on the TLS alert. You are trying to set there >something different, this is not going to work. > >More, qc->error is usually set in the send_alert callback as passed from >TLS, so no need to deal with it here. Other places are QUIC protocol- >specific additions to negotiate ALPN and to carry transport parameters, >they are managed elsewhere. > >The ALPN part is ok. Looks like it was missed from the 97adb87f149b >change, which went soon after ALPN checks added in a2c34e77cfc1. > ># HG changeset patch ># User Sergey Kandaurov ># Date 1676558213 -14400 ># Thu Feb 16 18:36:53 2023 +0400 ># Branch quic ># Node ID 2fcd590d85da9c3a0205a18cb295ec316c03f18e ># Parent 12b756caaf167d2239fd3bd7a75b270ca89ca26b >QUIC: using NGX_QUIC_ERR_CRYPTO macro in ALPN checks. > >Patch by Jiuzhou Cui. > >diff --git a/src/event/quic/ngx_event_quic_ssl.c b/src/event/quic/ngx_event_quic_ssl.c >--- a/src/event/quic/ngx_event_quic_ssl.c >+++ b/src/event/quic/ngx_event_quic_ssl.c >@@ -190,7 +190,7 @@ ngx_quic_add_handshake_data(ngx_ssl_conn > SSL_get0_alpn_selected(ssl_conn, &alpn_data, &alpn_len); > > if (alpn_len == 0) { >- qc->error = 0x100 + SSL_AD_NO_APPLICATION_PROTOCOL; >+ qc->error = NGX_QUIC_ERR_CRYPTO(SSL_AD_NO_APPLICATION_PROTOCOL); > qc->error_reason = "unsupported protocol in ALPN extension"; > > ngx_log_error(NGX_LOG_INFO, c->log, 0, > >-- >Sergey Kandaurov >_______________________________________________ >nginx-devel mailing list >nginx-devel at nginx.org >https://mailman.nginx.org/mailman/listinfo/nginx-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From pluknet at nginx.com Fri Feb 17 12:53:24 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Fri, 17 Feb 2023 16:53:24 +0400 Subject: QUIC: add error code for handshake failed. In-Reply-To: <170ee7d0.234e.1865dce18a2.Coremail.jiuzhoucui@163.com> References: <5b0eee35.4de7.1865a7130cb.Coremail.jiuzhoucui@163.com> <9CFEFC7C-19E1-4AFD-8CE5-2D55AAC02AA8@nginx.com> <170ee7d0.234e.1865dce18a2.Coremail.jiuzhoucui@163.com> Message-ID: <1787FA6E-24CB-4CCE-8A5D-3EFBA7BD9AEC@nginx.com> > On 17 Feb 2023, at 09:17, Jiuzhou Cui wrote: > > Maybe NGX_QUIC_ERR_CRYPTO(sslerr) is not suitable here. > > I think error and error_reason should be set together. > > My scenes is error_reason is "handshake failed", but error is 8 set by parse TRANSPORT_PARAMETER failed before. My actual problem is parse transport parameter failed, so maybe error_reason "failed to process transport parameters" is useful for me to positioning problem. What do you think about this? > Actually, both error and error_reason are usually set together. The problem is in overwriting error_reason that was previously set specifically for transport parameters parsing error, during handshake, with a generic "handshake failed" phrase. Please see the patch. # HG changeset patch # User Sergey Kandaurov # Date 1676638271 -14400 # Fri Feb 17 16:51:11 2023 +0400 # Branch quic # Node ID 09c7414487aa82b40d4ab33738dc578363fe3273 # Parent 2fcd590d85da9c3a0205a18cb295ec316c03f18e QUIC: using most specific handshake error reasons. A QUIC handshake may fail for a variety of reasons. Some of them arise from our own checks and have a knownledge to set a suitable reason phrase. Previously, it was overridden with a generic "handshake failed" phrase. The latter is used for handshake errors generated in the SSL library and reported with the send_alert callback. Now a specific phrase is preserved. Reported by Jiuzhou Cui. diff --git a/src/event/quic/ngx_event_quic_ssl.c b/src/event/quic/ngx_event_quic_ssl.c --- a/src/event/quic/ngx_event_quic_ssl.c +++ b/src/event/quic/ngx_event_quic_ssl.c @@ -423,7 +423,11 @@ ngx_quic_crypto_input(ngx_connection_t * if (sslerr != SSL_ERROR_WANT_READ) { ngx_ssl_error(NGX_LOG_ERR, c->log, 0, "SSL_do_handshake() failed"); - qc->error_reason = "handshake failed"; + + if (!qc->error_reason) { + qc->error_reason = "handshake failed"; + } + return NGX_ERROR; } } > > At 2023-02-16 22:42:27, "Sergey Kandaurov" wrote: > > > >> On 16 Feb 2023, at 17:36, Jiuzhou Cui wrote: > >> > >> Hello! > >> > >> # HG changeset patch > >> # User Jiuzhou Cui > >> # Date 1676554419 -28800 > >> # Thu Feb 16 21:33:39 2023 +0800 > >> # Branch quic > >> # Node ID 13396c3ad10bdc8c1ac6969e965ceac510dc162f > >> # Parent b87a0dbc1150f415def5bc1e1f00d02b33519026 > >> QUIC: add error code for handshake failed. > >> > >> diff -r b87a0dbc1150 -r 13396c3ad10b src/event/quic/ngx_event_quic_ssl.c > >> --- a/src/event/quic/ngx_event_quic_ssl.c Tue Oct 25 12:52:09 2022 +0400 > >> +++ b/src/event/quic/ngx_event_quic_ssl.c Thu Feb 16 21:33:39 2023 +0800 > >> @@ -202,7 +202,7 @@ > >> SSL_get0_alpn_selected(ssl_conn, &alpn_data, &alpn_len); > >> > >> if (alpn_len == 0) { > >> - qc->error = 0x100 + SSL_AD_NO_APPLICATION_PROTOCOL; > >> + qc->error = NGX_QUIC_ERR_CRYPTO(SSL_AD_NO_APPLICATION_PROTOCOL); > >> qc->error_reason = "unsupported protocol in ALPN extension"; > >> > >> ngx_log_error(NGX_LOG_INFO, c->log, 0, > >> @@ -452,6 +452,7 @@ > >> > >> if (sslerr != SSL_ERROR_WANT_READ) { > >> ngx_ssl_error(NGX_LOG_ERR, c->log, 0, "SSL_do_handshake() failed"); > >> + qc->error = NGX_QUIC_ERR_CRYPTO(sslerr); > >> qc->error_reason = "handshake failed"; > >> return NGX_ERROR; > >> } > > > >Thank you for the patch. > > > >Applying to TLS handshake, qc->error is used to keep CRYPTO_ERROR, > >a value based on the TLS alert. You are trying to set there > >something different, this is not going to work. > > > >More, qc->error is usually set in the send_alert callback as passed from > >TLS, so no need to deal with it here. Other places are QUIC protocol- > >specific additions to negotiate ALPN and to carry transport parameters, > >they are managed elsewhere. > > > >The ALPN part is ok. Looks like it was missed from the 97adb87f149b > >change, which went soon after ALPN checks added in a2c34e77cfc1. > > > ># HG changeset patch > ># User Sergey Kandaurov > ># Date 1676558213 -14400 > ># Thu Feb 16 18:36:53 2023 +0400 > ># Branch quic > ># Node ID 2fcd590d85da9c3a0205a18cb295ec316c03f18e > ># Parent 12b756caaf167d2239fd3bd7a75b270ca89ca26b > >QUIC: using NGX_QUIC_ERR_CRYPTO macro in ALPN checks. > > > >Patch by Jiuzhou Cui. > > > >diff --git a/src/event/quic/ngx_event_quic_ssl.c b/src/event/quic/ngx_event_quic_ssl.c > >--- a/src/event/quic/ngx_event_quic_ssl.c > >+++ b/src/event/quic/ngx_event_quic_ssl.c > >@@ -190,7 +190,7 @@ ngx_quic_add_handshake_data(ngx_ssl_conn > > SSL_get0_alpn_selected(ssl_conn, &alpn_data, &alpn_len); > > > > if (alpn_len == 0) { > >- qc->error = 0x100 + SSL_AD_NO_APPLICATION_PROTOCOL; > >+ qc->error = NGX_QUIC_ERR_CRYPTO(SSL_AD_NO_APPLICATION_PROTOCOL); > > qc->error_reason = "unsupported protocol in ALPN extension"; > > > > ngx_log_error(NGX_LOG_INFO, c->log, 0, > > > >-- > >Sergey Kandaurov > >_______________________________________________ > >nginx-devel mailing list > >nginx-devel at nginx.org > >https://mailman.nginx.org/mailman/listinfo/nginx-devel > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > https://mailman.nginx.org/mailman/listinfo/nginx-devel -- Sergey Kandaurov From mdounin at mdounin.ru Fri Feb 17 12:59:12 2023 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 17 Feb 2023 15:59:12 +0300 Subject: [PATCH] HTTP: Add new uri_normalization_percent_decode option In-Reply-To: <129437ade41b14a584fb.1676479813@dev2-97.sol-local> References: <129437ade41b14a584fb.1676479813@dev2-97.sol-local> Message-ID: Hello! On Wed, Feb 15, 2023 at 11:50:13AM -0500, Michael Kourlas via nginx-devel wrote: > # HG changeset patch > # User Michael Kourlas > # Date 1676408746 18000 > # Tue Feb 14 16:05:46 2023 -0500 > # Node ID 129437ade41b14a584fb4b7558accc1b8dee7f45 > # Parent cffaf3f2eec8fd33605c2a37814f5ffc30371989 > HTTP: Add new uri_normalization_percent_decode option > > This patch addresses ticket #2225 by adding a new > uri_normalization_percent_decode configuration option that controls which > characters are percent-decoded by nginx as part of its URI normalization. > > The option has two values: "all" and "all-except-reserved". "all" is the > default value and is the current behaviour. When the option is set to > "all-except-reserved", nginx percent-decodes all characters except those in the > reserved set defined by RFC 3986: > > reserved = gen-delims / sub-delims > > gen-delims = ":" / "/" / "?" / "#" / "[" / "]" / "@" > > sub-delims = "!" / "$" / "&" / "'" / "(" / ")" > / "*" / "+" / "," / ";" / "=" > > In addition, when "all-except-reserved" is used, nginx will not re-encode "%" > from the request URI when it observes that it is part of a percent-encoded > reserved character. > > When nginx percent-decodes reserved characters, this can often change the > request URI's semantics, making it impossible to use a normalized URI for > certain use cases. "uri_normalization_percent_decode" gives the configuration > author the freedom to determine which reserved characters are semantically > relevant and which are not. > > For example, consider the following location block, which handles part of a > hypothetical API: > > location ~ ^/api/objects/[^/]+/subobjects(/.*)?$ { > ... > } > > Because nginx always normalizes "%2F" to "/", this location block will not > match a path of /api/objects/sample%2Fname/subobjects, even if the API permits > "/" to appear percent-encoded in the URI as part of object names. nginx will > instead interpret this as /api/objects/sample/name/subobjects, a completely > different path. Setting "uri_normalization_percent_decode" to > "all-except-reserved" will leave "%2F" encoded, resulting in the expected > behaviour. Thanks for the patch. As far as I understand, it will irreversibly corrupt URIs with double-encoded reserved characters. For example, "%252F" will become "%2F" when proxying in the following configuration: location /foo/ { proxy_pass http://upstream/foo/; } Further, requests to static files with (properly escaped) reserved characters will simply fail, because nginx won't decode these characters. For example, in the following trivial configuration a request to "/foo%3Fbar" won't be decoded to match "/foo?bar" file under the document root: location / { # static files } Please also note that the configuration directive you've introduced in this patch applies to URI parsing from not-yet-final server block (see [1] for details), but the configuration from the final server block will be used for URI escaping. These configuration can be different, and this might result in various additional issues. Overall, I tend to think that the suggested patch will introduce much more problems than it tries to solve, and I would rather not. [1] http://nginx.org/en/docs/http/server_names.html#virtual_server_selection -- Maxim Dounin http://mdounin.ru/ From pluknet at nginx.com Fri Feb 17 14:38:54 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Fri, 17 Feb 2023 18:38:54 +0400 Subject: [PATCH 01 of 12] Win32: non-ASCII names support in autoindex (ticket #458) In-Reply-To: <60d845f9505fe1b97c1e.1673559324@1.0.0.127.in-addr.arpa> References: <60d845f9505fe1b97c1e.1673559324@1.0.0.127.in-addr.arpa> Message-ID: > On 13 Jan 2023, at 01:35, Maxim Dounin wrote: > > # HG changeset patch > # User Maxim Dounin > # Date 1673548890 -10800 > # Thu Jan 12 21:41:30 2023 +0300 > # Node ID 60d845f9505fe1b97c1e04b680523b790e29fdb1 > # Parent 07b0bee87f32be91a33210bc06973e07c4c1dac9 > Win32: non-ASCII names support in autoindex (ticket #458). > > Notably, ngx_open_dir() now supports opening directories with non-ASCII > characters, and directory entries returned by ngx_read_dir() are properly > converted to UTF-8. > > diff -r 07b0bee87f32 -r 60d845f9505f src/os/win32/ngx_files.c > --- a/src/os/win32/ngx_files.c Wed Dec 21 14:53:27 2022 +0300 > +++ b/src/os/win32/ngx_files.c Thu Jan 12 21:41:30 2023 +0300 > @@ -13,7 +13,11 @@ > > static ngx_int_t ngx_win32_check_filename(u_char *name, u_short *u, > size_t len); > -static u_short *ngx_utf8_to_utf16(u_short *utf16, u_char *utf8, size_t *len); > +static u_short *ngx_utf8_to_utf16(u_short *utf16, u_char *utf8, size_t *len, > + size_t reserved); > +static u_char *ngx_utf16_to_utf8(u_char *utf8, u_short *utf16, size_t *len, > + size_t *allocated); > +uint32_t ngx_utf16_decode(u_short **u, size_t n); > > > /* FILE_FLAG_BACKUP_SEMANTICS allows to obtain a handle to a directory */ > @@ -28,7 +32,7 @@ ngx_open_file(u_char *name, u_long mode, > u_short utf16[NGX_UTF16_BUFLEN]; > > len = NGX_UTF16_BUFLEN; > - u = ngx_utf8_to_utf16(utf16, name, &len); > + u = ngx_utf8_to_utf16(utf16, name, &len, 0); > > if (u == NULL) { > return INVALID_HANDLE_VALUE; > @@ -269,7 +273,7 @@ ngx_file_info(u_char *file, ngx_file_inf > > len = NGX_UTF16_BUFLEN; > > - u = ngx_utf8_to_utf16(utf16, file, &len); > + u = ngx_utf8_to_utf16(utf16, file, &len, 0); > > if (u == NULL) { > return NGX_FILE_ERROR; > @@ -427,49 +431,51 @@ ngx_realpath(u_char *path, u_char *resol > ngx_int_t > ngx_open_dir(ngx_str_t *name, ngx_dir_t *dir) > { > - u_char *pattern, *p; > + size_t len; > + u_short *u, *p; > ngx_err_t err; > + u_short utf16[NGX_UTF16_BUFLEN]; > > - pattern = malloc(name->len + 3); > - if (pattern == NULL) { > + len = NGX_UTF16_BUFLEN - 2; > + u = ngx_utf8_to_utf16(utf16, name->data, &len, 2); > + > + if (u == NULL) { > return NGX_ERROR; > } > > - p = ngx_cpymem(pattern, name->data, name->len); > + if (ngx_win32_check_filename(name->data, u, len) != NGX_OK) { > + goto failed; > + } > + > + p = &u[len - 1]; > > *p++ = '/'; > *p++ = '*'; > *p = '\0'; > > - dir->dir = FindFirstFile((const char *) pattern, &dir->finddata); > + dir->dir = FindFirstFileW(u, &dir->finddata); > > if (dir->dir == INVALID_HANDLE_VALUE) { > - err = ngx_errno; > - ngx_free(pattern); > - ngx_set_errno(err); > - return NGX_ERROR; > + goto failed; > } > > - ngx_free(pattern); > + if (u != utf16) { > + ngx_free(u); > + } > > dir->valid_info = 1; > dir->ready = 1; > + dir->name = NULL; > + dir->allocated = 0; > > return NGX_OK; > -} > > +failed: > > -ngx_int_t > -ngx_read_dir(ngx_dir_t *dir) > -{ > - if (dir->ready) { > - dir->ready = 0; > - return NGX_OK; > - } > - > - if (FindNextFile(dir->dir, &dir->finddata) != 0) { > - dir->type = 1; > - return NGX_OK; > + if (u != utf16) { > + err = ngx_errno; > + ngx_free(u); > + ngx_set_errno(err); > } > > return NGX_ERROR; > @@ -477,8 +483,56 @@ ngx_read_dir(ngx_dir_t *dir) > > > ngx_int_t > +ngx_read_dir(ngx_dir_t *dir) > +{ > + u_char *name; > + size_t len, allocated; > + > + if (dir->ready) { > + dir->ready = 0; > + goto convert; > + } > + > + if (FindNextFileW(dir->dir, &dir->finddata) != 0) { > + dir->type = 1; > + goto convert; > + } > + > + return NGX_ERROR; > + > +convert: > + > + name = dir->name; > + len = dir->allocated; > + > + name = ngx_utf16_to_utf8(name, dir->finddata.cFileName, &len, &allocated); > + if (name == NULL) { Note that you just return on NULL, but see below. > + return NGX_ERROR; > + } > + > + if (name != dir->name) { > + > + if (dir->name) { > + ngx_free(dir->name); > + } > + > + dir->name = name; > + dir->allocated = allocated; > + } > + > + dir->namelen = len - 1; > + > + return NGX_OK; > +} > + > + > +ngx_int_t > ngx_close_dir(ngx_dir_t *dir) > { > + if (dir->name) { > + ngx_free(dir->name); > + } > + > if (FindClose(dir->dir) == 0) { > return NGX_ERROR; > } > @@ -816,7 +870,7 @@ failed: > > > static u_short * > -ngx_utf8_to_utf16(u_short *utf16, u_char *utf8, size_t *len) > +ngx_utf8_to_utf16(u_short *utf16, u_char *utf8, size_t *len, size_t reserved) > { > u_char *p; > u_short *u, *last; > @@ -865,7 +919,7 @@ ngx_utf8_to_utf16(u_short *utf16, u_char > > /* the given buffer is not enough, allocate a new one */ > > - u = malloc(((p - utf8) + ngx_strlen(p) + 1) * sizeof(u_short)); > + u = malloc(((p - utf8) + ngx_strlen(p) + 1 + reserved) * sizeof(u_short)); > if (u == NULL) { > return NULL; > } > @@ -910,3 +964,170 @@ ngx_utf8_to_utf16(u_short *utf16, u_char > > /* unreachable */ > } > + > + > +static u_char * > +ngx_utf16_to_utf8(u_char *utf8, u_short *utf16, size_t *len, size_t *allocated) > +{ > + u_char *p, *last; > + u_short *u, *j; > + uint32_t n; > + > + u = utf16; > + p = utf8; > + last = utf8 + *len; > + > + while (p < last) { > + > + if (*u < 0x80) { > + *p++ = (u_char) *u; > + > + if (*u == 0) { > + *len = p - utf8; > + return utf8; > + } > + > + u++; > + > + continue; > + } > + > + if (p >= last - 4) { > + *len = p - utf8; > + break; > + } > + > + n = ngx_utf16_decode(&u, 2); > + > + if (n > 0x10ffff) { > + ngx_free(utf8); > + ngx_set_errno(NGX_EILSEQ); This can result in double-free on a subsequent ngx_close_dir() call. > + return NULL; > + } > + > + if (n >= 0x10000) { > + *p++ = (u_char) (0xf0 + (n >> 18)); > + *p++ = (u_char) (0x80 + ((n >> 12) & 0x3f)); > + *p++ = (u_char) (0x80 + ((n >> 6) & 0x3f)); > + *p++ = (u_char) (0x80 + (n & 0x3f)); > + continue; > + stray ws-only line (one more below) > + } > + > + if (n >= 0x0800) { > + *p++ = (u_char) (0xe0 + (n >> 12)); > + *p++ = (u_char) (0x80 + ((n >> 6) & 0x3f)); > + *p++ = (u_char) (0x80 + (n & 0x3f)); > + continue; > + } > + > + *p++ = (u_char) (0xc0 + (n >> 6)); > + *p++ = (u_char) (0x80 + (n & 0x3f)); > + } > + > + /* the given buffer is not enough, allocate a new one */ > + > + for (j = u; *j; j++) { /* void */ } > + > + p = malloc((j - utf16) * 4 + 1); [Note that pointer arithmetics is in u_short (2 bytes) units.] It seems you are allocating memory assuming that utf16 -> utf8 conversion can result in memory expansion up to 4 bytes per unit. At first glance that's true, because UTF-8 symbol may take 4 bytes. IIUC, 4-byte UTF-8 can result only from UTF-16 surrogate pairs, this gives 2x growth (2-paired UTF-16 to 4-bytes UTF-8). Otherwise, 1/2/3-byte encoded UTF-8 can result from 2-byte UTF-16, which gives from 1x to 3x growth per unit. So, "malloc((j - utf16) * 3 + 1)" should be enough there. > + if (p == NULL) { > + return NULL; > + } Old memory is still there, allocated and referenced by dir->name, see above. It seems ok, it should be freed by ngx_close_dir() call. If we happen to change callers to not call ngx_close_dir() on error, this could turn into memory leaks, though. > + > + if (allocated) { > + *allocated = (j - utf16) * 4 + 1; > + } > + > + ngx_memcpy(p, utf8, *len); > + > + utf8 = p; > + p += *len; > + > + for ( ;; ) { > + > + if (*u < 0x80) { > + *p++ = (u_char) *u; > + > + if (*u == 0) { > + *len = p - utf8; > + return utf8; > + } > + > + u++; > + > + continue; > + } > + > + n = ngx_utf16_decode(&u, 2); > + > + if (n > 0x10ffff) { > + ngx_free(utf8); > + ngx_set_errno(NGX_EILSEQ); > + return NULL; > + } > + > + if (n >= 0x10000) { > + *p++ = (u_char) (0xf0 + (n >> 18)); > + *p++ = (u_char) (0x80 + ((n >> 12) & 0x3f)); > + *p++ = (u_char) (0x80 + ((n >> 6) & 0x3f)); > + *p++ = (u_char) (0x80 + (n & 0x3f)); > + continue; > + > + } > + > + if (n >= 0x0800) { > + *p++ = (u_char) (0xe0 + (n >> 12)); > + *p++ = (u_char) (0x80 + ((n >> 6) & 0x3f)); > + *p++ = (u_char) (0x80 + (n & 0x3f)); > + continue; > + } > + > + *p++ = (u_char) (0xc0 + (n >> 6)); > + *p++ = (u_char) (0x80 + (n & 0x3f)); > + } > + > + /* unreachable */ > +} > + > + > +/* > + * ngx_utf16_decode() decodes one or two UTF-16 code units > + * the return values: > + * 0x80 - 0x10ffff valid character > + * 0x110000 - 0xfffffffd invalid sequence > + * 0xfffffffe incomplete sequence > + * 0xffffffff error > + */ > + > +uint32_t > +ngx_utf16_decode(u_short **u, size_t n) > +{ > + uint32_t k, m; > + > + k = **u; > + > + if (k < 0xd800 || k > 0xdfff) { > + (*u)++; > + return k; > + } > + > + if (k > 0xdbff) { > + (*u)++; > + return 0xffffffff; > + } > + > + if (n < 2) { > + return 0xfffffffe; > + } > + > + (*u)++; > + > + m = *(*u)++; > + > + if (m < 0xdc00 || m > 0xdfff) { > + return 0xffffffff; > + > + } > + > + return 0x10000 + ((k - 0xd800) << 10) + (m - 0xdc00); > +} > diff -r 07b0bee87f32 -r 60d845f9505f src/os/win32/ngx_files.h > --- a/src/os/win32/ngx_files.h Wed Dec 21 14:53:27 2022 +0300 > +++ b/src/os/win32/ngx_files.h Thu Jan 12 21:41:30 2023 +0300 > @@ -30,7 +30,11 @@ typedef struct { > > typedef struct { > HANDLE dir; > - WIN32_FIND_DATA finddata; > + WIN32_FIND_DATAW finddata; > + > + u_char *name; > + size_t namelen; > + size_t allocated; > > unsigned valid_info:1; > unsigned type:1; > @@ -205,8 +209,8 @@ ngx_int_t ngx_close_dir(ngx_dir_t *dir); > #define ngx_dir_access(a) (a) > > > -#define ngx_de_name(dir) ((u_char *) (dir)->finddata.cFileName) > -#define ngx_de_namelen(dir) ngx_strlen((dir)->finddata.cFileName) > +#define ngx_de_name(dir) (dir)->name > +#define ngx_de_namelen(dir) (dir)->namelen > > ngx_int_t ngx_de_info(u_char *name, ngx_dir_t *dir); > #define ngx_de_info_n "dummy()" Otherwise, looks good. -- Sergey Kandaurov From pluknet at nginx.com Fri Feb 17 14:53:11 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Fri, 17 Feb 2023 18:53:11 +0400 Subject: [PATCH 02 of 12] Win32: non-ASCII names support in "include" with wildcards In-Reply-To: References: Message-ID: > On 13 Jan 2023, at 01:35, Maxim Dounin wrote: > > # HG changeset patch > # User Maxim Dounin > # Date 1673548899 -10800 > # Thu Jan 12 21:41:39 2023 +0300 > # Node ID d05c0adf5890aecc68ce8906ef19ca07502ed071 > # Parent 60d845f9505fe1b97c1e04b680523b790e29fdb1 > Win32: non-ASCII names support in "include" with wildcards. > > Notably, ngx_open_glob() now supports opening directories with non-ASCII > characters, and pathnames returned by ngx_read_glob() are converted to UTF-8. > > diff -r 60d845f9505f -r d05c0adf5890 src/os/win32/ngx_files.c > --- a/src/os/win32/ngx_files.c Thu Jan 12 21:41:30 2023 +0300 > +++ b/src/os/win32/ngx_files.c Thu Jan 12 21:41:39 2023 +0300 > @@ -546,14 +546,27 @@ ngx_open_glob(ngx_glob_t *gl) > { > u_char *p; > size_t len; > + u_short *u; > ngx_err_t err; > + u_short utf16[NGX_UTF16_BUFLEN]; > > - gl->dir = FindFirstFile((const char *) gl->pattern, &gl->finddata); > + len = NGX_UTF16_BUFLEN; > + u = ngx_utf8_to_utf16(utf16, gl->pattern, &len, 0); > + > + if (u == NULL) { > + return NGX_ERROR; > + } > + > + gl->dir = FindFirstFileW(u, &gl->finddata); > > if (gl->dir == INVALID_HANDLE_VALUE) { > > err = ngx_errno; > > + if (u != utf16) { > + ngx_free(u); > + } > + > if ((err == ERROR_FILE_NOT_FOUND || err == ERROR_PATH_NOT_FOUND) > && gl->test) > { > @@ -561,6 +574,8 @@ ngx_open_glob(ngx_glob_t *gl) > return NGX_OK; > } > > + ngx_set_errno(err); > + > return NGX_ERROR; > } > > @@ -570,18 +585,10 @@ ngx_open_glob(ngx_glob_t *gl) > } > } > > - len = ngx_strlen(gl->finddata.cFileName); > - gl->name.len = gl->last + len; > - > - gl->name.data = ngx_alloc(gl->name.len + 1, gl->log); > - if (gl->name.data == NULL) { > - return NGX_ERROR; > + if (u != utf16) { > + ngx_free(u); > } > > - ngx_memcpy(gl->name.data, gl->pattern, gl->last); > - ngx_cpystrn(gl->name.data + gl->last, (u_char *) gl->finddata.cFileName, > - len + 1); > - > gl->ready = 1; > > return NGX_OK; > @@ -591,40 +598,25 @@ ngx_open_glob(ngx_glob_t *gl) > ngx_int_t > ngx_read_glob(ngx_glob_t *gl, ngx_str_t *name) > { > - size_t len; > - ngx_err_t err; > + u_char *p; > + size_t len; > + ngx_err_t err; > + u_char utf8[NGX_UTF16_BUFLEN]; Mixing UTF16-specific macro with u_char utf8[] looks suspicious. It may have sense to replace this with dynamic allocation inside ngx_utf16_to_utf8(), similar to ngx_read_dir() 1st call, or use NGX_MAX_PATH. > > if (gl->no_match) { > return NGX_DONE; > } > > if (gl->ready) { > - *name = gl->name; > - > gl->ready = 0; > - return NGX_OK; > + goto convert; > } > > ngx_free(gl->name.data); > gl->name.data = NULL; > > - if (FindNextFile(gl->dir, &gl->finddata) != 0) { > - > - len = ngx_strlen(gl->finddata.cFileName); > - gl->name.len = gl->last + len; > - > - gl->name.data = ngx_alloc(gl->name.len + 1, gl->log); > - if (gl->name.data == NULL) { > - return NGX_ERROR; > - } > - > - ngx_memcpy(gl->name.data, gl->pattern, gl->last); > - ngx_cpystrn(gl->name.data + gl->last, (u_char *) gl->finddata.cFileName, > - len + 1); > - > - *name = gl->name; > - > - return NGX_OK; > + if (FindNextFileW(gl->dir, &gl->finddata) != 0) { > + goto convert; > } > > err = ngx_errno; > @@ -637,6 +629,43 @@ ngx_read_glob(ngx_glob_t *gl, ngx_str_t > "FindNextFile(%s) failed", gl->pattern); > > return NGX_ERROR; > + > +convert: > + > + len = NGX_UTF16_BUFLEN; > + p = ngx_utf16_to_utf8(utf8, gl->finddata.cFileName, &len, NULL); Note that passing utf8 off the stack can result in an attempt to free the stack pointer inside ngx_utf16_to_utf8() on NGX_EILSEQ. > + Nitpicking: there (and in subsequent patches) you're starting to add an empty line after ngx_utf16_to_utf8(); it's not in the 1st path. > + if (p == NULL) { > + return NGX_ERROR; > + } > + > + gl->name.len = gl->last + len - 1; > + > + gl->name.data = ngx_alloc(gl->name.len + 1, gl->log); > + if (gl->name.data == NULL) { > + goto failed; > + } > + > + ngx_memcpy(gl->name.data, gl->pattern, gl->last); > + ngx_cpystrn(gl->name.data + gl->last, p, len); > + > + if (p != utf8) { > + ngx_free(p); > + } > + > + *name = gl->name; > + > + return NGX_OK; > + > +failed: > + > + if (p != utf8) { > + err = ngx_errno; > + ngx_free(p); > + ngx_set_errno(err); > + } > + > + return NGX_ERROR; > } > > > diff -r 60d845f9505f -r d05c0adf5890 src/os/win32/ngx_files.h > --- a/src/os/win32/ngx_files.h Thu Jan 12 21:41:30 2023 +0300 > +++ b/src/os/win32/ngx_files.h Thu Jan 12 21:41:39 2023 +0300 > @@ -44,7 +44,7 @@ typedef struct { > > typedef struct { > HANDLE dir; > - WIN32_FIND_DATA finddata; > + WIN32_FIND_DATAW finddata; > > unsigned ready:1; > unsigned test:1; -- Sergey Kandaurov From pluknet at nginx.com Fri Feb 17 15:04:04 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Fri, 17 Feb 2023 19:04:04 +0400 Subject: [PATCH 03 of 12] Win32: non-ASCII directory names support in ngx_getcwd() In-Reply-To: <7cf820c46860796cff91.1673559326@1.0.0.127.in-addr.arpa> References: <7cf820c46860796cff91.1673559326@1.0.0.127.in-addr.arpa> Message-ID: > On 13 Jan 2023, at 01:35, Maxim Dounin wrote: > > # HG changeset patch > # User Maxim Dounin > # Date 1673548916 -10800 > # Thu Jan 12 21:41:56 2023 +0300 > # Node ID 7cf820c46860796cff91f53a5d2db669bb5b5a6c > # Parent d05c0adf5890aecc68ce8906ef19ca07502ed071 > Win32: non-ASCII directory names support in ngx_getcwd(). > > This makes it possible to start nginx without a prefix explicitly set > in a directory with non-ASCII characters in it. > > diff -r d05c0adf5890 -r 7cf820c46860 src/os/win32/ngx_files.c > --- a/src/os/win32/ngx_files.c Thu Jan 12 21:41:39 2023 +0300 > +++ b/src/os/win32/ngx_files.c Thu Jan 12 21:41:56 2023 +0300 > @@ -428,6 +428,30 @@ ngx_realpath(u_char *path, u_char *resol > } > > > +size_t > +ngx_getcwd(u_char *buf, size_t size) > +{ > + u_char *p; > + size_t n; > + u_short utf16[NGX_MAX_PATH]; > + > + n = GetCurrentDirectoryW(NGX_MAX_PATH, utf16); > + > + if (n == 0) { > + return 0; > + } > + > + p = ngx_utf16_to_utf8(buf, utf16, &size, NULL); > + No error check may result in double-free, first freed after (re-)allocation on NGX_EILSEQ, then as below. > + if (p != buf) { > + ngx_free(p); > + return 0; Why return an error if (re-)allocation happened? Sizes (calculated in 1-byte units) above NGX_MAX_PATH seem perfectly valid with multibyte UTF-8. > + } > + > + return size - 1; > +} > + > + > ngx_int_t > ngx_open_dir(ngx_str_t *name, ngx_dir_t *dir) > { > diff -r d05c0adf5890 -r 7cf820c46860 src/os/win32/ngx_files.h > --- a/src/os/win32/ngx_files.h Thu Jan 12 21:41:39 2023 +0300 > +++ b/src/os/win32/ngx_files.h Thu Jan 12 21:41:56 2023 +0300 > @@ -178,8 +178,12 @@ void ngx_close_file_mapping(ngx_file_map > > u_char *ngx_realpath(u_char *path, u_char *resolved); > #define ngx_realpath_n "" > -#define ngx_getcwd(buf, size) GetCurrentDirectory(size, (char *) buf) > + > + > +size_t ngx_getcwd(u_char *buf, size_t size); > #define ngx_getcwd_n "GetCurrentDirectory()" > + > + > #define ngx_path_separator(c) ((c) == '/' || (c) == '\\') > > #define NGX_HAVE_MAX_PATH 1 -- Sergey Kandaurov From pluknet at nginx.com Fri Feb 17 15:12:24 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Fri, 17 Feb 2023 19:12:24 +0400 Subject: [PATCH 04 of 12] Win32: non-ASCII directory names support in ngx_create_dir() In-Reply-To: <331e40f73047c03455e1.1673559327@1.0.0.127.in-addr.arpa> References: <331e40f73047c03455e1.1673559327@1.0.0.127.in-addr.arpa> Message-ID: <418B152C-9163-4BE9-95FF-8051CBF4AA72@nginx.com> > On 13 Jan 2023, at 01:35, Maxim Dounin wrote: > > # HG changeset patch > # User Maxim Dounin > # Date 1673548958 -10800 > # Thu Jan 12 21:42:38 2023 +0300 > # Node ID 331e40f73047c03455e10212b2b8701ec5ec35d0 > # Parent 7cf820c46860796cff91f53a5d2db669bb5b5a6c > Win32: non-ASCII directory names support in ngx_create_dir(). > > This makes it possible to create directories under prefix with non-ASCII > characters, as well as makes it possible to create directories with non-ASCII > characters when using the dav module (ticket #1433). > > To ensure that the dav module operations are restricted similarly to > other file operations (in particular, short names are not allowed), the > ngx_win32_check_filename() function is used. It improved to support "it is/was improved" ? > checking of just dirname, and now can be used to check paths when creating > files or directories. > > diff -r 7cf820c46860 -r 331e40f73047 src/os/win32/ngx_files.c > --- a/src/os/win32/ngx_files.c Thu Jan 12 21:41:56 2023 +0300 > +++ b/src/os/win32/ngx_files.c Thu Jan 12 21:42:38 2023 +0300 > @@ -11,8 +11,8 @@ > > #define NGX_UTF16_BUFLEN 256 > > -static ngx_int_t ngx_win32_check_filename(u_char *name, u_short *u, > - size_t len); > +static ngx_int_t ngx_win32_check_filename(u_short *u, size_t len, > + ngx_uint_t dirname); > static u_short *ngx_utf8_to_utf16(u_short *utf16, u_char *utf8, size_t *len, > size_t reserved); > static u_char *ngx_utf16_to_utf8(u_char *utf8, u_short *utf16, size_t *len, > @@ -41,7 +41,7 @@ ngx_open_file(u_char *name, u_long mode, > fd = INVALID_HANDLE_VALUE; > > if (create == NGX_FILE_OPEN > - && ngx_win32_check_filename(name, u, len) != NGX_OK) > + && ngx_win32_check_filename(u, len, 0) != NGX_OK) > { > goto failed; > } > @@ -281,7 +281,7 @@ ngx_file_info(u_char *file, ngx_file_inf > > rc = NGX_FILE_ERROR; > > - if (ngx_win32_check_filename(file, u, len) != NGX_OK) { > + if (ngx_win32_check_filename(u, len, 0) != NGX_OK) { > goto failed; > } > > @@ -467,7 +467,7 @@ ngx_open_dir(ngx_str_t *name, ngx_dir_t > return NGX_ERROR; > } > > - if (ngx_win32_check_filename(name->data, u, len) != NGX_OK) { > + if (ngx_win32_check_filename(u, len, 0) != NGX_OK) { > goto failed; > } > > @@ -566,6 +566,42 @@ ngx_close_dir(ngx_dir_t *dir) > > > ngx_int_t > +ngx_create_dir(u_char *name, ngx_uint_t access) > +{ > + long rc; > + size_t len; > + u_short *u; > + ngx_err_t err; > + u_short utf16[NGX_UTF16_BUFLEN]; > + > + len = NGX_UTF16_BUFLEN; > + u = ngx_utf8_to_utf16(utf16, name, &len, 0); > + > + if (u == NULL) { > + return NGX_FILE_ERROR; > + } > + > + rc = NGX_FILE_ERROR; > + > + if (ngx_win32_check_filename(u, len, 1) != NGX_OK) { > + goto failed; > + } > + > + rc = CreateDirectoryW(u, NULL); > + > +failed: > + > + if (u != utf16) { > + err = ngx_errno; > + ngx_free(u); > + ngx_set_errno(err); > + } > + > + return rc; > +} > + > + > +ngx_int_t > ngx_open_glob(ngx_glob_t *gl) > { > u_char *p; > @@ -779,11 +815,10 @@ ngx_fs_available(u_char *name) > > > static ngx_int_t > -ngx_win32_check_filename(u_char *name, u_short *u, size_t len) > +ngx_win32_check_filename(u_short *u, size_t len, ngx_uint_t dirname) > { > - u_char *p, ch; > u_long n; > - u_short *lu; > + u_short *lu, *p, *slash, ch; > ngx_err_t err; > enum { > sw_start = 0, > @@ -796,9 +831,14 @@ ngx_win32_check_filename(u_char *name, u > /* check for NTFS streams (":"), trailing dots and spaces */ > > lu = NULL; > + slash = NULL; > state = sw_start; > > - for (p = name; *p; p++) { > +#if (NGX_SUPPRESS_WARN) > + ch = 0; > +#endif > + > + for (p = u; *p; p++) { > ch = *p; > > switch (state) { > @@ -812,6 +852,7 @@ ngx_win32_check_filename(u_char *name, u > > if (ch == '/' || ch == '\\') { > state = sw_after_slash; > + slash = p; > } > > break; > @@ -830,6 +871,7 @@ ngx_win32_check_filename(u_char *name, u > > if (ch == '/' || ch == '\\') { > state = sw_after_slash; > + slash = p; > break; > } > > @@ -857,6 +899,7 @@ ngx_win32_check_filename(u_char *name, u > > if (ch == '/' || ch == '\\') { > state = sw_after_slash; > + slash = p; > break; > } > > @@ -885,6 +928,12 @@ ngx_win32_check_filename(u_char *name, u > goto invalid; > } > > + if (dirname && slash) { > + ch = *slash; > + *slash = '\0'; > + len = slash - u + 1; > + } > + > /* check if long name match */ > > lu = malloc(len * 2); two minor points, unnoticeable in practice: - *slash is not restored on lu allocation error - it can be "restored" on parsing errors from a wrong ch, because it is still not remembered as shown above. > @@ -895,6 +944,11 @@ ngx_win32_check_filename(u_char *name, u > n = GetLongPathNameW(u, lu, len); > > if (n == 0) { > + > + if (dirname && slash && ngx_errno == NGX_ENOENT) { > + ngx_set_errno(NGX_ENOPATH); > + } > + > goto failed; > } > > @@ -902,6 +956,10 @@ ngx_win32_check_filename(u_char *name, u > goto invalid; > } > > + if (dirname && slash) { > + *slash = ch; > + } > + > ngx_free(lu); > > return NGX_OK; > @@ -912,6 +970,10 @@ invalid: > > failed: > > + if (dirname && slash) { > + *slash = ch; > + } > + > if (lu) { > err = ngx_errno; > ngx_free(lu); > diff -r 7cf820c46860 -r 331e40f73047 src/os/win32/ngx_files.h > --- a/src/os/win32/ngx_files.h Thu Jan 12 21:41:56 2023 +0300 > +++ b/src/os/win32/ngx_files.h Thu Jan 12 21:42:38 2023 +0300 > @@ -202,7 +202,7 @@ ngx_int_t ngx_close_dir(ngx_dir_t *dir); > #define ngx_close_dir_n "FindClose()" > > > -#define ngx_create_dir(name, access) CreateDirectory((const char *) name, NULL) > +ngx_int_t ngx_create_dir(u_char *name, ngx_uint_t access); > #define ngx_create_dir_n "CreateDirectory()" > > -- Sergey Kandaurov From pluknet at nginx.com Fri Feb 17 15:13:45 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Fri, 17 Feb 2023 19:13:45 +0400 Subject: [PATCH 05 of 12] Win32: non-ASCII directory names support in ngx_delete_dir() In-Reply-To: References: Message-ID: <0C7665B1-A687-45C3-B1EE-51D60252CFE7@nginx.com> > On 13 Jan 2023, at 01:35, Maxim Dounin wrote: > > # HG changeset patch > # User Maxim Dounin > # Date 1673548963 -10800 > # Thu Jan 12 21:42:43 2023 +0300 > # Node ID fbe7d76fe0398ca674b5d62b4849cd524e35bf89 > # Parent 331e40f73047c03455e10212b2b8701ec5ec35d0 > Win32: non-ASCII directory names support in ngx_delete_dir(). > > This makes it possible to delete directories with non-ASCII characters > when using the dav module (ticket #1433). > > [..] This and subsequent patches have no distinct issues I'm aware about. They looks good for me. -- Sergey Kandaurov From pluknet at nginx.com Fri Feb 17 15:17:02 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Fri, 17 Feb 2023 19:17:02 +0400 Subject: [PATCH 11 of 12] Win32: fixed ngx_fs_bsize() for symlinks In-Reply-To: References: Message-ID: > On 13 Jan 2023, at 01:35, Maxim Dounin wrote: > > # HG changeset patch > # User Maxim Dounin > # Date 1673549010 -10800 > # Thu Jan 12 21:43:30 2023 +0300 > # Node ID be7eb9ec28dcbfdfd2e850befc8d051c0e4d46fd > # Parent e62c8e9724ba68a698a2c3613edca73fe4e1c4ae > Win32: fixed ngx_fs_bsize() for symlinks. > > Just a drive letter might not correctly represent file system being used, > notably when using symlinks (as created by "mklink /d"). As such, instead > of calling GetDiskFreeSpace() with just a drive letter, we now always > use GetDiskFreeSpace() with full path. > > diff -r e62c8e9724ba -r be7eb9ec28dc src/os/win32/ngx_files.c > --- a/src/os/win32/ngx_files.c Thu Jan 12 21:43:14 2023 +0300 > +++ b/src/os/win32/ngx_files.c Thu Jan 12 21:43:30 2023 +0300 > @@ -955,14 +955,8 @@ ngx_directio_off(ngx_fd_t fd) > size_t > ngx_fs_bsize(u_char *name) > { > - u_char root[4]; > u_long sc, bs, nfree, ncl; > > - if (name[2] == ':') { > - ngx_cpystrn(root, name, 4); > - name = root; > - } > - BTW, I wonder how this condition could be true. Specifically, what name should represent in order to match. I'm happy that it's leaving though. > if (GetDiskFreeSpace((const char *) name, &sc, &bs, &nfree, &ncl) == 0) { > return 512; > } -- Sergey Kandaurov From mdounin at mdounin.ru Sun Feb 19 17:17:04 2023 From: mdounin at mdounin.ru (Maxim Dounin) Date: Sun, 19 Feb 2023 20:17:04 +0300 Subject: [PATCH 01 of 12] Win32: non-ASCII names support in autoindex (ticket #458) In-Reply-To: References: <60d845f9505fe1b97c1e.1673559324@1.0.0.127.in-addr.arpa> Message-ID: Hello! On Fri, Feb 17, 2023 at 06:38:54PM +0400, Sergey Kandaurov wrote: > > On 13 Jan 2023, at 01:35, Maxim Dounin wrote: > > > > # HG changeset patch > > # User Maxim Dounin > > # Date 1673548890 -10800 > > # Thu Jan 12 21:41:30 2023 +0300 > > # Node ID 60d845f9505fe1b97c1e04b680523b790e29fdb1 > > # Parent 07b0bee87f32be91a33210bc06973e07c4c1dac9 > > Win32: non-ASCII names support in autoindex (ticket #458). > > > > Notably, ngx_open_dir() now supports opening directories with non-ASCII > > characters, and directory entries returned by ngx_read_dir() are properly > > converted to UTF-8. > > > > diff -r 07b0bee87f32 -r 60d845f9505f src/os/win32/ngx_files.c > > --- a/src/os/win32/ngx_files.c Wed Dec 21 14:53:27 2022 +0300 > > +++ b/src/os/win32/ngx_files.c Thu Jan 12 21:41:30 2023 +0300 > > @@ -13,7 +13,11 @@ > > > > static ngx_int_t ngx_win32_check_filename(u_char *name, u_short *u, > > size_t len); > > -static u_short *ngx_utf8_to_utf16(u_short *utf16, u_char *utf8, size_t *len); > > +static u_short *ngx_utf8_to_utf16(u_short *utf16, u_char *utf8, size_t *len, > > + size_t reserved); > > +static u_char *ngx_utf16_to_utf8(u_char *utf8, u_short *utf16, size_t *len, > > + size_t *allocated); > > +uint32_t ngx_utf16_decode(u_short **u, size_t n); > > > > > > /* FILE_FLAG_BACKUP_SEMANTICS allows to obtain a handle to a directory */ > > @@ -28,7 +32,7 @@ ngx_open_file(u_char *name, u_long mode, > > u_short utf16[NGX_UTF16_BUFLEN]; > > > > len = NGX_UTF16_BUFLEN; > > - u = ngx_utf8_to_utf16(utf16, name, &len); > > + u = ngx_utf8_to_utf16(utf16, name, &len, 0); > > > > if (u == NULL) { > > return INVALID_HANDLE_VALUE; > > @@ -269,7 +273,7 @@ ngx_file_info(u_char *file, ngx_file_inf > > > > len = NGX_UTF16_BUFLEN; > > > > - u = ngx_utf8_to_utf16(utf16, file, &len); > > + u = ngx_utf8_to_utf16(utf16, file, &len, 0); > > > > if (u == NULL) { > > return NGX_FILE_ERROR; > > @@ -427,49 +431,51 @@ ngx_realpath(u_char *path, u_char *resol > > ngx_int_t > > ngx_open_dir(ngx_str_t *name, ngx_dir_t *dir) > > { > > - u_char *pattern, *p; > > + size_t len; > > + u_short *u, *p; > > ngx_err_t err; > > + u_short utf16[NGX_UTF16_BUFLEN]; > > > > - pattern = malloc(name->len + 3); > > - if (pattern == NULL) { > > + len = NGX_UTF16_BUFLEN - 2; > > + u = ngx_utf8_to_utf16(utf16, name->data, &len, 2); > > + > > + if (u == NULL) { > > return NGX_ERROR; > > } > > > > - p = ngx_cpymem(pattern, name->data, name->len); > > + if (ngx_win32_check_filename(name->data, u, len) != NGX_OK) { > > + goto failed; > > + } > > + > > + p = &u[len - 1]; > > > > *p++ = '/'; > > *p++ = '*'; > > *p = '\0'; > > > > - dir->dir = FindFirstFile((const char *) pattern, &dir->finddata); > > + dir->dir = FindFirstFileW(u, &dir->finddata); > > > > if (dir->dir == INVALID_HANDLE_VALUE) { > > - err = ngx_errno; > > - ngx_free(pattern); > > - ngx_set_errno(err); > > - return NGX_ERROR; > > + goto failed; > > } > > > > - ngx_free(pattern); > > + if (u != utf16) { > > + ngx_free(u); > > + } > > > > dir->valid_info = 1; > > dir->ready = 1; > > + dir->name = NULL; > > + dir->allocated = 0; > > > > return NGX_OK; > > -} > > > > +failed: > > > > -ngx_int_t > > -ngx_read_dir(ngx_dir_t *dir) > > -{ > > - if (dir->ready) { > > - dir->ready = 0; > > - return NGX_OK; > > - } > > - > > - if (FindNextFile(dir->dir, &dir->finddata) != 0) { > > - dir->type = 1; > > - return NGX_OK; > > + if (u != utf16) { > > + err = ngx_errno; > > + ngx_free(u); > > + ngx_set_errno(err); > > } > > > > return NGX_ERROR; > > @@ -477,8 +483,56 @@ ngx_read_dir(ngx_dir_t *dir) > > > > > > ngx_int_t > > +ngx_read_dir(ngx_dir_t *dir) > > +{ > > + u_char *name; > > + size_t len, allocated; > > + > > + if (dir->ready) { > > + dir->ready = 0; > > + goto convert; > > + } > > + > > + if (FindNextFileW(dir->dir, &dir->finddata) != 0) { > > + dir->type = 1; > > + goto convert; > > + } > > + > > + return NGX_ERROR; > > + > > +convert: > > + > > + name = dir->name; > > + len = dir->allocated; > > + > > + name = ngx_utf16_to_utf8(name, dir->finddata.cFileName, &len, &allocated); > > + if (name == NULL) { > > Note that you just return on NULL, but see below. > > > + return NGX_ERROR; > > + } > > + > > + if (name != dir->name) { > > + > > + if (dir->name) { > > + ngx_free(dir->name); > > + } > > + > > + dir->name = name; > > + dir->allocated = allocated; > > + } > > + > > + dir->namelen = len - 1; > > + > > + return NGX_OK; > > +} > > + > > + > > +ngx_int_t > > ngx_close_dir(ngx_dir_t *dir) > > { > > + if (dir->name) { > > + ngx_free(dir->name); > > + } > > + > > if (FindClose(dir->dir) == 0) { > > return NGX_ERROR; > > } > > @@ -816,7 +870,7 @@ failed: > > > > > > static u_short * > > -ngx_utf8_to_utf16(u_short *utf16, u_char *utf8, size_t *len) > > +ngx_utf8_to_utf16(u_short *utf16, u_char *utf8, size_t *len, size_t reserved) > > { > > u_char *p; > > u_short *u, *last; > > @@ -865,7 +919,7 @@ ngx_utf8_to_utf16(u_short *utf16, u_char > > > > /* the given buffer is not enough, allocate a new one */ > > > > - u = malloc(((p - utf8) + ngx_strlen(p) + 1) * sizeof(u_short)); > > + u = malloc(((p - utf8) + ngx_strlen(p) + 1 + reserved) * sizeof(u_short)); > > if (u == NULL) { > > return NULL; > > } > > @@ -910,3 +964,170 @@ ngx_utf8_to_utf16(u_short *utf16, u_char > > > > /* unreachable */ > > } > > + > > + > > +static u_char * > > +ngx_utf16_to_utf8(u_char *utf8, u_short *utf16, size_t *len, size_t *allocated) > > +{ > > + u_char *p, *last; > > + u_short *u, *j; > > + uint32_t n; > > + > > + u = utf16; > > + p = utf8; > > + last = utf8 + *len; > > + > > + while (p < last) { > > + > > + if (*u < 0x80) { > > + *p++ = (u_char) *u; > > + > > + if (*u == 0) { > > + *len = p - utf8; > > + return utf8; > > + } > > + > > + u++; > > + > > + continue; > > + } > > + > > + if (p >= last - 4) { > > + *len = p - utf8; > > + break; > > + } > > + > > + n = ngx_utf16_decode(&u, 2); > > + > > + if (n > 0x10ffff) { > > + ngx_free(utf8); > > + ngx_set_errno(NGX_EILSEQ); > > This can result in double-free on a subsequent ngx_close_dir() call. Thanks for catching, that's slipped in from the second loop. Removed ngx_free() here. > > > + return NULL; > > + } > > + > > + if (n >= 0x10000) { > > + *p++ = (u_char) (0xf0 + (n >> 18)); > > + *p++ = (u_char) (0x80 + ((n >> 12) & 0x3f)); > > + *p++ = (u_char) (0x80 + ((n >> 6) & 0x3f)); > > + *p++ = (u_char) (0x80 + (n & 0x3f)); > > + continue; > > + > > stray ws-only line (one more below) Fixed both, thanks. > > + } > > + > > + if (n >= 0x0800) { > > + *p++ = (u_char) (0xe0 + (n >> 12)); > > + *p++ = (u_char) (0x80 + ((n >> 6) & 0x3f)); > > + *p++ = (u_char) (0x80 + (n & 0x3f)); > > + continue; > > + } > > + > > + *p++ = (u_char) (0xc0 + (n >> 6)); > > + *p++ = (u_char) (0x80 + (n & 0x3f)); > > + } > > + > > + /* the given buffer is not enough, allocate a new one */ > > + > > + for (j = u; *j; j++) { /* void */ } > > + > > + p = malloc((j - utf16) * 4 + 1); > > [Note that pointer arithmetics is in u_short (2 bytes) units.] > It seems you are allocating memory assuming that utf16 -> utf8 > conversion can result in memory expansion up to 4 bytes per unit. > At first glance that's true, because UTF-8 symbol may take 4 bytes. > IIUC, 4-byte UTF-8 can result only from UTF-16 surrogate pairs, > this gives 2x growth (2-paired UTF-16 to 4-bytes UTF-8). > Otherwise, 1/2/3-byte encoded UTF-8 can result from 2-byte UTF-16, > which gives from 1x to 3x growth per unit. > So, "malloc((j - utf16) * 3 + 1)" should be enough there. That's correct, though this depends on not-directly-visible encoding details. I would rather prefer to keep 4 here, since it can be easily verified to be enough from the loop itself. I don't think that a minor optimization of memory usage by going from 4x to 3x of the particular name is important here. Further, extra allocated bytes are likely to be used as a preallocation for other names. > > + if (p == NULL) { > > + return NULL; > > + } > > Old memory is still there, allocated and referenced by dir->name, > see above. It seems ok, it should be freed by ngx_close_dir() call. > If we happen to change callers to not call ngx_close_dir() on error, > this could turn into memory leaks, though. The ngx_close_dir() is certainly required (including for other reasons), so this looks perfectly fine. > > + > > + if (allocated) { > > + *allocated = (j - utf16) * 4 + 1; > > + } > > + > > + ngx_memcpy(p, utf8, *len); > > + > > + utf8 = p; > > + p += *len; > > + > > + for ( ;; ) { > > + > > + if (*u < 0x80) { > > + *p++ = (u_char) *u; > > + > > + if (*u == 0) { > > + *len = p - utf8; > > + return utf8; > > + } > > + > > + u++; > > + > > + continue; > > + } > > + > > + n = ngx_utf16_decode(&u, 2); > > + > > + if (n > 0x10ffff) { > > + ngx_free(utf8); > > + ngx_set_errno(NGX_EILSEQ); > > + return NULL; > > + } > > + > > + if (n >= 0x10000) { > > + *p++ = (u_char) (0xf0 + (n >> 18)); > > + *p++ = (u_char) (0x80 + ((n >> 12) & 0x3f)); > > + *p++ = (u_char) (0x80 + ((n >> 6) & 0x3f)); > > + *p++ = (u_char) (0x80 + (n & 0x3f)); > > + continue; > > + > > + } > > + > > + if (n >= 0x0800) { > > + *p++ = (u_char) (0xe0 + (n >> 12)); > > + *p++ = (u_char) (0x80 + ((n >> 6) & 0x3f)); > > + *p++ = (u_char) (0x80 + (n & 0x3f)); > > + continue; > > + } > > + > > + *p++ = (u_char) (0xc0 + (n >> 6)); > > + *p++ = (u_char) (0x80 + (n & 0x3f)); > > + } > > + > > + /* unreachable */ > > +} > > + > > + > > +/* > > + * ngx_utf16_decode() decodes one or two UTF-16 code units > > + * the return values: > > + * 0x80 - 0x10ffff valid character > > + * 0x110000 - 0xfffffffd invalid sequence > > + * 0xfffffffe incomplete sequence > > + * 0xffffffff error > > + */ > > + > > +uint32_t > > +ngx_utf16_decode(u_short **u, size_t n) > > +{ > > + uint32_t k, m; > > + > > + k = **u; > > + > > + if (k < 0xd800 || k > 0xdfff) { > > + (*u)++; > > + return k; > > + } > > + > > + if (k > 0xdbff) { > > + (*u)++; > > + return 0xffffffff; > > + } > > + > > + if (n < 2) { > > + return 0xfffffffe; > > + } > > + > > + (*u)++; > > + > > + m = *(*u)++; > > + > > + if (m < 0xdc00 || m > 0xdfff) { > > + return 0xffffffff; > > + > > + } > > + > > + return 0x10000 + ((k - 0xd800) << 10) + (m - 0xdc00); > > +} > > diff -r 07b0bee87f32 -r 60d845f9505f src/os/win32/ngx_files.h > > --- a/src/os/win32/ngx_files.h Wed Dec 21 14:53:27 2022 +0300 > > +++ b/src/os/win32/ngx_files.h Thu Jan 12 21:41:30 2023 +0300 > > @@ -30,7 +30,11 @@ typedef struct { > > > > typedef struct { > > HANDLE dir; > > - WIN32_FIND_DATA finddata; > > + WIN32_FIND_DATAW finddata; > > + > > + u_char *name; > > + size_t namelen; > > + size_t allocated; > > > > unsigned valid_info:1; > > unsigned type:1; > > @@ -205,8 +209,8 @@ ngx_int_t ngx_close_dir(ngx_dir_t *dir); > > #define ngx_dir_access(a) (a) > > > > > > -#define ngx_de_name(dir) ((u_char *) (dir)->finddata.cFileName) > > -#define ngx_de_namelen(dir) ngx_strlen((dir)->finddata.cFileName) > > +#define ngx_de_name(dir) (dir)->name > > +#define ngx_de_namelen(dir) (dir)->namelen > > > > ngx_int_t ngx_de_info(u_char *name, ngx_dir_t *dir); > > #define ngx_de_info_n "dummy()" > > Otherwise, looks good. Fixes: diff -r d7521ae9e138 src/os/win32/ngx_files.c --- a/src/os/win32/ngx_files.c Sat Feb 18 12:08:25 2023 +0300 +++ b/src/os/win32/ngx_files.c Sat Feb 18 12:57:02 2023 +0300 @@ -506,6 +506,7 @@ convert: len = dir->allocated; name = ngx_utf16_to_utf8(name, dir->finddata.cFileName, &len, &allocated); + if (name == NULL) { return NGX_ERROR; } @@ -1000,7 +1000,6 @@ ngx_utf16_to_utf8(u_char *utf8, u_short n = ngx_utf16_decode(&u, 2); if (n > 0x10ffff) { - ngx_free(utf8); ngx_set_errno(NGX_EILSEQ); return NULL; } @@ -1011,7 +1010,6 @@ ngx_utf16_to_utf8(u_char *utf8, u_short *p++ = (u_char) (0x80 + ((n >> 6) & 0x3f)); *p++ = (u_char) (0x80 + (n & 0x3f)); continue; - } if (n >= 0x0800) { @@ -1072,7 +1070,6 @@ ngx_utf16_to_utf8(u_char *utf8, u_short *p++ = (u_char) (0x80 + ((n >> 6) & 0x3f)); *p++ = (u_char) (0x80 + (n & 0x3f)); continue; - } if (n >= 0x0800) { -- Maxim Dounin http://mdounin.ru/ From mdounin at mdounin.ru Sun Feb 19 17:18:58 2023 From: mdounin at mdounin.ru (Maxim Dounin) Date: Sun, 19 Feb 2023 20:18:58 +0300 Subject: [PATCH 02 of 12] Win32: non-ASCII names support in "include" with wildcards In-Reply-To: References: Message-ID: Hello! On Fri, Feb 17, 2023 at 06:53:11PM +0400, Sergey Kandaurov wrote: > > On 13 Jan 2023, at 01:35, Maxim Dounin wrote: > > > > # HG changeset patch > > # User Maxim Dounin > > # Date 1673548899 -10800 > > # Thu Jan 12 21:41:39 2023 +0300 > > # Node ID d05c0adf5890aecc68ce8906ef19ca07502ed071 > > # Parent 60d845f9505fe1b97c1e04b680523b790e29fdb1 > > Win32: non-ASCII names support in "include" with wildcards. > > > > Notably, ngx_open_glob() now supports opening directories with non-ASCII > > characters, and pathnames returned by ngx_read_glob() are converted to UTF-8. > > > > diff -r 60d845f9505f -r d05c0adf5890 src/os/win32/ngx_files.c > > --- a/src/os/win32/ngx_files.c Thu Jan 12 21:41:30 2023 +0300 > > +++ b/src/os/win32/ngx_files.c Thu Jan 12 21:41:39 2023 +0300 > > @@ -546,14 +546,27 @@ ngx_open_glob(ngx_glob_t *gl) > > { > > u_char *p; > > size_t len; > > + u_short *u; > > ngx_err_t err; > > + u_short utf16[NGX_UTF16_BUFLEN]; > > > > - gl->dir = FindFirstFile((const char *) gl->pattern, &gl->finddata); > > + len = NGX_UTF16_BUFLEN; > > + u = ngx_utf8_to_utf16(utf16, gl->pattern, &len, 0); > > + > > + if (u == NULL) { > > + return NGX_ERROR; > > + } > > + > > + gl->dir = FindFirstFileW(u, &gl->finddata); > > > > if (gl->dir == INVALID_HANDLE_VALUE) { > > > > err = ngx_errno; > > > > + if (u != utf16) { > > + ngx_free(u); > > + } > > + > > if ((err == ERROR_FILE_NOT_FOUND || err == ERROR_PATH_NOT_FOUND) > > && gl->test) > > { > > @@ -561,6 +574,8 @@ ngx_open_glob(ngx_glob_t *gl) > > return NGX_OK; > > } > > > > + ngx_set_errno(err); > > + > > return NGX_ERROR; > > } > > > > @@ -570,18 +585,10 @@ ngx_open_glob(ngx_glob_t *gl) > > } > > } > > > > - len = ngx_strlen(gl->finddata.cFileName); > > - gl->name.len = gl->last + len; > > - > > - gl->name.data = ngx_alloc(gl->name.len + 1, gl->log); > > - if (gl->name.data == NULL) { > > - return NGX_ERROR; > > + if (u != utf16) { > > + ngx_free(u); > > } > > > > - ngx_memcpy(gl->name.data, gl->pattern, gl->last); > > - ngx_cpystrn(gl->name.data + gl->last, (u_char *) gl->finddata.cFileName, > > - len + 1); > > - > > gl->ready = 1; > > > > return NGX_OK; > > @@ -591,40 +598,25 @@ ngx_open_glob(ngx_glob_t *gl) > > ngx_int_t > > ngx_read_glob(ngx_glob_t *gl, ngx_str_t *name) > > { > > - size_t len; > > - ngx_err_t err; > > + u_char *p; > > + size_t len; > > + ngx_err_t err; > > + u_char utf8[NGX_UTF16_BUFLEN]; > > Mixing UTF16-specific macro with u_char utf8[] looks suspicious. That's just some preallocated buffer size. If it feels better, a separate macro can be used instead. This will also make it possible to fine-tune the buffer: diff -r aca8812f2482 src/os/win32/ngx_files.c --- a/src/os/win32/ngx_files.c Sat Feb 18 12:58:54 2023 +0300 +++ b/src/os/win32/ngx_files.c Sat Feb 18 13:05:34 2023 +0300 @@ -10,6 +10,7 @@ #define NGX_UTF16_BUFLEN 256 +#define NGX_UTF8_BUFLEN 512 static ngx_int_t ngx_win32_check_filename(u_char *name, u_short *u, size_t len); @@ -601,7 +602,7 @@ ngx_read_glob(ngx_glob_t *gl, ngx_str_t u_char *p; size_t len; ngx_err_t err; - u_char utf8[NGX_UTF16_BUFLEN]; + u_char utf8[NGX_UTF8_BUFLEN]; if (gl->no_match) { return NGX_DONE; @@ -632,7 +633,7 @@ ngx_read_glob(ngx_glob_t *gl, ngx_str_t convert: - len = NGX_UTF16_BUFLEN; + len = NGX_UTF8_BUFLEN; p = ngx_utf16_to_utf8(utf8, gl->finddata.cFileName, &len, NULL); if (p == NULL) { > It may have sense to replace this with dynamic allocation > inside ngx_utf16_to_utf8(), similar to ngx_read_dir() 1st call, > or use NGX_MAX_PATH. Much like in other functions in this file, the idea is to avoid dynamic allocations in most cases by providing an easy to implement stack-based buffer. The approach used in ngx_read_dir() is more complex since the result of the conversion needs to be maintained between calls (to be available via the ngx_de_name() macro). For ngx_read_glob(), a better approach might be to combine conversion buffer and the final glob() result buffer (which also includes directory name, and currently allocated on each ngx_read_glob() call). Though I don't think it worth the effort, especially given that ngx_read_glob() is only used during configuration parsing. As for NGX_MAX_PATH, it does not look like a good replacement for a number of reasons. In particular, a) it might be incorrectly interpreted as a place when only paths up to MAX_PATH are supported, b) MAX_PATH on win32 is in characters, while the buffer in question is in bytes (and to be used for UTF-8), so it is no better than NGX_UTF16_BUFLEN. > > > > if (gl->no_match) { > > return NGX_DONE; > > } > > > > if (gl->ready) { > > - *name = gl->name; > > - > > gl->ready = 0; > > - return NGX_OK; > > + goto convert; > > } > > > > ngx_free(gl->name.data); > > gl->name.data = NULL; > > > > - if (FindNextFile(gl->dir, &gl->finddata) != 0) { > > - > > - len = ngx_strlen(gl->finddata.cFileName); > > - gl->name.len = gl->last + len; > > - > > - gl->name.data = ngx_alloc(gl->name.len + 1, gl->log); > > - if (gl->name.data == NULL) { > > - return NGX_ERROR; > > - } > > - > > - ngx_memcpy(gl->name.data, gl->pattern, gl->last); > > - ngx_cpystrn(gl->name.data + gl->last, (u_char *) gl->finddata.cFileName, > > - len + 1); > > - > > - *name = gl->name; > > - > > - return NGX_OK; > > + if (FindNextFileW(gl->dir, &gl->finddata) != 0) { > > + goto convert; > > } > > > > err = ngx_errno; > > @@ -637,6 +629,43 @@ ngx_read_glob(ngx_glob_t *gl, ngx_str_t > > "FindNextFile(%s) failed", gl->pattern); > > > > return NGX_ERROR; > > + > > +convert: > > + > > + len = NGX_UTF16_BUFLEN; > > + p = ngx_utf16_to_utf8(utf8, gl->finddata.cFileName, &len, NULL); > > Note that passing utf8 off the stack can result in an attempt to free > the stack pointer inside ngx_utf16_to_utf8() on NGX_EILSEQ. Yes, thanks, this shouldn't be a problem with the ngx_utf16_to_utf8() already fixed based on the 1st patch comments. > > + > > Nitpicking: there (and in subsequent patches) you're starting to add > an empty line after ngx_utf16_to_utf8(); it's not in the 1st path. In the first patch there is an empty line before the ngx_utf16_to_utf8() call, which somewhat balance things. Either way, added an empty line there as well. [...] -- Maxim Dounin http://mdounin.ru/ From mdounin at mdounin.ru Sun Feb 19 17:23:23 2023 From: mdounin at mdounin.ru (Maxim Dounin) Date: Sun, 19 Feb 2023 20:23:23 +0300 Subject: [PATCH 03 of 12] Win32: non-ASCII directory names support in ngx_getcwd() In-Reply-To: References: <7cf820c46860796cff91.1673559326@1.0.0.127.in-addr.arpa> Message-ID: Hello! On Fri, Feb 17, 2023 at 07:04:04PM +0400, Sergey Kandaurov wrote: > > On 13 Jan 2023, at 01:35, Maxim Dounin wrote: > > > > # HG changeset patch > > # User Maxim Dounin > > # Date 1673548916 -10800 > > # Thu Jan 12 21:41:56 2023 +0300 > > # Node ID 7cf820c46860796cff91f53a5d2db669bb5b5a6c > > # Parent d05c0adf5890aecc68ce8906ef19ca07502ed071 > > Win32: non-ASCII directory names support in ngx_getcwd(). > > > > This makes it possible to start nginx without a prefix explicitly set > > in a directory with non-ASCII characters in it. > > > > diff -r d05c0adf5890 -r 7cf820c46860 src/os/win32/ngx_files.c > > --- a/src/os/win32/ngx_files.c Thu Jan 12 21:41:39 2023 +0300 > > +++ b/src/os/win32/ngx_files.c Thu Jan 12 21:41:56 2023 +0300 > > @@ -428,6 +428,30 @@ ngx_realpath(u_char *path, u_char *resol > > } > > > > > > +size_t > > +ngx_getcwd(u_char *buf, size_t size) > > +{ > > + u_char *p; > > + size_t n; > > + u_short utf16[NGX_MAX_PATH]; > > + > > + n = GetCurrentDirectoryW(NGX_MAX_PATH, utf16); > > + > > + if (n == 0) { > > + return 0; > > + } Re-reading the documentation, I tend to think this might worth an additional test for "(n > NGX_MAX_PATH)" (https://learn.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-getcurrentdirectory): : If the function succeeds, the return value specifies the number : of characters that are written to the buffer, not including the : terminating null character. : If the function fails, the return value is zero. To get extended : error information, call GetLastError. : If the buffer that is pointed to by lpBuffer is not large : enough, the return value specifies the required size of the : buffer, in characters, including the null-terminating character. That is, there will be no explicit error (n == 0) if the buffer will happen to be insufficient. On the other hand, this probably can happen with long paths enabled (see https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=registry), so it might be a good idea to add an explicit handling while here. > > + > > + p = ngx_utf16_to_utf8(buf, utf16, &size, NULL); > > + > > No error check may result in double-free, first freed > after (re-)allocation on NGX_EILSEQ, then as below. Free on NGX_EILSEQ after reallocation only frees the memory allocated by the ngx_utf16_to_utf8() itself. The incorrect free() was before reallocation, and is now fixed. Since on errors ngx_utf16_to_utf8() returns NULL, it is matched by the "if (p != buf)" test, and processed identically to reallocations. The only questionable part is ngx_free(NULL), though as per C standard (starting at least with C89; and also win32 documentation) this is guaranteed to be a nop. > > + if (p != buf) { > > + ngx_free(p); > > + return 0; > > Why return an error if (re-)allocation happened? > Sizes (calculated in 1-byte units) above NGX_MAX_PATH > seem perfectly valid with multibyte UTF-8. The ngx_getcwd() interface does not provide a way to return a new pointer. Rather, it is strictly limited to the buffer provided by the caller, and can only return an error if the buffer is not large enough. One possible improvement here is to explicitly set the errno to a reasonable value if this happens. This would require an explicit handling of other errors though. Combined with the above: diff -r 6705dd675ace src/os/win32/ngx_files.c --- a/src/os/win32/ngx_files.c Sat Feb 18 16:01:20 2023 +0300 +++ b/src/os/win32/ngx_files.c Sun Feb 19 03:22:48 2023 +0300 @@ -442,10 +442,20 @@ ngx_getcwd(u_char *buf, size_t size) return 0; } + if (n > NGX_MAX_PATH) { + ngx_set_errno(ERROR_INSUFFICIENT_BUFFER); + return 0; + } + p = ngx_utf16_to_utf8(buf, utf16, &size, NULL); + if (p == NULL) { + return 0; + } + if (p != buf) { ngx_free(p); + ngx_set_errno(ERROR_INSUFFICIENT_BUFFER); return 0; } [...] -- Maxim Dounin http://mdounin.ru/ From mdounin at mdounin.ru Sun Feb 19 17:23:39 2023 From: mdounin at mdounin.ru (Maxim Dounin) Date: Sun, 19 Feb 2023 20:23:39 +0300 Subject: [PATCH 11 of 12] Win32: fixed ngx_fs_bsize() for symlinks In-Reply-To: References: Message-ID: Hello! On Fri, Feb 17, 2023 at 07:17:02PM +0400, Sergey Kandaurov wrote: > > On 13 Jan 2023, at 01:35, Maxim Dounin wrote: > > > > # HG changeset patch > > # User Maxim Dounin > > # Date 1673549010 -10800 > > # Thu Jan 12 21:43:30 2023 +0300 > > # Node ID be7eb9ec28dcbfdfd2e850befc8d051c0e4d46fd > > # Parent e62c8e9724ba68a698a2c3613edca73fe4e1c4ae > > Win32: fixed ngx_fs_bsize() for symlinks. > > > > Just a drive letter might not correctly represent file system being used, > > notably when using symlinks (as created by "mklink /d"). As such, instead > > of calling GetDiskFreeSpace() with just a drive letter, we now always > > use GetDiskFreeSpace() with full path. > > > > diff -r e62c8e9724ba -r be7eb9ec28dc src/os/win32/ngx_files.c > > --- a/src/os/win32/ngx_files.c Thu Jan 12 21:43:14 2023 +0300 > > +++ b/src/os/win32/ngx_files.c Thu Jan 12 21:43:30 2023 +0300 > > @@ -955,14 +955,8 @@ ngx_directio_off(ngx_fd_t fd) > > size_t > > ngx_fs_bsize(u_char *name) > > { > > - u_char root[4]; > > u_long sc, bs, nfree, ncl; > > > > - if (name[2] == ':') { > > - ngx_cpystrn(root, name, 4); > > - name = root; > > - } > > - > > BTW, I wonder how this condition could be true. > Specifically, what name should represent in order to match. > I'm happy that it's leaving though. I tend to think that this actually never worked, and the original intention was to test name[1] instead. Updated commit log: : Win32: removed attempt to use a drive letter in ngx_fs_bsize(). : : Just a drive letter might not correctly represent file system being used, : notably when using symlinks (as created by "mklink /d"). As such, instead : of trying to call GetDiskFreeSpace() with just a drive letter, we now always : use GetDiskFreeSpace() with full path. : : Further, it looks like the code to use just a drive letter never worked, : since it tried to test name[2] instead of name[1] to be ':'. [...] -- Maxim Dounin http://mdounin.ru/ From pluknet at nginx.com Mon Feb 20 12:54:42 2023 From: pluknet at nginx.com (=?iso-8859-1?q?Sergey_Kandaurov?=) Date: Mon, 20 Feb 2023 16:54:42 +0400 Subject: [PATCH] QUIC: using ngx_ssl_handshake_log() Message-ID: # HG changeset patch # User Sergey Kandaurov # Date 1676897567 -14400 # Mon Feb 20 16:52:47 2023 +0400 # Branch quic # Node ID ed403c1fe1f060ee362155bb00c43aaab356e30f # Parent 12b756caaf167d2239fd3bd7a75b270ca89ca26b QUIC: using ngx_ssl_handshake_log(). diff --git a/src/event/ngx_event_openssl.c b/src/event/ngx_event_openssl.c --- a/src/event/ngx_event_openssl.c +++ b/src/event/ngx_event_openssl.c @@ -33,9 +33,6 @@ static int ngx_ssl_new_client_session(ng #ifdef SSL_READ_EARLY_DATA_SUCCESS static ngx_int_t ngx_ssl_try_early_data(ngx_connection_t *c); #endif -#if (NGX_DEBUG) -static void ngx_ssl_handshake_log(ngx_connection_t *c); -#endif static void ngx_ssl_handshake_handler(ngx_event_t *ev); #ifdef SSL_READ_EARLY_DATA_SUCCESS static ssize_t ngx_ssl_recv_early(ngx_connection_t *c, u_char *buf, @@ -2052,7 +2049,7 @@ ngx_ssl_try_early_data(ngx_connection_t #if (NGX_DEBUG) -static void +void ngx_ssl_handshake_log(ngx_connection_t *c) { char buf[129], *s, *d; diff --git a/src/event/ngx_event_openssl.h b/src/event/ngx_event_openssl.h --- a/src/event/ngx_event_openssl.h +++ b/src/event/ngx_event_openssl.h @@ -310,6 +310,9 @@ ngx_int_t ngx_ssl_get_client_v_remain(ng ngx_int_t ngx_ssl_handshake(ngx_connection_t *c); +#if (NGX_DEBUG) +void ngx_ssl_handshake_log(ngx_connection_t *c); +#endif ssize_t ngx_ssl_recv(ngx_connection_t *c, u_char *buf, size_t size); ssize_t ngx_ssl_write(ngx_connection_t *c, u_char *data, size_t size); ssize_t ngx_ssl_recv_chain(ngx_connection_t *c, ngx_chain_t *cl, off_t limit); diff --git a/src/event/quic/ngx_event_quic_ssl.c b/src/event/quic/ngx_event_quic_ssl.c --- a/src/event/quic/ngx_event_quic_ssl.c +++ b/src/event/quic/ngx_event_quic_ssl.c @@ -440,11 +440,9 @@ ngx_quic_crypto_input(ngx_connection_t * return NGX_OK; } - ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0, - "quic ssl cipher:%s", SSL_get_cipher(ssl_conn)); - - ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0, - "quic handshake completed successfully"); +#if (NGX_DEBUG) + ngx_ssl_handshake_log(c); +#endif c->ssl->handshaked = 1; From pluknet at nginx.com Mon Feb 20 15:37:04 2023 From: pluknet at nginx.com (=?iso-8859-1?q?Sergey_Kandaurov?=) Date: Mon, 20 Feb 2023 19:37:04 +0400 Subject: [PATCH] QUIC: improved ssl_reject_handshake error logging Message-ID: <60b9d9ef878cc87d8a18.1676907424@enoparse.local> # HG changeset patch # User Sergey Kandaurov # Date 1676907370 -14400 # Mon Feb 20 19:36:10 2023 +0400 # Branch quic # Node ID 60b9d9ef878cc87d8a18e968b4f9122534a7deb8 # Parent ed403c1fe1f060ee362155bb00c43aaab356e30f QUIC: improved ssl_reject_handshake error logging. The check follows the ngx_ssl_handshake() change in 59e1c73fe02b. diff --git a/src/event/quic/ngx_event_quic_ssl.c b/src/event/quic/ngx_event_quic_ssl.c --- a/src/event/quic/ngx_event_quic_ssl.c +++ b/src/event/quic/ngx_event_quic_ssl.c @@ -422,8 +422,17 @@ ngx_quic_crypto_input(ngx_connection_t * sslerr); if (sslerr != SSL_ERROR_WANT_READ) { + + qc->error_reason = "handshake failed"; + + if (c->ssl->handshake_rejected) { + ngx_connection_error(c, 0, "handshake rejected"); + ERR_clear_error(); + + return NGX_ERROR; + } + ngx_ssl_error(NGX_LOG_ERR, c->log, 0, "SSL_do_handshake() failed"); - qc->error_reason = "handshake failed"; return NGX_ERROR; } } From pluknet at nginx.com Tue Feb 21 10:46:05 2023 From: pluknet at nginx.com (=?iso-8859-1?q?Sergey_Kandaurov?=) Date: Tue, 21 Feb 2023 14:46:05 +0400 Subject: [PATCH] Documented the use of cyclic memory buffer log with lldb Message-ID: <8cb435f59e89b34371ba.1676976365@enoparse.local> # HG changeset patch # User Sergey Kandaurov # Date 1676976309 -14400 # Tue Feb 21 14:45:09 2023 +0400 # Node ID 8cb435f59e89b34371bab83f8032efa0bea37817 # Parent 159dc1fda0d7404aa012cad2ae37a770c4bb8164 Documented the use of cyclic memory buffer log with lldb. diff --git a/xml/en/docs/debugging_log.xml b/xml/en/docs/debugging_log.xml --- a/xml/en/docs/debugging_log.xml +++ b/xml/en/docs/debugging_log.xml @@ -8,7 +8,7 @@
+ rev="6">
@@ -122,6 +122,13 @@ end set $buf = (ngx_log_memory_buf_t *) $log->wdata dump binary memory debug_log.txt $buf->start $buf->end +Or using an lldb script as follows: + +expr ngx_log_t *$log = ngx_cycle->log +expr while ($log->writer != ngx_log_memory_writer) { $log = $log->next; } +expr ngx_log_memory_buf_t *$buf = (ngx_log_memory_buf_t *) $log->wdata +memory read --force --outfile debug_log.txt --binary $buf->start $buf->end +
diff --git a/xml/ru/docs/debugging_log.xml b/xml/ru/docs/debugging_log.xml --- a/xml/ru/docs/debugging_log.xml +++ b/xml/ru/docs/debugging_log.xml @@ -8,7 +8,7 @@
+ rev="6">
@@ -121,6 +121,13 @@ end set $buf = (ngx_log_memory_buf_t *) $log->wdata dump binary memory debug_log.txt $buf->start $buf->end +Или при помощи такого lldb-скрипта: + +expr ngx_log_t *$log = ngx_cycle->log +expr while ($log->writer != ngx_log_memory_writer) { $log = $log->next; } +expr ngx_log_memory_buf_t *$buf = (ngx_log_memory_buf_t *) $log->wdata +memory read --force --outfile debug_log.txt --binary $buf->start $buf->end +
From arut at nginx.com Tue Feb 21 11:40:06 2023 From: arut at nginx.com (Roman Arutyunyan) Date: Tue, 21 Feb 2023 15:40:06 +0400 Subject: [PATCH] QUIC: using ngx_ssl_handshake_log() In-Reply-To: References: Message-ID: <20230221114006.oysdbrnvtlywdt6w@N00W24XTQX> On Mon, Feb 20, 2023 at 04:54:42PM +0400, Sergey Kandaurov wrote: > # HG changeset patch > # User Sergey Kandaurov > # Date 1676897567 -14400 > # Mon Feb 20 16:52:47 2023 +0400 > # Branch quic > # Node ID ed403c1fe1f060ee362155bb00c43aaab356e30f > # Parent 12b756caaf167d2239fd3bd7a75b270ca89ca26b > QUIC: using ngx_ssl_handshake_log(). > > diff --git a/src/event/ngx_event_openssl.c b/src/event/ngx_event_openssl.c > --- a/src/event/ngx_event_openssl.c > +++ b/src/event/ngx_event_openssl.c > @@ -33,9 +33,6 @@ static int ngx_ssl_new_client_session(ng > #ifdef SSL_READ_EARLY_DATA_SUCCESS > static ngx_int_t ngx_ssl_try_early_data(ngx_connection_t *c); > #endif > -#if (NGX_DEBUG) > -static void ngx_ssl_handshake_log(ngx_connection_t *c); > -#endif > static void ngx_ssl_handshake_handler(ngx_event_t *ev); > #ifdef SSL_READ_EARLY_DATA_SUCCESS > static ssize_t ngx_ssl_recv_early(ngx_connection_t *c, u_char *buf, > @@ -2052,7 +2049,7 @@ ngx_ssl_try_early_data(ngx_connection_t > > #if (NGX_DEBUG) > > -static void > +void > ngx_ssl_handshake_log(ngx_connection_t *c) > { > char buf[129], *s, *d; > diff --git a/src/event/ngx_event_openssl.h b/src/event/ngx_event_openssl.h > --- a/src/event/ngx_event_openssl.h > +++ b/src/event/ngx_event_openssl.h > @@ -310,6 +310,9 @@ ngx_int_t ngx_ssl_get_client_v_remain(ng > > > ngx_int_t ngx_ssl_handshake(ngx_connection_t *c); > +#if (NGX_DEBUG) > +void ngx_ssl_handshake_log(ngx_connection_t *c); > +#endif > ssize_t ngx_ssl_recv(ngx_connection_t *c, u_char *buf, size_t size); > ssize_t ngx_ssl_write(ngx_connection_t *c, u_char *data, size_t size); > ssize_t ngx_ssl_recv_chain(ngx_connection_t *c, ngx_chain_t *cl, off_t limit); > diff --git a/src/event/quic/ngx_event_quic_ssl.c b/src/event/quic/ngx_event_quic_ssl.c > --- a/src/event/quic/ngx_event_quic_ssl.c > +++ b/src/event/quic/ngx_event_quic_ssl.c > @@ -440,11 +440,9 @@ ngx_quic_crypto_input(ngx_connection_t * > return NGX_OK; > } > > - ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0, > - "quic ssl cipher:%s", SSL_get_cipher(ssl_conn)); > - > - ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0, > - "quic handshake completed successfully"); > +#if (NGX_DEBUG) > + ngx_ssl_handshake_log(c); > +#endif > > c->ssl->handshaked = 1; > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > https://mailman.nginx.org/mailman/listinfo/nginx-devel Looks ok From arut at nginx.com Tue Feb 21 11:58:05 2023 From: arut at nginx.com (Roman Arutyunyan) Date: Tue, 21 Feb 2023 15:58:05 +0400 Subject: [PATCH] QUIC: improved ssl_reject_handshake error logging In-Reply-To: <60b9d9ef878cc87d8a18.1676907424@enoparse.local> References: <60b9d9ef878cc87d8a18.1676907424@enoparse.local> Message-ID: <20230221115805.j7aefptanzhkhyfh@N00W24XTQX> On Mon, Feb 20, 2023 at 07:37:04PM +0400, Sergey Kandaurov wrote: > # HG changeset patch > # User Sergey Kandaurov > # Date 1676907370 -14400 > # Mon Feb 20 19:36:10 2023 +0400 > # Branch quic > # Node ID 60b9d9ef878cc87d8a18e968b4f9122534a7deb8 > # Parent ed403c1fe1f060ee362155bb00c43aaab356e30f > QUIC: improved ssl_reject_handshake error logging. > > The check follows the ngx_ssl_handshake() change in 59e1c73fe02b. > > diff --git a/src/event/quic/ngx_event_quic_ssl.c b/src/event/quic/ngx_event_quic_ssl.c > --- a/src/event/quic/ngx_event_quic_ssl.c > +++ b/src/event/quic/ngx_event_quic_ssl.c > @@ -422,8 +422,17 @@ ngx_quic_crypto_input(ngx_connection_t * > sslerr); > > if (sslerr != SSL_ERROR_WANT_READ) { > + > + qc->error_reason = "handshake failed"; > + > + if (c->ssl->handshake_rejected) { > + ngx_connection_error(c, 0, "handshake rejected"); > + ERR_clear_error(); > + > + return NGX_ERROR; > + } > + > ngx_ssl_error(NGX_LOG_ERR, c->log, 0, "SSL_do_handshake() failed"); > - qc->error_reason = "handshake failed"; > return NGX_ERROR; > } > } > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > https://mailman.nginx.org/mailman/listinfo/nginx-devel Looks ok From arut at nginx.com Tue Feb 21 12:23:00 2023 From: arut at nginx.com (Roman Arutyunyan) Date: Tue, 21 Feb 2023 16:23:00 +0400 Subject: QUIC: add error code for handshake failed. In-Reply-To: <9CFEFC7C-19E1-4AFD-8CE5-2D55AAC02AA8@nginx.com> References: <5b0eee35.4de7.1865a7130cb.Coremail.jiuzhoucui@163.com> <9CFEFC7C-19E1-4AFD-8CE5-2D55AAC02AA8@nginx.com> Message-ID: <20230221122300.x5bw6yo32geilods@N00W24XTQX> On Thu, Feb 16, 2023 at 06:42:27PM +0400, Sergey Kandaurov wrote: [..] > # HG changeset patch > # User Sergey Kandaurov > # Date 1676558213 -14400 > # Thu Feb 16 18:36:53 2023 +0400 > # Branch quic > # Node ID 2fcd590d85da9c3a0205a18cb295ec316c03f18e > # Parent 12b756caaf167d2239fd3bd7a75b270ca89ca26b > QUIC: using NGX_QUIC_ERR_CRYPTO macro in ALPN checks. > > Patch by Jiuzhou Cui. > > diff --git a/src/event/quic/ngx_event_quic_ssl.c b/src/event/quic/ngx_event_quic_ssl.c > --- a/src/event/quic/ngx_event_quic_ssl.c > +++ b/src/event/quic/ngx_event_quic_ssl.c > @@ -190,7 +190,7 @@ ngx_quic_add_handshake_data(ngx_ssl_conn > SSL_get0_alpn_selected(ssl_conn, &alpn_data, &alpn_len); > > if (alpn_len == 0) { > - qc->error = 0x100 + SSL_AD_NO_APPLICATION_PROTOCOL; > + qc->error = NGX_QUIC_ERR_CRYPTO(SSL_AD_NO_APPLICATION_PROTOCOL); > qc->error_reason = "unsupported protocol in ALPN extension"; > > ngx_log_error(NGX_LOG_INFO, c->log, 0, > > -- > Sergey Kandaurov > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > https://mailman.nginx.org/mailman/listinfo/nginx-devel Looks ok From pluknet at nginx.com Tue Feb 21 13:54:42 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Tue, 21 Feb 2023 17:54:42 +0400 Subject: QUIC: add error code for handshake failed. In-Reply-To: <1787FA6E-24CB-4CCE-8A5D-3EFBA7BD9AEC@nginx.com> References: <5b0eee35.4de7.1865a7130cb.Coremail.jiuzhoucui@163.com> <9CFEFC7C-19E1-4AFD-8CE5-2D55AAC02AA8@nginx.com> <170ee7d0.234e.1865dce18a2.Coremail.jiuzhoucui@163.com> <1787FA6E-24CB-4CCE-8A5D-3EFBA7BD9AEC@nginx.com> Message-ID: <6EE82B90-09DA-4C49-96B6-EBCC0B963122@nginx.com> > On 17 Feb 2023, at 16:53, Sergey Kandaurov wrote: > >> >> On 17 Feb 2023, at 09:17, Jiuzhou Cui wrote: >> >> Maybe NGX_QUIC_ERR_CRYPTO(sslerr) is not suitable here. >> >> I think error and error_reason should be set together. >> >> My scenes is error_reason is "handshake failed", but error is 8 set by parse TRANSPORT_PARAMETER failed before. My actual problem is parse transport parameter failed, so maybe error_reason "failed to process transport parameters" is useful for me to positioning problem. What do you think about this? >> > > Actually, both error and error_reason are usually set together. > The problem is in overwriting error_reason that was previously set > specifically for transport parameters parsing error, during handshake, > with a generic "handshake failed" phrase. Please see the patch. > > # HG changeset patch > # User Sergey Kandaurov > # Date 1676638271 -14400 > # Fri Feb 17 16:51:11 2023 +0400 > # Branch quic > # Node ID 09c7414487aa82b40d4ab33738dc578363fe3273 > # Parent 2fcd590d85da9c3a0205a18cb295ec316c03f18e > QUIC: using most specific handshake error reasons. > > A QUIC handshake may fail for a variety of reasons. Some of them arise > from our own checks and have a knownledge to set a suitable reason phrase. > Previously, it was overridden with a generic "handshake failed" phrase. > The latter is used for handshake errors generated in the SSL library and > reported with the send_alert callback. Now a specific phrase is preserved. > > Reported by Jiuzhou Cui. > > diff --git a/src/event/quic/ngx_event_quic_ssl.c b/src/event/quic/ngx_event_quic_ssl.c > --- a/src/event/quic/ngx_event_quic_ssl.c > +++ b/src/event/quic/ngx_event_quic_ssl.c > @@ -423,7 +423,11 @@ ngx_quic_crypto_input(ngx_connection_t * > > if (sslerr != SSL_ERROR_WANT_READ) { > ngx_ssl_error(NGX_LOG_ERR, c->log, 0, "SSL_do_handshake() failed"); > - qc->error_reason = "handshake failed"; > + > + if (!qc->error_reason) { > + qc->error_reason = "handshake failed"; > + } > + > return NGX_ERROR; > } > } Alternatively, the error_reason assignment can just be lifted. # HG changeset patch # User Sergey Kandaurov # Date 1676987626 -14400 # Tue Feb 21 17:53:46 2023 +0400 # Branch quic # Node ID e012c3999592fc935bfc786a232903567c512bfe # Parent baada8d864cbedc7557b879f89eee5c412f3ca85 QUIC: moved "handshake failed" reason to send_alert. A QUIC handshake may fail for a variety of reasons, which breaks down into several cases, in order of precedence: - generally, the error code is reported with the send_alert callback - else, our own QUIC checks may set specific error code / reason phrase - as a last resort, handshake may fail for some reason, which falls back to send INTERNAL_ERROR in the CONNECTION_CLOSE frame. Now, in the first case, both error code and generic phrase are set in the send_alert callback, to preserve a specific reason phrase from overriding. Additionally, absent error code / reason phrase are now converted to just INTERNAL_ERROR, without reason phrase set. Reported by Jiuzhou Cui. diff --git a/src/event/quic/ngx_event_quic_ssl.c b/src/event/quic/ngx_event_quic_ssl.c --- a/src/event/quic/ngx_event_quic_ssl.c +++ b/src/event/quic/ngx_event_quic_ssl.c @@ -301,6 +301,7 @@ ngx_quic_send_alert(ngx_ssl_conn_t *ssl_ } qc->error = NGX_QUIC_ERR_CRYPTO(alert); + qc->error_reason = "handshake failed"; return 1; } @@ -423,7 +424,6 @@ ngx_quic_crypto_input(ngx_connection_t * if (sslerr != SSL_ERROR_WANT_READ) { ngx_ssl_error(NGX_LOG_ERR, c->log, 0, "SSL_do_handshake() failed"); - qc->error_reason = "handshake failed"; return NGX_ERROR; } } -- Sergey Kandaurov From mdounin at mdounin.ru Wed Feb 22 03:42:29 2023 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 22 Feb 2023 06:42:29 +0300 Subject: [PATCH] Documented the use of cyclic memory buffer log with lldb In-Reply-To: <8cb435f59e89b34371ba.1676976365@enoparse.local> References: <8cb435f59e89b34371ba.1676976365@enoparse.local> Message-ID: Hello! On Tue, Feb 21, 2023 at 02:46:05PM +0400, Sergey Kandaurov wrote: > # HG changeset patch > # User Sergey Kandaurov > # Date 1676976309 -14400 > # Tue Feb 21 14:45:09 2023 +0400 > # Node ID 8cb435f59e89b34371bab83f8032efa0bea37817 > # Parent 159dc1fda0d7404aa012cad2ae37a770c4bb8164 > Documented the use of cyclic memory buffer log with lldb. > > diff --git a/xml/en/docs/debugging_log.xml b/xml/en/docs/debugging_log.xml > --- a/xml/en/docs/debugging_log.xml > +++ b/xml/en/docs/debugging_log.xml > @@ -8,7 +8,7 @@ >
link="/en/docs/debugging_log.html" > lang="en" > - rev="5"> > + rev="6"> > > >
> @@ -122,6 +122,13 @@ end > set $buf = (ngx_log_memory_buf_t *) $log->wdata > dump binary memory debug_log.txt $buf->start $buf->end > > +Or using an lldb script as follows: > + > +expr ngx_log_t *$log = ngx_cycle->log > +expr while ($log->writer != ngx_log_memory_writer) { $log = $log->next; } > +expr ngx_log_memory_buf_t *$buf = (ngx_log_memory_buf_t *) $log->wdata > +memory read --force --outfile debug_log.txt --binary $buf->start $buf->end > + > > >
> diff --git a/xml/ru/docs/debugging_log.xml b/xml/ru/docs/debugging_log.xml > --- a/xml/ru/docs/debugging_log.xml > +++ b/xml/ru/docs/debugging_log.xml > @@ -8,7 +8,7 @@ >
link="/ru/docs/debugging_log.html" > lang="ru" > - rev="5"> > + rev="6"> > > >
> @@ -121,6 +121,13 @@ end > set $buf = (ngx_log_memory_buf_t *) $log->wdata > dump binary memory debug_log.txt $buf->start $buf->end > > +Или при помощи такого lldb-скрипта: > + > +expr ngx_log_t *$log = ngx_cycle->log > +expr while ($log->writer != ngx_log_memory_writer) { $log = $log->next; } > +expr ngx_log_memory_buf_t *$buf = (ngx_log_memory_buf_t *) $log->wdata > +memory read --force --outfile debug_log.txt --binary $buf->start $buf->end > + > > >
Looks fine. -- Maxim Dounin http://mdounin.ru/ From pluknet at nginx.com Wed Feb 22 11:37:51 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Wed, 22 Feb 2023 15:37:51 +0400 Subject: [PATCH 1 of 2] Core: connect() error log message made more verbose In-Reply-To: References: <55553146bd984be7e9e3.1675871646@ssafarly-nb> Message-ID: > On 9 Feb 2023, at 12:11, Maxim Dounin wrote: > > Hello! > > On Wed, Feb 08, 2023 at 06:54:06PM +0300, Safar Safarly via nginx-devel wrote: > >> # HG changeset patch >> # User Safar Safarly >> # Date 1675779866 -10800 >> # Tue Feb 07 17:24:26 2023 +0300 >> # Node ID 55553146bd984be7e9e3bbfa851c282feda82d93 >> # Parent cffaf3f2eec8fd33605c2a37814f5ffc30371989 >> Core: connect() error log message made more verbose >> >> There was a major problem in logs: we could not identify to which servers >> connect() has failed. Previously log produced: >> >> ngx_log_error(NGX_LOG_ALERT, ngx_cycle->log, ngx_socket_errno, >> "connect() failed"); >> >> And now we'll have an address or unix socket in log: >> >> ngx_log_error(NGX_LOG_ALERT, ngx_cycle->log, ngx_socket_errno, >> "connect() to %V failed", &peer->server.name); >> >> Message format has chosen to be exact as it is used in ngx_event_connect.c:242 >> with similar error logging: >> >> ngx_log_error(level, c->log, err, "connect() to %V failed", >> pc->name); >> >> So everywhere connect() could fail we'd get a uniform and verbose error message >> in log. > > Thanks for the patch. > > Indeed, it might be non-trivial to find out what goes wrong when > no context is provided in error messages, and logging to syslog > fails to provide any. > > Please see comments below. > > [..] >> >> goto failed; >> } >> diff -r cffaf3f2eec8 -r 55553146bd98 src/core/ngx_syslog.c >> --- a/src/core/ngx_syslog.c Thu Feb 02 23:38:48 2023 +0300 >> +++ b/src/core/ngx_syslog.c Tue Feb 07 17:24:26 2023 +0300 >> @@ -337,7 +337,7 @@ >> >> if (connect(fd, peer->server.sockaddr, peer->server.socklen) == -1) { >> ngx_log_error(NGX_LOG_ALERT, ngx_cycle->log, ngx_socket_errno, >> - "connect() failed"); >> + "connect() to %V failed", &peer->server.name); >> goto failed; >> } > > For syslog logging, I tend to think that this isn't really enough. > For example, consider send() errors which also might happen when > logging to syslog. Right now these are logged as: > > 2023/02/09 07:29:11 [alert] 23230#100162: send() failed (49: Can't assign requested address) > 2023/02/09 07:29:11 [alert] 23230#100162: send() failed (49: Can't assign requested address) > > Clearly no better than connect() errors: > > 2023/02/09 07:29:11 [alert] 23230#100162: connect() failed (2: No such file or directory) > > A better fix would be to provide a log handler, similarly to > ngx_resolver_log_error() mentioned above. With a simple log > handler the above messages will instead look like: > > 2023/02/09 07:48:11 [alert] 23699#100145: send() failed (49: Can't assign requested address) while logging to syslog, server: 127.0.0.2:514 > 2023/02/09 07:48:11 [alert] 23699#100145: send() failed (49: Can't assign requested address) while logging to syslog, server: 127.0.0.2:514 > 2023/02/09 07:48:11 [alert] 23699#100145: connect() failed (2: No such file or directory) while logging to syslog, server: unix:/log.socket > > Patch: > > # HG changeset patch > # User Maxim Dounin > # Date 1675929813 -10800 > # Thu Feb 09 11:03:33 2023 +0300 > # Node ID 6b662855bf77c678a3954939aefe3fd4b4af4c70 > # Parent cffaf3f2eec8fd33605c2a37814f5ffc30371989 > Syslog: introduced error log handler. > > This ensures that errors which happen during logging to syslog are logged > with proper context, such as "while logging to syslog" and the server name. > > Prodded by Safar Safarly. > > diff --git a/src/core/ngx_syslog.c b/src/core/ngx_syslog.c > --- a/src/core/ngx_syslog.c > +++ b/src/core/ngx_syslog.c > @@ -18,6 +18,7 @@ > static char *ngx_syslog_parse_args(ngx_conf_t *cf, ngx_syslog_peer_t *peer); > static ngx_int_t ngx_syslog_init_peer(ngx_syslog_peer_t *peer); > static void ngx_syslog_cleanup(void *data); > +static u_char *ngx_syslog_log_error(ngx_log_t *log, u_char *buf, size_t len); > > > static char *facilities[] = { > @@ -66,6 +67,8 @@ ngx_syslog_process_conf(ngx_conf_t *cf, > ngx_str_set(&peer->tag, "nginx"); > } > > + peer->logp = &cf->cycle->new_log; > + You may want to reflect this change in the description. That's, this now follows other error logging by using log from the configuration that is going to be applied (cycle->new_log): src/core/ngx_log.c: dummy = &cf->cycle->new_log; src/mail/ngx_mail_core_module.c: conf->error_log = &cf->cycle->new_log; src/stream/ngx_stream_core_module.c: conf->error_log = &cf->cycle->new_log; src/http/ngx_http_core_module.c: conf->error_log = &cf->cycle->new_log; src/core/ngx_resolver.c: r->log = &cf->cycle->new_log; src/core/ngx_cycle.c: cycle->log = &cycle->new_log; Previously, before configuration was read, it used init_cycle configuration, that's builtin error_log on the NGX_LOG_NOTICE level, which means that its own ngx_send debug appeared only after the configuration was read, and other syslog error logging was limited to the builtin error log. > peer->conn.fd = (ngx_socket_t) -1; > > peer->conn.read = &ngx_syslog_dummy_event; > @@ -286,15 +289,19 @@ ngx_syslog_send(ngx_syslog_peer_t *peer, > { > ssize_t n; > > + if (peer->log.handler == NULL) { > + peer->log = *peer->logp; > + peer->log.handler = ngx_syslog_log_error; > + peer->log.data = peer; > + peer->log.action = "logging to syslog"; > + } > + > if (peer->conn.fd == (ngx_socket_t) -1) { > if (ngx_syslog_init_peer(peer) != NGX_OK) { > return NGX_ERROR; > } > } > > - /* log syslog socket events with valid log */ > - peer->conn.log = ngx_cycle->log; > - > if (ngx_send) { > n = ngx_send(&peer->conn, buf, len); > One conversion to &peer->log is missing in the ngx_send error handling: diff --git a/src/core/ngx_syslog.c b/src/core/ngx_syslog.c --- a/src/core/ngx_syslog.c +++ b/src/core/ngx_syslog.c @@ -313,7 +313,7 @@ ngx_syslog_send(ngx_syslog_peer_t *peer, if (n == NGX_ERROR) { if (ngx_close_socket(peer->conn.fd) == -1) { - ngx_log_error(NGX_LOG_ALERT, ngx_cycle->log, ngx_socket_errno, + ngx_log_error(NGX_LOG_ALERT, &peer->log, ngx_socket_errno, ngx_close_socket_n " failed"); } Other than that it looks good. > @@ -324,24 +331,25 @@ ngx_syslog_init_peer(ngx_syslog_peer_t * > > fd = ngx_socket(peer->server.sockaddr->sa_family, SOCK_DGRAM, 0); > if (fd == (ngx_socket_t) -1) { > - ngx_log_error(NGX_LOG_ALERT, ngx_cycle->log, ngx_socket_errno, > + ngx_log_error(NGX_LOG_ALERT, &peer->log, ngx_socket_errno, > ngx_socket_n " failed"); > return NGX_ERROR; > } > > if (ngx_nonblocking(fd) == -1) { > - ngx_log_error(NGX_LOG_ALERT, ngx_cycle->log, ngx_socket_errno, > + ngx_log_error(NGX_LOG_ALERT, &peer->log, ngx_socket_errno, > ngx_nonblocking_n " failed"); > goto failed; > } > > if (connect(fd, peer->server.sockaddr, peer->server.socklen) == -1) { > - ngx_log_error(NGX_LOG_ALERT, ngx_cycle->log, ngx_socket_errno, > + ngx_log_error(NGX_LOG_ALERT, &peer->log, ngx_socket_errno, > "connect() failed"); > goto failed; > } > > peer->conn.fd = fd; > + peer->conn.log = &peer->log; > > /* UDP sockets are always ready to write */ > peer->conn.write->ready = 1; > @@ -351,7 +359,7 @@ ngx_syslog_init_peer(ngx_syslog_peer_t * > failed: > > if (ngx_close_socket(fd) == -1) { > - ngx_log_error(NGX_LOG_ALERT, ngx_cycle->log, ngx_socket_errno, > + ngx_log_error(NGX_LOG_ALERT, &peer->log, ngx_socket_errno, > ngx_close_socket_n " failed"); > } > > @@ -372,7 +380,30 @@ ngx_syslog_cleanup(void *data) > } > > if (ngx_close_socket(peer->conn.fd) == -1) { > - ngx_log_error(NGX_LOG_ALERT, ngx_cycle->log, ngx_socket_errno, > + ngx_log_error(NGX_LOG_ALERT, &peer->log, ngx_socket_errno, > ngx_close_socket_n " failed"); > } > } > + > + > +static u_char * > +ngx_syslog_log_error(ngx_log_t *log, u_char *buf, size_t len) > +{ > + u_char *p; > + ngx_syslog_peer_t *peer; > + > + p = buf; > + > + if (log->action) { > + p = ngx_snprintf(buf, len, " while %s", log->action); > + len -= p - buf; > + } > + > + peer = log->data; > + > + if (peer) { > + p = ngx_snprintf(p, len, ", server: %V", &peer->server.name); > + } > + > + return p; > +} > diff --git a/src/core/ngx_syslog.h b/src/core/ngx_syslog.h > --- a/src/core/ngx_syslog.h > +++ b/src/core/ngx_syslog.h > @@ -9,14 +9,18 @@ > > > typedef struct { > - ngx_uint_t facility; > - ngx_uint_t severity; > - ngx_str_t tag; > + ngx_uint_t facility; > + ngx_uint_t severity; > + ngx_str_t tag; > > - ngx_addr_t server; > - ngx_connection_t conn; > - unsigned busy:1; > - unsigned nohostname:1; > + ngx_addr_t server; > + ngx_connection_t conn; > + > + ngx_log_t log; > + ngx_log_t *logp; > + > + unsigned busy:1; > + unsigned nohostname:1; > } ngx_syslog_peer_t; > > -- Sergey Kandaurov From mdounin at mdounin.ru Wed Feb 22 12:55:28 2023 From: mdounin at mdounin.ru (=?utf-8?q?Maxim_Dounin?=) Date: Wed, 22 Feb 2023 15:55:28 +0300 Subject: [PATCH 2 of 2] HTTP/2: finalize request as bad if header validation fails In-Reply-To: <61bd779a868c4021c232.1677070527@vm-bsd.mdounin.ru> References: <61bd779a868c4021c232.1677070527@vm-bsd.mdounin.ru> Message-ID: # HG changeset patch # User Maxim Dounin # Date 1677070454 -10800 # Wed Feb 22 15:54:14 2023 +0300 # Node ID ec3819e66f40924efad183c291c850d9ccef16e7 # Parent 61bd779a868c4021c232dddfe7abda7e8ad32575 HTTP/2: finalize request as bad if header validation fails. Similarly to 7192:d5a535774861, this avoids spurious zero statuses in access.log, and in line with other header-related errors. diff --git a/src/http/v2/ngx_http_v2.c b/src/http/v2/ngx_http_v2.c --- a/src/http/v2/ngx_http_v2.c +++ b/src/http/v2/ngx_http_v2.c @@ -1794,14 +1794,7 @@ ngx_http_v2_state_process_header(ngx_htt /* TODO Optimization: validate headers while parsing. */ if (ngx_http_v2_validate_header(r, header) != NGX_OK) { - if (ngx_http_v2_terminate_stream(h2c, h2c->state.stream, - NGX_HTTP_V2_PROTOCOL_ERROR) - == NGX_ERROR) - { - return ngx_http_v2_connection_error(h2c, - NGX_HTTP_V2_INTERNAL_ERROR); - } - + ngx_http_finalize_request(r, NGX_HTTP_BAD_REQUEST); goto error; } From mdounin at mdounin.ru Wed Feb 22 12:55:27 2023 From: mdounin at mdounin.ru (=?utf-8?q?Maxim_Dounin?=) Date: Wed, 22 Feb 2023 15:55:27 +0300 Subject: [PATCH 1 of 2] HTTP/2: socket leak with "return 444" in error_page (ticket #2455) Message-ID: <61bd779a868c4021c232.1677070527@vm-bsd.mdounin.ru> # HG changeset patch # User Maxim Dounin # Date 1677070356 -10800 # Wed Feb 22 15:52:36 2023 +0300 # Node ID 61bd779a868c4021c232dddfe7abda7e8ad32575 # Parent cffaf3f2eec8fd33605c2a37814f5ffc30371989 HTTP/2: socket leak with "return 444" in error_page (ticket #2455). Similarly to ticket #274 (7354:1812f1d79d84), early request finalization without calling ngx_http_run_posted_requests() resulted in a connection hang (a socket leak) if the 400 (Bad Request) error was generated in ngx_http_v2_state_process_header() due to invalid request headers and "return 444" was used in error_page 400. diff --git a/src/http/v2/ngx_http_v2.c b/src/http/v2/ngx_http_v2.c --- a/src/http/v2/ngx_http_v2.c +++ b/src/http/v2/ngx_http_v2.c @@ -1730,6 +1730,7 @@ ngx_http_v2_state_process_header(ngx_htt size_t len; ngx_int_t rc; ngx_table_elt_t *h; + ngx_connection_t *fc; ngx_http_header_t *hh; ngx_http_request_t *r; ngx_http_v2_header_t *header; @@ -1789,6 +1790,7 @@ ngx_http_v2_state_process_header(ngx_htt } r = h2c->state.stream->request; + fc = r->connection; /* TODO Optimization: validate headers while parsing. */ if (ngx_http_v2_validate_header(r, header) != NGX_OK) { @@ -1886,6 +1888,8 @@ error: h2c->state.stream = NULL; + ngx_http_run_posted_requests(fc); + return ngx_http_v2_state_header_complete(h2c, pos, end); } From arut at nginx.com Wed Feb 22 13:15:46 2023 From: arut at nginx.com (Roman Arutyunyan) Date: Wed, 22 Feb 2023 17:15:46 +0400 Subject: QUIC: add error code for handshake failed. In-Reply-To: <6EE82B90-09DA-4C49-96B6-EBCC0B963122@nginx.com> References: <5b0eee35.4de7.1865a7130cb.Coremail.jiuzhoucui@163.com> <9CFEFC7C-19E1-4AFD-8CE5-2D55AAC02AA8@nginx.com> <170ee7d0.234e.1865dce18a2.Coremail.jiuzhoucui@163.com> <1787FA6E-24CB-4CCE-8A5D-3EFBA7BD9AEC@nginx.com> <6EE82B90-09DA-4C49-96B6-EBCC0B963122@nginx.com> Message-ID: <20230222131546.2sgyga72jnp7gsiv@N00W24XTQX> On Tue, Feb 21, 2023 at 05:54:42PM +0400, Sergey Kandaurov wrote: [..] > # HG changeset patch > # User Sergey Kandaurov > # Date 1676987626 -14400 > # Tue Feb 21 17:53:46 2023 +0400 > # Branch quic > # Node ID e012c3999592fc935bfc786a232903567c512bfe > # Parent baada8d864cbedc7557b879f89eee5c412f3ca85 > QUIC: moved "handshake failed" reason to send_alert. > > A QUIC handshake may fail for a variety of reasons, which breaks down into > several cases, in order of precedence: > - generally, the error code is reported with the send_alert callback > - else, our own QUIC checks may set specific error code / reason phrase > - as a last resort, handshake may fail for some reason, which falls back > to send INTERNAL_ERROR in the CONNECTION_CLOSE frame. > Now, in the first case, both error code and generic phrase are set in the > send_alert callback, to preserve a specific reason phrase from overriding. > Additionally, absent error code / reason phrase are now converted to just > INTERNAL_ERROR, without reason phrase set. i> > Reported by Jiuzhou Cui. I suggest the following commit message: QUIC: moved "handshake failed" reason to send_alert. A QUIC handshake may fail for a variety of reasons, which breaks down into several cases, in order of precedence: - a handshake error which leads to a send_alert call - an error triggered by add_handshake_data callback - internal errors (allocation etc) Previously, in the first case, only error code was set in send_alert callback. Now "handshake failed" is set there as a reason. In the second case, error code and reason are set by add_handshake_data. In the last case, setting error reason is now removed. Returning NGX_ERROR will lead to closing the connection with just INTERNAL_ERROR. Reported by Jiuzhou Cui. > diff --git a/src/event/quic/ngx_event_quic_ssl.c b/src/event/quic/ngx_event_quic_ssl.c > --- a/src/event/quic/ngx_event_quic_ssl.c > +++ b/src/event/quic/ngx_event_quic_ssl.c > @@ -301,6 +301,7 @@ ngx_quic_send_alert(ngx_ssl_conn_t *ssl_ > } > > qc->error = NGX_QUIC_ERR_CRYPTO(alert); > + qc->error_reason = "handshake failed"; > > return 1; > } > @@ -423,7 +424,6 @@ ngx_quic_crypto_input(ngx_connection_t * > > if (sslerr != SSL_ERROR_WANT_READ) { > ngx_ssl_error(NGX_LOG_ERR, c->log, 0, "SSL_do_handshake() failed"); > - qc->error_reason = "handshake failed"; > return NGX_ERROR; > } > } > > -- > Sergey Kandaurov > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > https://mailman.nginx.org/mailman/listinfo/nginx-devel The patch looks good. -- Roman Arutyunyan From arut at nginx.com Wed Feb 22 14:20:35 2023 From: arut at nginx.com (Roman Arutyunyan) Date: Wed, 22 Feb 2023 18:20:35 +0400 Subject: [PATCH] QUIC: OpenSSL compatibility layer In-Reply-To: <20230208151709.74ttczym3bgwfdli@Y9MQ9X2QVV> References: <64a365dcb52503e91d91.1671606452@arut-laptop> <20230109111316.yfqj24unoubibihz@N00W24XTQX> <20230202183522.o4w5xbjvegbchts4@Y9MQ9X2QVV> <20230206142701.pxzaq3irckqads7y@N00W24XTQX> <20230208122810.tsxynyhlqgt2ubsm@Y9MQ9X2QVV> <20230208144138.u7pozeds34nqr6mi@N00W24XTQX> <20230208151709.74ttczym3bgwfdli@Y9MQ9X2QVV> Message-ID: <20230222142035.yixtoxtqqvlbxwbc@N00W24XTQX> Hi, On Wed, Feb 08, 2023 at 07:17:09PM +0400, Sergey Kandaurov wrote: > On Wed, Feb 08, 2023 at 06:41:38PM +0400, Roman Arutyunyan wrote: > > # HG changeset patch > > # User Roman Arutyunyan > > # Date 1675867049 -14400 > > # Wed Feb 08 18:37:29 2023 +0400 > > # Branch quic > > # Node ID a3142c8833f5bf1186599e7938141f5062fac4a2 > > # Parent 3c33d39a51d334d99fcc7d2b45e8d8190c431492 > > QUIC: OpenSSL compatibility layer. > > > > The change allows to compile QUIC with OpenSSL which lacks BoringSSL QUIC API. > > > > This implementation does not support 0-RTT. > > > > Looks good. A short summary about 0-RTT support in the compatibility mode. One of the differences between QUIC and TLS protocols is missing EndOfEarlyData message in QUIC compared to TLS[1]. It does not manifest itself unless 0-RTT is sent by client. When 0-RTT is sent, EndOfEarlyData should be injected into the TLS handshake message stream while converting QUIC to TLS to satisfy OpenSSL state machine. The message itself is quite simple, but it changes the handshake hash, which leads to significant consequences like mismatching Finished MAC and PSK Binder. While in theory they can be recalculated, the solution looks relatively complex. I suggest that we postpone this work. Also, here's a small update to the original patch: diff --git a/README b/README --- a/README +++ b/README @@ -71,6 +71,7 @@ 2. Building from sources Alternatively, nginx can be configured with OpenSSL compatibility layer, which emulates BoringSSL QUIC API for OpenSSL. This mode is enabled by default if native QUIC support is not detected. + 0-RTT is not supported in OpenSSL compatibility mode. Clone the NGINX QUIC repository [1] https://datatracker.ietf.org/doc/html/rfc9001#section-8.3 -- Roman Arutyunyan From u5.horie at gmail.com Wed Feb 22 15:10:27 2023 From: u5.horie at gmail.com (u5h) Date: Thu, 23 Feb 2023 00:10:27 +0900 Subject: [PATCH] Core: return error when the first byte is above 0xf5 in utf-8 Message-ID: # HG changeset patch # User Yugo Horie # Date 1677077775 -32400 # Wed Feb 22 23:56:15 2023 +0900 # Node ID 1a9487706c6af90baf2ed770db29f689c3850721 # Parent cffaf3f2eec8fd33605c2a37814f5ffc30371989 core: return error when the first byte is above 0xf5 in utf-8 * see https://datatracker.ietf.org/doc/html/rfc3629#section-4 diff -r cffaf3f2eec8 -r 1a9487706c6a src/core/ngx_string.c --- a/src/core/ngx_string.c Thu Feb 02 23:38:48 2023 +0300 +++ b/src/core/ngx_string.c Wed Feb 22 23:56:15 2023 +0900 @@ -1364,7 +1364,7 @@ u = **p; - if (u >= 0xf0) { + if (u < 0xf5 && u >= 0xf0) { u &= 0x07; valid = 0xffff; -------------- next part -------------- An HTML attachment was scrubbed... URL: From pluknet at nginx.com Wed Feb 22 15:39:13 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Wed, 22 Feb 2023 19:39:13 +0400 Subject: [PATCH 01 of 12] Win32: non-ASCII names support in autoindex (ticket #458) In-Reply-To: References: <60d845f9505fe1b97c1e.1673559324@1.0.0.127.in-addr.arpa> Message-ID: <011AED5C-7D9F-4B3B-8E9A-646110B9108D@nginx.com> > On 19 Feb 2023, at 21:17, Maxim Dounin wrote: > > Hello! > > On Fri, Feb 17, 2023 at 06:38:54PM +0400, Sergey Kandaurov wrote: > >>> On 13 Jan 2023, at 01:35, Maxim Dounin wrote: >>> >>> # HG changeset patch >>> # User Maxim Dounin >>> # Date 1673548890 -10800 >>> # Thu Jan 12 21:41:30 2023 +0300 >>> # Node ID 60d845f9505fe1b97c1e04b680523b790e29fdb1 >>> # Parent 07b0bee87f32be91a33210bc06973e07c4c1dac9 >>> Win32: non-ASCII names support in autoindex (ticket #458). >>> >>> Notably, ngx_open_dir() now supports opening directories with non-ASCII >>> characters, and directory entries returned by ngx_read_dir() are properly >>> converted to UTF-8. >>> >>> diff -r 07b0bee87f32 -r 60d845f9505f src/os/win32/ngx_files.c >>> --- a/src/os/win32/ngx_files.c Wed Dec 21 14:53:27 2022 +0300 >>> +++ b/src/os/win32/ngx_files.c Thu Jan 12 21:41:30 2023 +0300 >>> @@ -13,7 +13,11 @@ >>> >>> static ngx_int_t ngx_win32_check_filename(u_char *name, u_short *u, >>> size_t len); >>> -static u_short *ngx_utf8_to_utf16(u_short *utf16, u_char *utf8, size_t *len); >>> +static u_short *ngx_utf8_to_utf16(u_short *utf16, u_char *utf8, size_t *len, >>> + size_t reserved); >>> +static u_char *ngx_utf16_to_utf8(u_char *utf8, u_short *utf16, size_t *len, >>> + size_t *allocated); >>> +uint32_t ngx_utf16_decode(u_short **u, size_t n); >>> >>> >>> /* FILE_FLAG_BACKUP_SEMANTICS allows to obtain a handle to a directory */ >>> @@ -28,7 +32,7 @@ ngx_open_file(u_char *name, u_long mode, >>> u_short utf16[NGX_UTF16_BUFLEN]; >>> >>> len = NGX_UTF16_BUFLEN; >>> - u = ngx_utf8_to_utf16(utf16, name, &len); >>> + u = ngx_utf8_to_utf16(utf16, name, &len, 0); >>> >>> if (u == NULL) { >>> return INVALID_HANDLE_VALUE; >>> @@ -269,7 +273,7 @@ ngx_file_info(u_char *file, ngx_file_inf >>> >>> len = NGX_UTF16_BUFLEN; >>> >>> - u = ngx_utf8_to_utf16(utf16, file, &len); >>> + u = ngx_utf8_to_utf16(utf16, file, &len, 0); >>> >>> if (u == NULL) { >>> return NGX_FILE_ERROR; >>> @@ -427,49 +431,51 @@ ngx_realpath(u_char *path, u_char *resol >>> ngx_int_t >>> ngx_open_dir(ngx_str_t *name, ngx_dir_t *dir) >>> { >>> - u_char *pattern, *p; >>> + size_t len; >>> + u_short *u, *p; >>> ngx_err_t err; >>> + u_short utf16[NGX_UTF16_BUFLEN]; >>> >>> - pattern = malloc(name->len + 3); >>> - if (pattern == NULL) { >>> + len = NGX_UTF16_BUFLEN - 2; >>> + u = ngx_utf8_to_utf16(utf16, name->data, &len, 2); >>> + >>> + if (u == NULL) { >>> return NGX_ERROR; >>> } >>> >>> - p = ngx_cpymem(pattern, name->data, name->len); >>> + if (ngx_win32_check_filename(name->data, u, len) != NGX_OK) { >>> + goto failed; >>> + } >>> + >>> + p = &u[len - 1]; >>> >>> *p++ = '/'; >>> *p++ = '*'; >>> *p = '\0'; >>> >>> - dir->dir = FindFirstFile((const char *) pattern, &dir->finddata); >>> + dir->dir = FindFirstFileW(u, &dir->finddata); >>> >>> if (dir->dir == INVALID_HANDLE_VALUE) { >>> - err = ngx_errno; >>> - ngx_free(pattern); >>> - ngx_set_errno(err); >>> - return NGX_ERROR; >>> + goto failed; >>> } >>> >>> - ngx_free(pattern); >>> + if (u != utf16) { >>> + ngx_free(u); >>> + } >>> >>> dir->valid_info = 1; >>> dir->ready = 1; >>> + dir->name = NULL; >>> + dir->allocated = 0; >>> >>> return NGX_OK; >>> -} >>> >>> +failed: >>> >>> -ngx_int_t >>> -ngx_read_dir(ngx_dir_t *dir) >>> -{ >>> - if (dir->ready) { >>> - dir->ready = 0; >>> - return NGX_OK; >>> - } >>> - >>> - if (FindNextFile(dir->dir, &dir->finddata) != 0) { >>> - dir->type = 1; >>> - return NGX_OK; >>> + if (u != utf16) { >>> + err = ngx_errno; >>> + ngx_free(u); >>> + ngx_set_errno(err); >>> } >>> >>> return NGX_ERROR; >>> @@ -477,8 +483,56 @@ ngx_read_dir(ngx_dir_t *dir) >>> >>> >>> ngx_int_t >>> +ngx_read_dir(ngx_dir_t *dir) >>> +{ >>> + u_char *name; >>> + size_t len, allocated; >>> + >>> + if (dir->ready) { >>> + dir->ready = 0; >>> + goto convert; >>> + } >>> + >>> + if (FindNextFileW(dir->dir, &dir->finddata) != 0) { >>> + dir->type = 1; >>> + goto convert; >>> + } >>> + >>> + return NGX_ERROR; >>> + >>> +convert: >>> + >>> + name = dir->name; >>> + len = dir->allocated; >>> + >>> + name = ngx_utf16_to_utf8(name, dir->finddata.cFileName, &len, &allocated); >>> + if (name == NULL) { >> >> Note that you just return on NULL, but see below. >> >>> + return NGX_ERROR; >>> + } >>> + >>> + if (name != dir->name) { >>> + >>> + if (dir->name) { >>> + ngx_free(dir->name); >>> + } >>> + >>> + dir->name = name; >>> + dir->allocated = allocated; >>> + } >>> + >>> + dir->namelen = len - 1; >>> + >>> + return NGX_OK; >>> +} >>> + >>> + >>> +ngx_int_t >>> ngx_close_dir(ngx_dir_t *dir) >>> { >>> + if (dir->name) { >>> + ngx_free(dir->name); >>> + } >>> + >>> if (FindClose(dir->dir) == 0) { >>> return NGX_ERROR; >>> } >>> @@ -816,7 +870,7 @@ failed: >>> >>> >>> static u_short * >>> -ngx_utf8_to_utf16(u_short *utf16, u_char *utf8, size_t *len) >>> +ngx_utf8_to_utf16(u_short *utf16, u_char *utf8, size_t *len, size_t reserved) >>> { >>> u_char *p; >>> u_short *u, *last; >>> @@ -865,7 +919,7 @@ ngx_utf8_to_utf16(u_short *utf16, u_char >>> >>> /* the given buffer is not enough, allocate a new one */ >>> >>> - u = malloc(((p - utf8) + ngx_strlen(p) + 1) * sizeof(u_short)); >>> + u = malloc(((p - utf8) + ngx_strlen(p) + 1 + reserved) * sizeof(u_short)); >>> if (u == NULL) { >>> return NULL; >>> } >>> @@ -910,3 +964,170 @@ ngx_utf8_to_utf16(u_short *utf16, u_char >>> >>> /* unreachable */ >>> } >>> + >>> + >>> +static u_char * >>> +ngx_utf16_to_utf8(u_char *utf8, u_short *utf16, size_t *len, size_t *allocated) >>> +{ >>> + u_char *p, *last; >>> + u_short *u, *j; >>> + uint32_t n; >>> + >>> + u = utf16; >>> + p = utf8; >>> + last = utf8 + *len; >>> + >>> + while (p < last) { >>> + >>> + if (*u < 0x80) { >>> + *p++ = (u_char) *u; >>> + >>> + if (*u == 0) { >>> + *len = p - utf8; >>> + return utf8; >>> + } >>> + >>> + u++; >>> + >>> + continue; >>> + } >>> + >>> + if (p >= last - 4) { >>> + *len = p - utf8; >>> + break; >>> + } >>> + >>> + n = ngx_utf16_decode(&u, 2); >>> + >>> + if (n > 0x10ffff) { >>> + ngx_free(utf8); >>> + ngx_set_errno(NGX_EILSEQ); >> >> This can result in double-free on a subsequent ngx_close_dir() call. > > Thanks for catching, that's slipped in from the second loop. > Removed ngx_free() here. I believe it should be fine now. > >> >>> + return NULL; >>> + } >>> + >>> + if (n >= 0x10000) { >>> + *p++ = (u_char) (0xf0 + (n >> 18)); >>> + *p++ = (u_char) (0x80 + ((n >> 12) & 0x3f)); >>> + *p++ = (u_char) (0x80 + ((n >> 6) & 0x3f)); >>> + *p++ = (u_char) (0x80 + (n & 0x3f)); >>> + continue; >>> + >> >> stray ws-only line (one more below) > > Fixed both, thanks. > >>> + } >>> + >>> + if (n >= 0x0800) { >>> + *p++ = (u_char) (0xe0 + (n >> 12)); >>> + *p++ = (u_char) (0x80 + ((n >> 6) & 0x3f)); >>> + *p++ = (u_char) (0x80 + (n & 0x3f)); >>> + continue; >>> + } >>> + >>> + *p++ = (u_char) (0xc0 + (n >> 6)); >>> + *p++ = (u_char) (0x80 + (n & 0x3f)); >>> + } >>> + >>> + /* the given buffer is not enough, allocate a new one */ >>> + >>> + for (j = u; *j; j++) { /* void */ } >>> + >>> + p = malloc((j - utf16) * 4 + 1); >> >> [Note that pointer arithmetics is in u_short (2 bytes) units.] >> It seems you are allocating memory assuming that utf16 -> utf8 >> conversion can result in memory expansion up to 4 bytes per unit. >> At first glance that's true, because UTF-8 symbol may take 4 bytes. >> IIUC, 4-byte UTF-8 can result only from UTF-16 surrogate pairs, >> this gives 2x growth (2-paired UTF-16 to 4-bytes UTF-8). >> Otherwise, 1/2/3-byte encoded UTF-8 can result from 2-byte UTF-16, >> which gives from 1x to 3x growth per unit. >> So, "malloc((j - utf16) * 3 + 1)" should be enough there. > > That's correct, though this depends on not-directly-visible > encoding details. I would rather prefer to keep 4 here, since it > can be easily verified to be enough from the loop itself. > > I don't think that a minor optimization of memory usage by going > from 4x to 3x of the particular name is important here. Further, > extra allocated bytes are likely to be used as a preallocation for > other names. > Ok, it makes sense then. >>> + if (p == NULL) { >>> + return NULL; >>> + } >> >> Old memory is still there, allocated and referenced by dir->name, >> see above. It seems ok, it should be freed by ngx_close_dir() call. >> If we happen to change callers to not call ngx_close_dir() on error, >> this could turn into memory leaks, though. > > The ngx_close_dir() is certainly required (including for other > reasons), so this looks perfectly fine. > >>> + >>> + if (allocated) { >>> + *allocated = (j - utf16) * 4 + 1; >>> + } >>> + >>> + ngx_memcpy(p, utf8, *len); >>> + >>> + utf8 = p; >>> + p += *len; >>> + >>> + for ( ;; ) { >>> + >>> + if (*u < 0x80) { >>> + *p++ = (u_char) *u; >>> + >>> + if (*u == 0) { >>> + *len = p - utf8; >>> + return utf8; >>> + } >>> + >>> + u++; >>> + >>> + continue; >>> + } >>> + >>> + n = ngx_utf16_decode(&u, 2); >>> + >>> + if (n > 0x10ffff) { >>> + ngx_free(utf8); >>> + ngx_set_errno(NGX_EILSEQ); >>> + return NULL; >>> + } >>> + >>> + if (n >= 0x10000) { >>> + *p++ = (u_char) (0xf0 + (n >> 18)); >>> + *p++ = (u_char) (0x80 + ((n >> 12) & 0x3f)); >>> + *p++ = (u_char) (0x80 + ((n >> 6) & 0x3f)); >>> + *p++ = (u_char) (0x80 + (n & 0x3f)); >>> + continue; >>> + >>> + } >>> + >>> + if (n >= 0x0800) { >>> + *p++ = (u_char) (0xe0 + (n >> 12)); >>> + *p++ = (u_char) (0x80 + ((n >> 6) & 0x3f)); >>> + *p++ = (u_char) (0x80 + (n & 0x3f)); >>> + continue; >>> + } >>> + >>> + *p++ = (u_char) (0xc0 + (n >> 6)); >>> + *p++ = (u_char) (0x80 + (n & 0x3f)); >>> + } >>> + >>> + /* unreachable */ >>> +} >>> + >>> + >>> +/* >>> + * ngx_utf16_decode() decodes one or two UTF-16 code units >>> + * the return values: >>> + * 0x80 - 0x10ffff valid character >>> + * 0x110000 - 0xfffffffd invalid sequence >>> + * 0xfffffffe incomplete sequence >>> + * 0xffffffff error >>> + */ >>> + >>> +uint32_t >>> +ngx_utf16_decode(u_short **u, size_t n) >>> +{ >>> + uint32_t k, m; >>> + >>> + k = **u; >>> + >>> + if (k < 0xd800 || k > 0xdfff) { >>> + (*u)++; >>> + return k; >>> + } >>> + >>> + if (k > 0xdbff) { >>> + (*u)++; >>> + return 0xffffffff; >>> + } >>> + >>> + if (n < 2) { >>> + return 0xfffffffe; >>> + } >>> + >>> + (*u)++; >>> + >>> + m = *(*u)++; >>> + >>> + if (m < 0xdc00 || m > 0xdfff) { >>> + return 0xffffffff; >>> + >>> + } >>> + >>> + return 0x10000 + ((k - 0xd800) << 10) + (m - 0xdc00); >>> +} >>> diff -r 07b0bee87f32 -r 60d845f9505f src/os/win32/ngx_files.h >>> --- a/src/os/win32/ngx_files.h Wed Dec 21 14:53:27 2022 +0300 >>> +++ b/src/os/win32/ngx_files.h Thu Jan 12 21:41:30 2023 +0300 >>> @@ -30,7 +30,11 @@ typedef struct { >>> >>> typedef struct { >>> HANDLE dir; >>> - WIN32_FIND_DATA finddata; >>> + WIN32_FIND_DATAW finddata; >>> + >>> + u_char *name; >>> + size_t namelen; >>> + size_t allocated; >>> >>> unsigned valid_info:1; >>> unsigned type:1; >>> @@ -205,8 +209,8 @@ ngx_int_t ngx_close_dir(ngx_dir_t *dir); >>> #define ngx_dir_access(a) (a) >>> >>> >>> -#define ngx_de_name(dir) ((u_char *) (dir)->finddata.cFileName) >>> -#define ngx_de_namelen(dir) ngx_strlen((dir)->finddata.cFileName) >>> +#define ngx_de_name(dir) (dir)->name >>> +#define ngx_de_namelen(dir) (dir)->namelen >>> >>> ngx_int_t ngx_de_info(u_char *name, ngx_dir_t *dir); >>> #define ngx_de_info_n "dummy()" >> >> Otherwise, looks good. > > Fixes: > > diff -r d7521ae9e138 src/os/win32/ngx_files.c > --- a/src/os/win32/ngx_files.c Sat Feb 18 12:08:25 2023 +0300 > +++ b/src/os/win32/ngx_files.c Sat Feb 18 12:57:02 2023 +0300 > @@ -506,6 +506,7 @@ convert: > len = dir->allocated; > > name = ngx_utf16_to_utf8(name, dir->finddata.cFileName, &len, &allocated); > + > if (name == NULL) { > return NGX_ERROR; > } > @@ -1000,7 +1000,6 @@ ngx_utf16_to_utf8(u_char *utf8, u_short > n = ngx_utf16_decode(&u, 2); > > if (n > 0x10ffff) { > - ngx_free(utf8); > ngx_set_errno(NGX_EILSEQ); > return NULL; > } > @@ -1011,7 +1010,6 @@ ngx_utf16_to_utf8(u_char *utf8, u_short > *p++ = (u_char) (0x80 + ((n >> 6) & 0x3f)); > *p++ = (u_char) (0x80 + (n & 0x3f)); > continue; > - > } > > if (n >= 0x0800) { > @@ -1072,7 +1070,6 @@ ngx_utf16_to_utf8(u_char *utf8, u_short > *p++ = (u_char) (0x80 + ((n >> 6) & 0x3f)); > *p++ = (u_char) (0x80 + (n & 0x3f)); > continue; > - > } > > if (n >= 0x0800) { > Looks good. -- Sergey Kandaurov From pluknet at nginx.com Wed Feb 22 15:49:57 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Wed, 22 Feb 2023 19:49:57 +0400 Subject: [PATCH 02 of 12] Win32: non-ASCII names support in "include" with wildcards In-Reply-To: References: Message-ID: <6F208EF1-CEBE-43F2-982E-DB8AC9EF312B@nginx.com> > On 19 Feb 2023, at 21:18, Maxim Dounin wrote: > > Hello! > > On Fri, Feb 17, 2023 at 06:53:11PM +0400, Sergey Kandaurov wrote: > >>> On 13 Jan 2023, at 01:35, Maxim Dounin wrote: >>> >>> # HG changeset patch >>> # User Maxim Dounin >>> # Date 1673548899 -10800 >>> # Thu Jan 12 21:41:39 2023 +0300 >>> # Node ID d05c0adf5890aecc68ce8906ef19ca07502ed071 >>> # Parent 60d845f9505fe1b97c1e04b680523b790e29fdb1 >>> Win32: non-ASCII names support in "include" with wildcards. >>> >>> Notably, ngx_open_glob() now supports opening directories with non-ASCII >>> characters, and pathnames returned by ngx_read_glob() are converted to UTF-8. >>> >>> diff -r 60d845f9505f -r d05c0adf5890 src/os/win32/ngx_files.c >>> --- a/src/os/win32/ngx_files.c Thu Jan 12 21:41:30 2023 +0300 >>> +++ b/src/os/win32/ngx_files.c Thu Jan 12 21:41:39 2023 +0300 >>> @@ -546,14 +546,27 @@ ngx_open_glob(ngx_glob_t *gl) >>> { >>> u_char *p; >>> size_t len; >>> + u_short *u; >>> ngx_err_t err; >>> + u_short utf16[NGX_UTF16_BUFLEN]; >>> >>> - gl->dir = FindFirstFile((const char *) gl->pattern, &gl->finddata); >>> + len = NGX_UTF16_BUFLEN; >>> + u = ngx_utf8_to_utf16(utf16, gl->pattern, &len, 0); >>> + >>> + if (u == NULL) { >>> + return NGX_ERROR; >>> + } >>> + >>> + gl->dir = FindFirstFileW(u, &gl->finddata); >>> >>> if (gl->dir == INVALID_HANDLE_VALUE) { >>> >>> err = ngx_errno; >>> >>> + if (u != utf16) { >>> + ngx_free(u); >>> + } >>> + >>> if ((err == ERROR_FILE_NOT_FOUND || err == ERROR_PATH_NOT_FOUND) >>> && gl->test) >>> { >>> @@ -561,6 +574,8 @@ ngx_open_glob(ngx_glob_t *gl) >>> return NGX_OK; >>> } >>> >>> + ngx_set_errno(err); >>> + >>> return NGX_ERROR; >>> } >>> >>> @@ -570,18 +585,10 @@ ngx_open_glob(ngx_glob_t *gl) >>> } >>> } >>> >>> - len = ngx_strlen(gl->finddata.cFileName); >>> - gl->name.len = gl->last + len; >>> - >>> - gl->name.data = ngx_alloc(gl->name.len + 1, gl->log); >>> - if (gl->name.data == NULL) { >>> - return NGX_ERROR; >>> + if (u != utf16) { >>> + ngx_free(u); >>> } >>> >>> - ngx_memcpy(gl->name.data, gl->pattern, gl->last); >>> - ngx_cpystrn(gl->name.data + gl->last, (u_char *) gl->finddata.cFileName, >>> - len + 1); >>> - >>> gl->ready = 1; >>> >>> return NGX_OK; >>> @@ -591,40 +598,25 @@ ngx_open_glob(ngx_glob_t *gl) >>> ngx_int_t >>> ngx_read_glob(ngx_glob_t *gl, ngx_str_t *name) >>> { >>> - size_t len; >>> - ngx_err_t err; >>> + u_char *p; >>> + size_t len; >>> + ngx_err_t err; >>> + u_char utf8[NGX_UTF16_BUFLEN]; >> >> Mixing UTF16-specific macro with u_char utf8[] looks suspicious. > > That's just some preallocated buffer size. If it feels better, a > separate macro can be used instead. This will also make it > possible to fine-tune the buffer: > I don't have a strong position here, just caught my eye. Feel free to commit the version you prefer. > diff -r aca8812f2482 src/os/win32/ngx_files.c > --- a/src/os/win32/ngx_files.c Sat Feb 18 12:58:54 2023 +0300 > +++ b/src/os/win32/ngx_files.c Sat Feb 18 13:05:34 2023 +0300 > @@ -10,6 +10,7 @@ > > > #define NGX_UTF16_BUFLEN 256 > +#define NGX_UTF8_BUFLEN 512 > > static ngx_int_t ngx_win32_check_filename(u_char *name, u_short *u, > size_t len); > @@ -601,7 +602,7 @@ ngx_read_glob(ngx_glob_t *gl, ngx_str_t > u_char *p; > size_t len; > ngx_err_t err; > - u_char utf8[NGX_UTF16_BUFLEN]; > + u_char utf8[NGX_UTF8_BUFLEN]; > > if (gl->no_match) { > return NGX_DONE; > @@ -632,7 +633,7 @@ ngx_read_glob(ngx_glob_t *gl, ngx_str_t > > convert: > > - len = NGX_UTF16_BUFLEN; > + len = NGX_UTF8_BUFLEN; > p = ngx_utf16_to_utf8(utf8, gl->finddata.cFileName, &len, NULL); > > if (p == NULL) { > > >> It may have sense to replace this with dynamic allocation >> inside ngx_utf16_to_utf8(), similar to ngx_read_dir() 1st call, >> or use NGX_MAX_PATH. > > Much like in other functions in this file, the idea is to avoid > dynamic allocations in most cases by providing an easy to > implement stack-based buffer. The approach used in ngx_read_dir() > is more complex since the result of the conversion needs to be > maintained between calls (to be available via the ngx_de_name() > macro). For ngx_read_glob(), a better approach might be to > combine conversion buffer and the final glob() result buffer > (which also includes directory name, and currently allocated on > each ngx_read_glob() call). Though I don't think it worth the > effort, especially given that ngx_read_glob() is only used during > configuration parsing. > > As for NGX_MAX_PATH, it does not look like a good replacement for > a number of reasons. In particular, a) it might be incorrectly > interpreted as a place when only paths up to MAX_PATH are > supported, b) MAX_PATH on win32 is in characters, while the buffer > in question is in bytes (and to be used for UTF-8), so it is no > better than NGX_UTF16_BUFLEN. > Agree. >>> >>> if (gl->no_match) { >>> return NGX_DONE; >>> } >>> >>> if (gl->ready) { >>> - *name = gl->name; >>> - >>> gl->ready = 0; >>> - return NGX_OK; >>> + goto convert; >>> } >>> >>> ngx_free(gl->name.data); >>> gl->name.data = NULL; >>> >>> - if (FindNextFile(gl->dir, &gl->finddata) != 0) { >>> - >>> - len = ngx_strlen(gl->finddata.cFileName); >>> - gl->name.len = gl->last + len; >>> - >>> - gl->name.data = ngx_alloc(gl->name.len + 1, gl->log); >>> - if (gl->name.data == NULL) { >>> - return NGX_ERROR; >>> - } >>> - >>> - ngx_memcpy(gl->name.data, gl->pattern, gl->last); >>> - ngx_cpystrn(gl->name.data + gl->last, (u_char *) gl->finddata.cFileName, >>> - len + 1); >>> - >>> - *name = gl->name; >>> - >>> - return NGX_OK; >>> + if (FindNextFileW(gl->dir, &gl->finddata) != 0) { >>> + goto convert; >>> } >>> >>> err = ngx_errno; >>> @@ -637,6 +629,43 @@ ngx_read_glob(ngx_glob_t *gl, ngx_str_t >>> "FindNextFile(%s) failed", gl->pattern); >>> >>> return NGX_ERROR; >>> + >>> +convert: >>> + >>> + len = NGX_UTF16_BUFLEN; >>> + p = ngx_utf16_to_utf8(utf8, gl->finddata.cFileName, &len, NULL); >> >> Note that passing utf8 off the stack can result in an attempt to free >> the stack pointer inside ngx_utf16_to_utf8() on NGX_EILSEQ. > > Yes, thanks, this shouldn't be a problem with the > ngx_utf16_to_utf8() already fixed based on the 1st patch comments. > >>> + >> >> Nitpicking: there (and in subsequent patches) you're starting to add >> an empty line after ngx_utf16_to_utf8(); it's not in the 1st path. > > In the first patch there is an empty line before the > ngx_utf16_to_utf8() call, which somewhat balance things. Either > way, added an empty line there as well. > > [...] > -- Sergey Kandaurov From pluknet at nginx.com Wed Feb 22 16:00:24 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Wed, 22 Feb 2023 20:00:24 +0400 Subject: [PATCH 03 of 12] Win32: non-ASCII directory names support in ngx_getcwd() In-Reply-To: References: <7cf820c46860796cff91.1673559326@1.0.0.127.in-addr.arpa> Message-ID: > On 19 Feb 2023, at 21:23, Maxim Dounin wrote: > > Hello! > > On Fri, Feb 17, 2023 at 07:04:04PM +0400, Sergey Kandaurov wrote: > >>> On 13 Jan 2023, at 01:35, Maxim Dounin wrote: >>> >>> # HG changeset patch >>> # User Maxim Dounin >>> # Date 1673548916 -10800 >>> # Thu Jan 12 21:41:56 2023 +0300 >>> # Node ID 7cf820c46860796cff91f53a5d2db669bb5b5a6c >>> # Parent d05c0adf5890aecc68ce8906ef19ca07502ed071 >>> Win32: non-ASCII directory names support in ngx_getcwd(). >>> >>> This makes it possible to start nginx without a prefix explicitly set >>> in a directory with non-ASCII characters in it. >>> >>> diff -r d05c0adf5890 -r 7cf820c46860 src/os/win32/ngx_files.c >>> --- a/src/os/win32/ngx_files.c Thu Jan 12 21:41:39 2023 +0300 >>> +++ b/src/os/win32/ngx_files.c Thu Jan 12 21:41:56 2023 +0300 >>> @@ -428,6 +428,30 @@ ngx_realpath(u_char *path, u_char *resol >>> } >>> >>> >>> +size_t >>> +ngx_getcwd(u_char *buf, size_t size) >>> +{ >>> + u_char *p; >>> + size_t n; >>> + u_short utf16[NGX_MAX_PATH]; >>> + >>> + n = GetCurrentDirectoryW(NGX_MAX_PATH, utf16); >>> + >>> + if (n == 0) { >>> + return 0; >>> + } > > Re-reading the documentation, I tend to think this might worth an > additional test for "(n > NGX_MAX_PATH)" > (https://learn.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-getcurrentdirectory): > > : If the function succeeds, the return value specifies the number > : of characters that are written to the buffer, not including the > : terminating null character. > > : If the function fails, the return value is zero. To get extended > : error information, call GetLastError. > > : If the buffer that is pointed to by lpBuffer is not large > : enough, the return value specifies the required size of the > : buffer, in characters, including the null-terminating character. > > That is, there will be no explicit error (n == 0) if the buffer > will happen to be insufficient. On the other hand, this probably > can happen with long paths enabled (see > https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=registry), > so it might be a good idea to add an explicit handling while here. > Yes, I looked through GetCurrentDirectory return values, but didn't pay an attention on long paths support. Considering them, it may have sense to add such checks. >>> + >>> + p = ngx_utf16_to_utf8(buf, utf16, &size, NULL); >>> + >> >> No error check may result in double-free, first freed >> after (re-)allocation on NGX_EILSEQ, then as below. > > Free on NGX_EILSEQ after reallocation only frees the memory > allocated by the ngx_utf16_to_utf8() itself. The incorrect free() > was before reallocation, and is now fixed. > > Since on errors ngx_utf16_to_utf8() returns NULL, it is matched by > the "if (p != buf)" test, and processed identically to > reallocations. The only questionable part is ngx_free(NULL), > though as per C standard (starting at least with C89; and also > win32 documentation) this is guaranteed to be a nop. > >>> + if (p != buf) { >>> + ngx_free(p); >>> + return 0; >> >> Why return an error if (re-)allocation happened? >> Sizes (calculated in 1-byte units) above NGX_MAX_PATH >> seem perfectly valid with multibyte UTF-8. > > The ngx_getcwd() interface does not provide a way to return a new > pointer. Rather, it is strictly limited to the buffer provided by > the caller, and can only return an error if the buffer is not > large enough. Sure, missed that point. > > One possible improvement here is to explicitly set the errno to a > reasonable value if this happens. This would require an explicit > handling of other errors though. > > Combined with the above: > > diff -r 6705dd675ace src/os/win32/ngx_files.c > --- a/src/os/win32/ngx_files.c Sat Feb 18 16:01:20 2023 +0300 > +++ b/src/os/win32/ngx_files.c Sun Feb 19 03:22:48 2023 +0300 > @@ -442,10 +442,20 @@ ngx_getcwd(u_char *buf, size_t size) > return 0; > } > > + if (n > NGX_MAX_PATH) { > + ngx_set_errno(ERROR_INSUFFICIENT_BUFFER); > + return 0; > + } > + > p = ngx_utf16_to_utf8(buf, utf16, &size, NULL); > > + if (p == NULL) { > + return 0; > + } > + > if (p != buf) { > ngx_free(p); > + ngx_set_errno(ERROR_INSUFFICIENT_BUFFER); > return 0; > } > > > [...] > Thanks for the lengthy explanation. The proposed addition looks good to me. -- Sergey Kandaurov From pluknet at nginx.com Wed Feb 22 16:01:15 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Wed, 22 Feb 2023 20:01:15 +0400 Subject: [PATCH 11 of 12] Win32: fixed ngx_fs_bsize() for symlinks In-Reply-To: References: Message-ID: <5F74999F-228E-43AB-B059-879964530E55@nginx.com> > On 19 Feb 2023, at 21:23, Maxim Dounin wrote: > > Hello! > > On Fri, Feb 17, 2023 at 07:17:02PM +0400, Sergey Kandaurov wrote: > >>> On 13 Jan 2023, at 01:35, Maxim Dounin wrote: >>> >>> # HG changeset patch >>> # User Maxim Dounin >>> # Date 1673549010 -10800 >>> # Thu Jan 12 21:43:30 2023 +0300 >>> # Node ID be7eb9ec28dcbfdfd2e850befc8d051c0e4d46fd >>> # Parent e62c8e9724ba68a698a2c3613edca73fe4e1c4ae >>> Win32: fixed ngx_fs_bsize() for symlinks. >>> >>> Just a drive letter might not correctly represent file system being used, >>> notably when using symlinks (as created by "mklink /d"). As such, instead >>> of calling GetDiskFreeSpace() with just a drive letter, we now always >>> use GetDiskFreeSpace() with full path. >>> >>> diff -r e62c8e9724ba -r be7eb9ec28dc src/os/win32/ngx_files.c >>> --- a/src/os/win32/ngx_files.c Thu Jan 12 21:43:14 2023 +0300 >>> +++ b/src/os/win32/ngx_files.c Thu Jan 12 21:43:30 2023 +0300 >>> @@ -955,14 +955,8 @@ ngx_directio_off(ngx_fd_t fd) >>> size_t >>> ngx_fs_bsize(u_char *name) >>> { >>> - u_char root[4]; >>> u_long sc, bs, nfree, ncl; >>> >>> - if (name[2] == ':') { >>> - ngx_cpystrn(root, name, 4); >>> - name = root; >>> - } >>> - >> >> BTW, I wonder how this condition could be true. >> Specifically, what name should represent in order to match. >> I'm happy that it's leaving though. > > I tend to think that this actually never worked, and the original > intention was to test name[1] instead. > > Updated commit log: > > : Win32: removed attempt to use a drive letter in ngx_fs_bsize(). > : > : Just a drive letter might not correctly represent file system being used, > : notably when using symlinks (as created by "mklink /d"). As such, instead > : of trying to call GetDiskFreeSpace() with just a drive letter, we now always > : use GetDiskFreeSpace() with full path. > : > : Further, it looks like the code to use just a drive letter never worked, > : since it tried to test name[2] instead of name[1] to be ':'. > > [...] > Looks fine, thanks. -- Sergey Kandaurov From mdounin at mdounin.ru Wed Feb 22 19:55:29 2023 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 22 Feb 2023 22:55:29 +0300 Subject: [PATCH 1 of 2] Core: connect() error log message made more verbose In-Reply-To: References: <55553146bd984be7e9e3.1675871646@ssafarly-nb> Message-ID: Hello! On Wed, Feb 22, 2023 at 03:37:51PM +0400, Sergey Kandaurov wrote: > > On 9 Feb 2023, at 12:11, Maxim Dounin wrote: > > > >> # HG changeset patch > >> # User Safar Safarly > >> # Date 1675779866 -10800 > >> # Tue Feb 07 17:24:26 2023 +0300 > >> # Node ID 55553146bd984be7e9e3bbfa851c282feda82d93 > >> # Parent cffaf3f2eec8fd33605c2a37814f5ffc30371989 > >> Core: connect() error log message made more verbose > >> > >> There was a major problem in logs: we could not identify to which servers > >> connect() has failed. Previously log produced: > >> > >> ngx_log_error(NGX_LOG_ALERT, ngx_cycle->log, ngx_socket_errno, > >> "connect() failed"); > >> > >> And now we'll have an address or unix socket in log: > >> > >> ngx_log_error(NGX_LOG_ALERT, ngx_cycle->log, ngx_socket_errno, > >> "connect() to %V failed", &peer->server.name); > >> > >> Message format has chosen to be exact as it is used in ngx_event_connect.c:242 > >> with similar error logging: > >> > >> ngx_log_error(level, c->log, err, "connect() to %V failed", > >> pc->name); > >> > >> So everywhere connect() could fail we'd get a uniform and verbose error message > >> in log. > > > > Thanks for the patch. > > > > Indeed, it might be non-trivial to find out what goes wrong when > > no context is provided in error messages, and logging to syslog > > fails to provide any. > > > > Please see comments below. > > > > [..] > >> > >> goto failed; > >> } > >> diff -r cffaf3f2eec8 -r 55553146bd98 src/core/ngx_syslog.c > >> --- a/src/core/ngx_syslog.c Thu Feb 02 23:38:48 2023 +0300 > >> +++ b/src/core/ngx_syslog.c Tue Feb 07 17:24:26 2023 +0300 > >> @@ -337,7 +337,7 @@ > >> > >> if (connect(fd, peer->server.sockaddr, peer->server.socklen) == -1) { > >> ngx_log_error(NGX_LOG_ALERT, ngx_cycle->log, ngx_socket_errno, > >> - "connect() failed"); > >> + "connect() to %V failed", &peer->server.name); > >> goto failed; > >> } > > > > For syslog logging, I tend to think that this isn't really enough. > > For example, consider send() errors which also might happen when > > logging to syslog. Right now these are logged as: > > > > 2023/02/09 07:29:11 [alert] 23230#100162: send() failed (49: Can't assign requested address) > > 2023/02/09 07:29:11 [alert] 23230#100162: send() failed (49: Can't assign requested address) > > > > Clearly no better than connect() errors: > > > > 2023/02/09 07:29:11 [alert] 23230#100162: connect() failed (2: No such file or directory) > > > > A better fix would be to provide a log handler, similarly to > > ngx_resolver_log_error() mentioned above. With a simple log > > handler the above messages will instead look like: > > > > 2023/02/09 07:48:11 [alert] 23699#100145: send() failed (49: Can't assign requested address) while logging to syslog, server: 127.0.0.2:514 > > 2023/02/09 07:48:11 [alert] 23699#100145: send() failed (49: Can't assign requested address) while logging to syslog, server: 127.0.0.2:514 > > 2023/02/09 07:48:11 [alert] 23699#100145: connect() failed (2: No such file or directory) while logging to syslog, server: unix:/log.socket > > > > Patch: > > > > # HG changeset patch > > # User Maxim Dounin > > # Date 1675929813 -10800 > > # Thu Feb 09 11:03:33 2023 +0300 > > # Node ID 6b662855bf77c678a3954939aefe3fd4b4af4c70 > > # Parent cffaf3f2eec8fd33605c2a37814f5ffc30371989 > > Syslog: introduced error log handler. > > > > This ensures that errors which happen during logging to syslog are logged > > with proper context, such as "while logging to syslog" and the server name. > > > > Prodded by Safar Safarly. > > > > diff --git a/src/core/ngx_syslog.c b/src/core/ngx_syslog.c > > --- a/src/core/ngx_syslog.c > > +++ b/src/core/ngx_syslog.c > > @@ -18,6 +18,7 @@ > > static char *ngx_syslog_parse_args(ngx_conf_t *cf, ngx_syslog_peer_t *peer); > > static ngx_int_t ngx_syslog_init_peer(ngx_syslog_peer_t *peer); > > static void ngx_syslog_cleanup(void *data); > > +static u_char *ngx_syslog_log_error(ngx_log_t *log, u_char *buf, size_t len); > > > > > > static char *facilities[] = { > > @@ -66,6 +67,8 @@ ngx_syslog_process_conf(ngx_conf_t *cf, > > ngx_str_set(&peer->tag, "nginx"); > > } > > > > + peer->logp = &cf->cycle->new_log; > > + > > You may want to reflect this change in the description. > That's, this now follows other error logging by using log from > the configuration that is going to be applied (cycle->new_log): > > src/core/ngx_log.c: dummy = &cf->cycle->new_log; > src/mail/ngx_mail_core_module.c: conf->error_log = &cf->cycle->new_log; > src/stream/ngx_stream_core_module.c: conf->error_log = &cf->cycle->new_log; > src/http/ngx_http_core_module.c: conf->error_log = &cf->cycle->new_log; > src/core/ngx_resolver.c: r->log = &cf->cycle->new_log; > src/core/ngx_cycle.c: cycle->log = &cycle->new_log; > > Previously, before configuration was read, it used init_cycle configuration, > that's builtin error_log on the NGX_LOG_NOTICE level, which means that its > own ngx_send debug appeared only after the configuration was read, and other > syslog error logging was limited to the builtin error log. The main goal of introduction of peer->logp is to avoid re-initializing peer->log on each ngx_syslog_send() call. Previous code used to initialize peer->conn.log on each call, though now there are more things to initialize, and doing this on each call makes the issue obvious. But yes, the resulting code is consistent with other uses and matches how logging is expected to be used: when something is used in a context of a configuration, it uses logging from the configuration. A side note: it looks like ngx_syslog_add_header() uses ngx_cycle->hostname. During initial startup this will use init_cycle->hostname, which is empty. Looking at all this again I tend to think it might be a good idea to ditch ngx_cycle usage in a separate patch (both for ngx_cycle->log and ngx_cycle->hostname), and implement proper logging context on top of it. Patches below. > > peer->conn.fd = (ngx_socket_t) -1; > > > > peer->conn.read = &ngx_syslog_dummy_event; > > @@ -286,15 +289,19 @@ ngx_syslog_send(ngx_syslog_peer_t *peer, > > { > > ssize_t n; > > > > + if (peer->log.handler == NULL) { > > + peer->log = *peer->logp; > > + peer->log.handler = ngx_syslog_log_error; > > + peer->log.data = peer; > > + peer->log.action = "logging to syslog"; > > + } > > + > > if (peer->conn.fd == (ngx_socket_t) -1) { > > if (ngx_syslog_init_peer(peer) != NGX_OK) { > > return NGX_ERROR; > > } > > } > > > > - /* log syslog socket events with valid log */ > > - peer->conn.log = ngx_cycle->log; > > - > > if (ngx_send) { > > n = ngx_send(&peer->conn, buf, len); > > > > One conversion to &peer->log is missing in the ngx_send error handling: > > diff --git a/src/core/ngx_syslog.c b/src/core/ngx_syslog.c > --- a/src/core/ngx_syslog.c > +++ b/src/core/ngx_syslog.c > @@ -313,7 +313,7 @@ ngx_syslog_send(ngx_syslog_peer_t *peer, > if (n == NGX_ERROR) { > > if (ngx_close_socket(peer->conn.fd) == -1) { > - ngx_log_error(NGX_LOG_ALERT, ngx_cycle->log, ngx_socket_errno, > + ngx_log_error(NGX_LOG_ALERT, &peer->log, ngx_socket_errno, > ngx_close_socket_n " failed"); > } > > > Other than that it looks good. Applied, thanks. Updated patches below. # HG changeset patch # User Maxim Dounin # Date 1677095349 -10800 # Wed Feb 22 22:49:09 2023 +0300 # Node ID a964a013031dabbdd05fb0637de496640070c416 # Parent cffaf3f2eec8fd33605c2a37814f5ffc30371989 Syslog: removed usage of ngx_cycle->log and ngx_cycle->hostname. During initial startup the ngx_cycle->hostname is not available, and previously this resulted in incorrect logging. Instead, hostname from the configuration being parsed is now preserved in the syslog peer structure and then used during logging. Similarly, ngx_cycle->log might not match the configuration where the syslog peer is defined if the configuration is not yet fully applied, and previously this resulted in unexpected logging of syslog errors and debug information. Instead, cf->cycle->new_log is now referenced in the syslog peer structure and used for logging, similarly to how it is done in other modules. diff --git a/src/core/ngx_syslog.c b/src/core/ngx_syslog.c --- a/src/core/ngx_syslog.c +++ b/src/core/ngx_syslog.c @@ -66,6 +66,9 @@ ngx_syslog_process_conf(ngx_conf_t *cf, ngx_str_set(&peer->tag, "nginx"); } + peer->hostname = &cf->cycle->hostname; + peer->log = &cf->cycle->new_log; + peer->conn.fd = (ngx_socket_t) -1; peer->conn.read = &ngx_syslog_dummy_event; @@ -243,7 +246,7 @@ ngx_syslog_add_header(ngx_syslog_peer_t } return ngx_sprintf(buf, "<%ui>%V %V %V: ", pri, &ngx_cached_syslog_time, - &ngx_cycle->hostname, &peer->tag); + peer->hostname, &peer->tag); } @@ -292,9 +295,6 @@ ngx_syslog_send(ngx_syslog_peer_t *peer, } } - /* log syslog socket events with valid log */ - peer->conn.log = ngx_cycle->log; - if (ngx_send) { n = ngx_send(&peer->conn, buf, len); @@ -306,7 +306,7 @@ ngx_syslog_send(ngx_syslog_peer_t *peer, if (n == NGX_ERROR) { if (ngx_close_socket(peer->conn.fd) == -1) { - ngx_log_error(NGX_LOG_ALERT, ngx_cycle->log, ngx_socket_errno, + ngx_log_error(NGX_LOG_ALERT, peer->log, ngx_socket_errno, ngx_close_socket_n " failed"); } @@ -324,24 +324,25 @@ ngx_syslog_init_peer(ngx_syslog_peer_t * fd = ngx_socket(peer->server.sockaddr->sa_family, SOCK_DGRAM, 0); if (fd == (ngx_socket_t) -1) { - ngx_log_error(NGX_LOG_ALERT, ngx_cycle->log, ngx_socket_errno, + ngx_log_error(NGX_LOG_ALERT, peer->log, ngx_socket_errno, ngx_socket_n " failed"); return NGX_ERROR; } if (ngx_nonblocking(fd) == -1) { - ngx_log_error(NGX_LOG_ALERT, ngx_cycle->log, ngx_socket_errno, + ngx_log_error(NGX_LOG_ALERT, peer->log, ngx_socket_errno, ngx_nonblocking_n " failed"); goto failed; } if (connect(fd, peer->server.sockaddr, peer->server.socklen) == -1) { - ngx_log_error(NGX_LOG_ALERT, ngx_cycle->log, ngx_socket_errno, + ngx_log_error(NGX_LOG_ALERT, peer->log, ngx_socket_errno, "connect() failed"); goto failed; } peer->conn.fd = fd; + peer->conn.log = peer->log; /* UDP sockets are always ready to write */ peer->conn.write->ready = 1; @@ -351,7 +352,7 @@ ngx_syslog_init_peer(ngx_syslog_peer_t * failed: if (ngx_close_socket(fd) == -1) { - ngx_log_error(NGX_LOG_ALERT, ngx_cycle->log, ngx_socket_errno, + ngx_log_error(NGX_LOG_ALERT, peer->log, ngx_socket_errno, ngx_close_socket_n " failed"); } @@ -372,7 +373,7 @@ ngx_syslog_cleanup(void *data) } if (ngx_close_socket(peer->conn.fd) == -1) { - ngx_log_error(NGX_LOG_ALERT, ngx_cycle->log, ngx_socket_errno, + ngx_log_error(NGX_LOG_ALERT, peer->log, ngx_socket_errno, ngx_close_socket_n " failed"); } } diff --git a/src/core/ngx_syslog.h b/src/core/ngx_syslog.h --- a/src/core/ngx_syslog.h +++ b/src/core/ngx_syslog.h @@ -9,14 +9,19 @@ typedef struct { - ngx_uint_t facility; - ngx_uint_t severity; - ngx_str_t tag; + ngx_uint_t facility; + ngx_uint_t severity; + ngx_str_t tag; + + ngx_str_t *hostname; - ngx_addr_t server; - ngx_connection_t conn; - unsigned busy:1; - unsigned nohostname:1; + ngx_addr_t server; + ngx_connection_t conn; + + ngx_log_t *log; + + unsigned busy:1; + unsigned nohostname:1; } ngx_syslog_peer_t; # HG changeset patch # User Maxim Dounin # Date 1677095351 -10800 # Wed Feb 22 22:49:11 2023 +0300 # Node ID 8f7c464c54e0b18bdb4d505866755cd600fac9fb # Parent a964a013031dabbdd05fb0637de496640070c416 Syslog: introduced error log handler. This ensures that errors which happen during logging to syslog are logged with proper context, such as "while logging to syslog" and the server name. Prodded by Safar Safarly. diff --git a/src/core/ngx_syslog.c b/src/core/ngx_syslog.c --- a/src/core/ngx_syslog.c +++ b/src/core/ngx_syslog.c @@ -18,6 +18,7 @@ static char *ngx_syslog_parse_args(ngx_conf_t *cf, ngx_syslog_peer_t *peer); static ngx_int_t ngx_syslog_init_peer(ngx_syslog_peer_t *peer); static void ngx_syslog_cleanup(void *data); +static u_char *ngx_syslog_log_error(ngx_log_t *log, u_char *buf, size_t len); static char *facilities[] = { @@ -67,7 +68,7 @@ ngx_syslog_process_conf(ngx_conf_t *cf, } peer->hostname = &cf->cycle->hostname; - peer->log = &cf->cycle->new_log; + peer->logp = &cf->cycle->new_log; peer->conn.fd = (ngx_socket_t) -1; @@ -289,6 +290,13 @@ ngx_syslog_send(ngx_syslog_peer_t *peer, { ssize_t n; + if (peer->log.handler == NULL) { + peer->log = *peer->logp; + peer->log.handler = ngx_syslog_log_error; + peer->log.data = peer; + peer->log.action = "logging to syslog"; + } + if (peer->conn.fd == (ngx_socket_t) -1) { if (ngx_syslog_init_peer(peer) != NGX_OK) { return NGX_ERROR; @@ -306,7 +314,7 @@ ngx_syslog_send(ngx_syslog_peer_t *peer, if (n == NGX_ERROR) { if (ngx_close_socket(peer->conn.fd) == -1) { - ngx_log_error(NGX_LOG_ALERT, peer->log, ngx_socket_errno, + ngx_log_error(NGX_LOG_ALERT, &peer->log, ngx_socket_errno, ngx_close_socket_n " failed"); } @@ -324,25 +332,25 @@ ngx_syslog_init_peer(ngx_syslog_peer_t * fd = ngx_socket(peer->server.sockaddr->sa_family, SOCK_DGRAM, 0); if (fd == (ngx_socket_t) -1) { - ngx_log_error(NGX_LOG_ALERT, peer->log, ngx_socket_errno, + ngx_log_error(NGX_LOG_ALERT, &peer->log, ngx_socket_errno, ngx_socket_n " failed"); return NGX_ERROR; } if (ngx_nonblocking(fd) == -1) { - ngx_log_error(NGX_LOG_ALERT, peer->log, ngx_socket_errno, + ngx_log_error(NGX_LOG_ALERT, &peer->log, ngx_socket_errno, ngx_nonblocking_n " failed"); goto failed; } if (connect(fd, peer->server.sockaddr, peer->server.socklen) == -1) { - ngx_log_error(NGX_LOG_ALERT, peer->log, ngx_socket_errno, + ngx_log_error(NGX_LOG_ALERT, &peer->log, ngx_socket_errno, "connect() failed"); goto failed; } peer->conn.fd = fd; - peer->conn.log = peer->log; + peer->conn.log = &peer->log; /* UDP sockets are always ready to write */ peer->conn.write->ready = 1; @@ -352,7 +360,7 @@ ngx_syslog_init_peer(ngx_syslog_peer_t * failed: if (ngx_close_socket(fd) == -1) { - ngx_log_error(NGX_LOG_ALERT, peer->log, ngx_socket_errno, + ngx_log_error(NGX_LOG_ALERT, &peer->log, ngx_socket_errno, ngx_close_socket_n " failed"); } @@ -373,7 +381,30 @@ ngx_syslog_cleanup(void *data) } if (ngx_close_socket(peer->conn.fd) == -1) { - ngx_log_error(NGX_LOG_ALERT, peer->log, ngx_socket_errno, + ngx_log_error(NGX_LOG_ALERT, &peer->log, ngx_socket_errno, ngx_close_socket_n " failed"); } } + + +static u_char * +ngx_syslog_log_error(ngx_log_t *log, u_char *buf, size_t len) +{ + u_char *p; + ngx_syslog_peer_t *peer; + + p = buf; + + if (log->action) { + p = ngx_snprintf(buf, len, " while %s", log->action); + len -= p - buf; + } + + peer = log->data; + + if (peer) { + p = ngx_snprintf(p, len, ", server: %V", &peer->server.name); + } + + return p; +} diff --git a/src/core/ngx_syslog.h b/src/core/ngx_syslog.h --- a/src/core/ngx_syslog.h +++ b/src/core/ngx_syslog.h @@ -18,7 +18,8 @@ typedef struct { ngx_addr_t server; ngx_connection_t conn; - ngx_log_t *log; + ngx_log_t log; + ngx_log_t *logp; unsigned busy:1; unsigned nohostname:1; -- Maxim Dounin http://mdounin.ru/ From mdounin at mdounin.ru Wed Feb 22 21:40:54 2023 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 23 Feb 2023 00:40:54 +0300 Subject: [PATCH] Core: return error when the first byte is above 0xf5 in utf-8 In-Reply-To: References: Message-ID: Hello! On Thu, Feb 23, 2023 at 12:10:27AM +0900, u5h wrote: > # HG changeset patch > # User Yugo Horie > # Date 1677077775 -32400 > # Wed Feb 22 23:56:15 2023 +0900 > # Node ID 1a9487706c6af90baf2ed770db29f689c3850721 > # Parent cffaf3f2eec8fd33605c2a37814f5ffc30371989 > core: return error when the first byte is above 0xf5 in utf-8 > > * see https://datatracker.ietf.org/doc/html/rfc3629#section-4 > Thanks for catching, this needs to be addressed. The existing code accepts invalid UTF-8 sequences with the first byte being 11111xxx (>= 0xf8), and interprets them as if the first byte was 11110xxx (as a 4-byte sequence). This is going to misinterpret future UTF-8 if it will be ever expanded to 5-byte sequences (unlikely though), and might cause other problems (including security ones, as the RFC rightfully warns, though also unlikely in this particular case). A more verbose commit log might be beneficial here, as well as the one matching nginx style for commit logs. See http://nginx.org/en/docs/contributing_changes.html for some basic tips. > diff -r cffaf3f2eec8 -r 1a9487706c6a src/core/ngx_string.c > --- a/src/core/ngx_string.c Thu Feb 02 23:38:48 2023 +0300 > +++ b/src/core/ngx_string.c Wed Feb 22 23:56:15 2023 +0900 > @@ -1364,7 +1364,7 @@ > > u = **p; > > - if (u >= 0xf0) { > + if (u < 0xf5 && u >= 0xf0) { > > u &= 0x07; > valid = 0xffff; The suggested change doesn't look correct to me. In particular, the "u == 0xf5" (as well as 0xf6 and 0xf7) case is valid in terms of the codepoint decoding, though resulting value will be in the invalid range as per the comment above the function. So it does not really need any special handling, except may be to simplify things. The interesting part here is "u >= 0xf8" (which is complimentary to "u &= 0x07" being used in "if"). Further, as the condition is written, "u >= 0xf5" is now excluded from the first "if", but will be matched by the "if (u >= 0xe0)" below, and misinterpreted as a 3-byte sequence. This looks worse than the existing behaviour. I would rather consider something like: diff --git a/src/core/ngx_string.c b/src/core/ngx_string.c --- a/src/core/ngx_string.c +++ b/src/core/ngx_string.c @@ -1364,7 +1364,12 @@ ngx_utf8_decode(u_char **p, size_t n) u = **p; - if (u >= 0xf0) { + if (u >= 0xf8) { + + (*p)++; + return 0xffffffff; + + } else if (u >= 0xf0) { u &= 0x07; valid = 0xffff; -- Maxim Dounin http://mdounin.ru/ From u5.horie at gmail.com Thu Feb 23 00:24:52 2023 From: u5.horie at gmail.com (u5h) Date: Thu, 23 Feb 2023 09:24:52 +0900 Subject: [PATCH] Core: return error when the first byte is above 0xf5 in utf-8 In-Reply-To: References: Message-ID: Thanks reviewing! I agree with your early return strategy and I would reconsider that condition below. # HG changeset patch # User Yugo Horie # Date 1677107390 -32400 # Thu Feb 23 08:09:50 2023 +0900 # Node ID a3ca45d39fcfd32ca92a6bd25ec18b6359b90f1a # Parent f4653576ffcd286bed7229e18ee30ec3c713b4de Core: restrict the rule of utf-8 decode. The first byte being above 0xf8 which is referred to 5byte over length older utf-8 becomes invalid. Even the range of the first byte from 0xf5 to 0xf7 is valid in the term of the codepoint decoding. See https://datatracker.ietf.org/doc/html/rfc3629#section-4. diff -r f4653576ffcd -r a3ca45d39fcf src/core/ngx_string.c --- a/src/core/ngx_string.c Thu Feb 23 07:56:44 2023 +0900 +++ b/src/core/ngx_string.c Thu Feb 23 08:09:50 2023 +0900 @@ -1363,8 +1363,12 @@ uint32_t u, i, valid; u = **p; - - if (u >= 0xf0) { + if (u >= 0xf8) { + + (*p)++; + return 0xffffffff; + + } else if (u >= 0xf0) { u &= 0x07; valid = 0xffff; On Thu, Feb 23, 2023 at 6:41 Maxim Dounin wrote: > Hello! > > On Thu, Feb 23, 2023 at 12:10:27AM +0900, u5h wrote: > > > # HG changeset patch > > # User Yugo Horie > > # Date 1677077775 -32400 > > # Wed Feb 22 23:56:15 2023 +0900 > > # Node ID 1a9487706c6af90baf2ed770db29f689c3850721 > > # Parent cffaf3f2eec8fd33605c2a37814f5ffc30371989 > > core: return error when the first byte is above 0xf5 in utf-8 > > > > * see https://datatracker.ietf.org/doc/html/rfc3629#section-4 > > > > Thanks for catching, this needs to be addressed. > > The existing code accepts invalid UTF-8 sequences with the first > byte being 11111xxx (>= 0xf8), and interprets them as if the first > byte was 11110xxx (as a 4-byte sequence). This is going to > misinterpret future UTF-8 if it will be ever expanded to 5-byte > sequences (unlikely though), and might cause other problems > (including security ones, as the RFC rightfully warns, though also > unlikely in this particular case). > > A more verbose commit log might be beneficial here, as well as > the one matching nginx style for commit logs. See > http://nginx.org/en/docs/contributing_changes.html for some basic > tips. > > > diff -r cffaf3f2eec8 -r 1a9487706c6a src/core/ngx_string.c > > --- a/src/core/ngx_string.c Thu Feb 02 23:38:48 2023 +0300 > > +++ b/src/core/ngx_string.c Wed Feb 22 23:56:15 2023 +0900 > > @@ -1364,7 +1364,7 @@ > > > > u = **p; > > > > - if (u >= 0xf0) { > > + if (u < 0xf5 && u >= 0xf0) { > > > > u &= 0x07; > > valid = 0xffff; > > The suggested change doesn't look correct to me. > > In particular, the "u == 0xf5" (as well as 0xf6 and 0xf7) case is > valid in terms of the codepoint decoding, though resulting value > will be in the invalid range as per the comment above the > function. So it does not really need any special handling, except > may be to simplify things. The interesting part here is "u >= 0xf8" > (which is complimentary to "u &= 0x07" being used in "if"). > > Further, as the condition is written, "u >= 0xf5" is now excluded > from the first "if", but will be matched by the "if (u >= 0xe0)" > below, and misinterpreted as a 3-byte sequence. This looks worse > than the existing behaviour. > > I would rather consider something like: > > diff --git a/src/core/ngx_string.c b/src/core/ngx_string.c > --- a/src/core/ngx_string.c > +++ b/src/core/ngx_string.c > @@ -1364,7 +1364,12 @@ ngx_utf8_decode(u_char **p, size_t n) > > u = **p; > > - if (u >= 0xf0) { > + if (u >= 0xf8) { > + > + (*p)++; > + return 0xffffffff; > + > + } else if (u >= 0xf0) { > > u &= 0x07; > valid = 0xffff; > > > -- > Maxim Dounin > http://mdounin.ru/ > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > https://mailman.nginx.org/mailman/listinfo/nginx-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xeioex at nginx.com Thu Feb 23 04:13:42 2023 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Thu, 23 Feb 2023 04:13:42 +0000 Subject: [njs] Added njs_vm_value_to_c_string(). Message-ID: details: https://hg.nginx.org/njs/rev/2af586015b65 branches: changeset: 2048:2af586015b65 user: Dmitry Volyntsev date: Fri Feb 17 22:38:25 2023 -0800 description: Added njs_vm_value_to_c_string(). diffstat: src/njs.h | 7 +++++++ src/njs_string.c | 38 -------------------------------------- src/njs_string.h | 1 - src/njs_vm.c | 40 ++++++++++++++++++++++++++++++++++++++++ 4 files changed, 47 insertions(+), 39 deletions(-) diffs (126 lines): diff -r 29fbf8f85c09 -r 2af586015b65 src/njs.h --- a/src/njs.h Thu Feb 09 18:34:51 2023 -0800 +++ b/src/njs.h Fri Feb 17 22:38:25 2023 -0800 @@ -447,6 +447,13 @@ NJS_EXPORT njs_int_t njs_vm_value_to_str */ NJS_EXPORT njs_int_t njs_vm_value_string(njs_vm_t *vm, njs_str_t *dst, njs_value_t *src); +/* + * If string value is null-terminated the corresponding C string + * is returned as is, otherwise the new copy is allocated with + * the terminating zero byte. + */ +NJS_EXPORT const char *njs_vm_value_to_c_string(njs_vm_t *vm, + njs_value_t *value); NJS_EXPORT njs_int_t njs_vm_retval_string(njs_vm_t *vm, njs_str_t *dst); NJS_EXPORT njs_int_t njs_vm_value_dump(njs_vm_t *vm, njs_str_t *dst, diff -r 29fbf8f85c09 -r 2af586015b65 src/njs_string.c --- a/src/njs_string.c Thu Feb 09 18:34:51 2023 -0800 +++ b/src/njs_string.c Fri Feb 17 22:38:25 2023 -0800 @@ -4075,44 +4075,6 @@ njs_string_to_index(const njs_value_t *v } -/* - * If string value is null-terminated the corresponding C string - * is returned as is, otherwise the new copy is allocated with - * the terminating zero byte. - */ -const char * -njs_string_to_c_string(njs_vm_t *vm, njs_value_t *value) -{ - u_char *p, *data, *start; - size_t size; - - if (value->short_string.size != NJS_STRING_LONG) { - start = value->short_string.start; - size = value->short_string.size; - - if (size < NJS_STRING_SHORT) { - start[size] = '\0'; - return (const char *) start; - } - - } else { - start = value->long_string.data->start; - size = value->long_string.size; - } - - data = njs_mp_alloc(vm->mem_pool, size + 1); - if (njs_slow_path(data == NULL)) { - njs_memory_error(vm); - return NULL; - } - - p = njs_cpymem(data, start, size); - *p++ = '\0'; - - return (const char *) data; -} - - static const njs_object_prop_t njs_string_prototype_properties[] = { NJS_DECLARE_PROP_HANDLER("__proto__", njs_primitive_prototype_get_proto, diff -r 29fbf8f85c09 -r 2af586015b65 src/njs_string.h --- a/src/njs_string.h Thu Feb 09 18:34:51 2023 -0800 +++ b/src/njs_string.h Fri Feb 17 22:38:25 2023 -0800 @@ -248,7 +248,6 @@ const u_char *njs_string_offset(const u_ uint32_t njs_string_index(njs_string_prop_t *string, uint32_t offset); void njs_string_offset_map_init(const u_char *start, size_t size); double njs_string_to_index(const njs_value_t *value); -const char *njs_string_to_c_string(njs_vm_t *vm, njs_value_t *value); njs_int_t njs_string_encode_uri(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, njs_index_t component); njs_int_t njs_string_decode_uri(njs_vm_t *vm, njs_value_t *args, diff -r 29fbf8f85c09 -r 2af586015b65 src/njs_vm.c --- a/src/njs_vm.c Thu Feb 09 18:34:51 2023 -0800 +++ b/src/njs_vm.c Fri Feb 17 22:38:25 2023 -0800 @@ -1257,6 +1257,46 @@ njs_vm_value_to_string(njs_vm_t *vm, njs } +/* + * If string value is null-terminated the corresponding C string + * is returned as is, otherwise the new copy is allocated with + * the terminating zero byte. + */ +const char * +njs_vm_value_to_c_string(njs_vm_t *vm, njs_value_t *value) +{ + u_char *p, *data, *start; + size_t size; + + njs_assert(njs_is_string(value)); + + if (value->short_string.size != NJS_STRING_LONG) { + start = value->short_string.start; + size = value->short_string.size; + + if (size < NJS_STRING_SHORT) { + start[size] = '\0'; + return (const char *) start; + } + + } else { + start = value->long_string.data->start; + size = value->long_string.size; + } + + data = njs_mp_alloc(vm->mem_pool, size + njs_length("\0")); + if (njs_slow_path(data == NULL)) { + njs_memory_error(vm); + return NULL; + } + + p = njs_cpymem(data, start, size); + *p++ = '\0'; + + return (const char *) data; +} + + njs_int_t njs_vm_value_to_bytes(njs_vm_t *vm, njs_str_t *dst, njs_value_t *src) { From xeioex at nginx.com Thu Feb 23 04:13:43 2023 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Thu, 23 Feb 2023 04:13:43 +0000 Subject: [njs] XML: added XMLNode API to modify XML documents. Message-ID: details: https://hg.nginx.org/njs/rev/3891f338e2c9 branches: changeset: 2049:3891f338e2c9 user: Dmitry Volyntsev date: Wed Feb 22 19:13:08 2023 -0800 description: XML: added XMLNode API to modify XML documents. - delete node.$attr$a is a shorthand syntax for node.removeAttribute('a') - node.removeAllAttributes() removes all attributes of the node. - node.removeAttribute(attr_name) removes attribute named attr_name. - node.setAttribute(attr_name, value || null) sets a value for an attr_name. When value is null attribute named attr_name is deleted. - node.$attr$a = 'xxx' is a shorthand syntax for node.setAttribute('a', 'xxx'); - The methods and operations below make copy-on-write changes to the original XML parsed document. For example: var doc = xml.parse(); var b = doc.$root.a.b; doc.$root.removeAllChildren(); console.log((new TextDecoder()).decode(xml.c14n(doc))); /* */ console.log(b); /* XMLNode {$name:'b'} */ console.log(b.$parent); /* XMLNode {$name:'a',$tags:[XMLNode {$name:'b'}] */ "b" is valid after removeAllChildren() call, but is not a part of the document tree anymore. - node.addChild(nd) adds XMLNode as a child to node. nd recursively copied before adding to the node. - node.removeChildren(tag_name?) removes all the children tags named tag_name. If tag_name is absent all children tags are removed. - node.removeText() removes the node's text value. - node.setText(string || null) sets a text value for the node. When value is null the node's text is deleted. - node.$tags = [node1, node2, ..] is a shorthand syntax for node.removeChildren(); node.addChild(node1); node.addChild(node2) - node.$text = 'xxx' is a shorthand syntax for node.setText('xxx'); In addition the following method were added: - xml.serialize() is the same as xml.c14n() - xml.serializeToString() is the same as xml.c14n() except it returns the result as string. Example: const xml = require("xml"); let data = `ToveJani`; let doc = xml.parse(data); doc.$root.to.$attr$b = 'bar2'; doc.$root.to.setAttribute('c', 'baz'); delete doc.$root.to.$attr$a; console.log(xml.serializeToString(doc.$root.to)) /* 'Tove' */ doc.$root.to.removeAllAttributes(); doc.$root.from.$text = 'Jani2'; console.log(xml.serializeToString(doc)) /* 'ToveJani2' */ doc.$root.to.$tags = [xml.parse(``), xml.parse(``)]; doc.$root.to.addChild(xml.parse(``)); console.log(xml.serializeToString(doc.$root.to)) /* '' */ doc.$root.to.removeChildren('a'); console.log(xml.serializeToString(doc.$root.to)) /* '' */ diffstat: external/njs_xml_module.c | 1064 ++++++++++++++++++++++++++++++++++++++------ src/test/njs_unit_test.c | 203 ++++++++- test/xml/saml_verify.t.js | 105 +++- ts/njs_modules/xml.d.ts | 42 +- 4 files changed, 1221 insertions(+), 193 deletions(-) diffs (truncated from 1788 to 1000 lines): diff -r 2af586015b65 -r 3891f338e2c9 external/njs_xml_module.c --- a/external/njs_xml_module.c Fri Feb 17 22:38:25 2023 -0800 +++ b/external/njs_xml_module.c Wed Feb 22 19:13:08 2023 -0800 @@ -39,12 +39,11 @@ struct njs_xml_nset_s { static njs_int_t njs_xml_ext_parse(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, njs_index_t unused); static njs_int_t njs_xml_ext_canonicalization(njs_vm_t *vm, njs_value_t *args, - njs_uint_t nargs, njs_index_t unused); + njs_uint_t nargs, njs_index_t magic); static njs_int_t njs_xml_doc_ext_prop_keys(njs_vm_t *vm, njs_value_t *value, njs_value_t *keys); static njs_int_t njs_xml_doc_ext_root(njs_vm_t *vm, njs_object_prop_t *prop, njs_value_t *value, njs_value_t *unused, njs_value_t *retval); -static void njs_xml_doc_cleanup(void *data); static njs_int_t njs_xml_node_ext_prop_keys(njs_vm_t *vm, njs_value_t *value, njs_value_t *keys); static njs_int_t njs_xml_node_ext_prop_handler(njs_vm_t *vm, @@ -55,6 +54,8 @@ static njs_int_t njs_xml_attr_ext_prop_k static njs_int_t njs_xml_attr_ext_prop_handler(njs_vm_t *vm, njs_object_prop_t *prop, njs_value_t *value, njs_value_t *unused, njs_value_t *retval); +static njs_int_t njs_xml_node_ext_add_child(njs_vm_t *vm, njs_value_t *args, + njs_uint_t nargs, njs_index_t unused); static njs_int_t njs_xml_node_ext_attrs(njs_vm_t *vm, njs_object_prop_t *prop, njs_value_t *value, njs_value_t *setval, njs_value_t *retval); static njs_int_t njs_xml_node_ext_name(njs_vm_t *vm, njs_object_prop_t *prop, @@ -63,11 +64,44 @@ static njs_int_t njs_xml_node_ext_ns(njs njs_value_t *value, njs_value_t *setval, njs_value_t *retval); static njs_int_t njs_xml_node_ext_parent(njs_vm_t *vm, njs_object_prop_t *prop, njs_value_t *value, njs_value_t *setval, njs_value_t *retval); +static njs_int_t njs_xml_node_ext_remove_all_attributes(njs_vm_t *vm, + njs_value_t *args, njs_uint_t nargs, njs_index_t unused); +static njs_int_t njs_xml_node_ext_remove_attribute(njs_vm_t *vm, + njs_value_t *args, njs_uint_t nargs, njs_index_t unused); +static njs_int_t njs_xml_node_ext_remove_children(njs_vm_t *vm, + njs_value_t *args, njs_uint_t nargs, njs_index_t unused); +static njs_int_t njs_xml_node_ext_remove_text(njs_vm_t *vm, + njs_value_t *args, njs_uint_t nargs, njs_index_t unused); +static njs_int_t njs_xml_node_ext_set_attribute(njs_vm_t *vm, + njs_value_t *args, njs_uint_t nargs, njs_index_t unused); +static njs_int_t njs_xml_node_ext_set_text(njs_vm_t *vm, + njs_value_t *args, njs_uint_t nargs, njs_index_t unused); static njs_int_t njs_xml_node_ext_tags(njs_vm_t *vm, njs_object_prop_t *prop, njs_value_t *value, njs_value_t *setval, njs_value_t *retval); static njs_int_t njs_xml_node_ext_text(njs_vm_t *vm, njs_object_prop_t *prop, njs_value_t *value, njs_value_t *setval, njs_value_t *retval); +static njs_int_t njs_xml_node_attr_handler(njs_vm_t *vm, xmlNode *current, + njs_str_t *name, njs_value_t *setval, njs_value_t *retval); +static njs_int_t njs_xml_node_tag_remove(njs_vm_t *vm, xmlNode *current, + njs_str_t *name); +static njs_int_t njs_xml_node_tag_handler(njs_vm_t *vm, xmlNode *current, + njs_str_t *name, njs_value_t *setval, njs_value_t *retval); +static njs_int_t njs_xml_node_tags_handler(njs_vm_t *vm, xmlNode *current, + njs_str_t *name, njs_value_t *setval, njs_value_t *retval); + +static xmlNode *njs_xml_external_node(njs_vm_t *vm, njs_value_t *value); +static njs_int_t njs_xml_str_to_c_string(njs_vm_t *vm, njs_str_t *str, + u_char *dst, size_t size); +static const u_char *njs_xml_value_to_c_string(njs_vm_t *vm, njs_value_t *value, + u_char *dst, size_t size); +static njs_int_t njs_xml_encode_special_chars(njs_vm_t *vm, njs_str_t *src, + njs_str_t *out); +static njs_int_t njs_xml_replace_node(njs_vm_t *vm, xmlNode *old, + xmlNode *current); +static void njs_xml_node_cleanup(void *data); +static void njs_xml_doc_cleanup(void *data); + static njs_xml_nset_t *njs_xml_nset_create(njs_vm_t *vm, xmlDoc *doc, xmlNode *current, njs_xml_nset_type_t type); static njs_xml_nset_t *njs_xml_nset_add(njs_xml_nset_t *nset, @@ -122,6 +156,29 @@ static njs_external_t njs_ext_xml[] = { } }, + { + .flags = NJS_EXTERN_METHOD, + .name.string = njs_str("serialize"), + .writable = 1, + .configurable = 1, + .enumerable = 1, + .u.method = { + .native = njs_xml_ext_canonicalization, + } + }, + + { + .flags = NJS_EXTERN_METHOD, + .name.string = njs_str("serializeToString"), + .writable = 1, + .configurable = 1, + .enumerable = 1, + .u.method = { + .native = njs_xml_ext_canonicalization, + .magic8 = 3, + } + }, + }; @@ -171,12 +228,25 @@ static njs_external_t njs_ext_xml_node[ .flags = NJS_EXTERN_SELF, .u.object = { .enumerable = 1, + .writable = 1, + .configurable = 1, .prop_handler = njs_xml_node_ext_prop_handler, .keys = njs_xml_node_ext_prop_keys, } }, { + .flags = NJS_EXTERN_METHOD, + .name.string = njs_str("addChild"), + .writable = 1, + .configurable = 1, + .enumerable = 1, + .u.method = { + .native = njs_xml_node_ext_add_child, + } + }, + + { .flags = NJS_EXTERN_PROPERTY, .name.string = njs_str("$attrs"), .enumerable = 1, @@ -213,9 +283,77 @@ static njs_external_t njs_ext_xml_node[ }, { + .flags = NJS_EXTERN_METHOD, + .name.string = njs_str("removeAllAttributes"), + .writable = 1, + .configurable = 1, + .enumerable = 1, + .u.method = { + .native = njs_xml_node_ext_remove_all_attributes, + } + }, + + { + .flags = NJS_EXTERN_METHOD, + .name.string = njs_str("removeAttribute"), + .writable = 1, + .configurable = 1, + .enumerable = 1, + .u.method = { + .native = njs_xml_node_ext_remove_attribute, + } + }, + + { + .flags = NJS_EXTERN_METHOD, + .name.string = njs_str("removeChildren"), + .writable = 1, + .configurable = 1, + .enumerable = 1, + .u.method = { + .native = njs_xml_node_ext_remove_children, + } + }, + + { + .flags = NJS_EXTERN_METHOD, + .name.string = njs_str("removeText"), + .writable = 1, + .configurable = 1, + .enumerable = 1, + .u.method = { + .native = njs_xml_node_ext_remove_text, + } + }, + + { + .flags = NJS_EXTERN_METHOD, + .name.string = njs_str("setAttribute"), + .writable = 1, + .configurable = 1, + .enumerable = 1, + .u.method = { + .native = njs_xml_node_ext_set_attribute, + } + }, + + { + .flags = NJS_EXTERN_METHOD, + .name.string = njs_str("setText"), + .writable = 1, + .configurable = 1, + .enumerable = 1, + .u.method = { + .native = njs_xml_node_ext_set_text, + } + }, + + { .flags = NJS_EXTERN_PROPERTY, .name.string = njs_str("$tags"), .enumerable = 1, + .writable = 1, + .configurable = 1, .u.property = { .handler = njs_xml_node_ext_tags, } @@ -225,6 +363,7 @@ static njs_external_t njs_ext_xml_node[ .flags = NJS_EXTERN_PROPERTY, .name.string = njs_str("$text"), .enumerable = 1, + .writable = 1, .u.property = { .handler = njs_xml_node_ext_text, } @@ -410,16 +549,6 @@ njs_xml_doc_ext_root(njs_vm_t *vm, njs_o } -static void -njs_xml_doc_cleanup(void *data) -{ - njs_xml_doc_t *current = data; - - xmlFreeDoc(current->doc); - xmlFreeParserCtxt(current->ctx); -} - - static njs_int_t njs_xml_node_ext_prop_keys(njs_vm_t *vm, njs_value_t *value, njs_value_t *keys) { @@ -515,15 +644,11 @@ njs_xml_node_ext_prop_keys(njs_vm_t *vm, static njs_int_t njs_xml_node_ext_prop_handler(njs_vm_t *vm, njs_object_prop_t *prop, - njs_value_t *value, njs_value_t *unused, njs_value_t *retval) + njs_value_t *value, njs_value_t *setval, njs_value_t *retval) { - size_t size; - xmlAttr *attr; - xmlNode *node, *current; - njs_int_t ret; - njs_str_t name; - njs_value_t *push; - const u_char *content; + xmlNode *current; + njs_int_t ret; + njs_str_t name; /* * $tag$foo - the first tag child with the name "foo" @@ -548,108 +673,93 @@ njs_xml_node_ext_prop_handler(njs_vm_t * if (name.length > njs_length("$attr$") && njs_strncmp(&name.start[1], "attr$", njs_length("attr$")) == 0) { - for (attr = current->properties; attr != NULL; attr = attr->next) { - if (attr->type != XML_ATTRIBUTE_NODE) { - continue; - } - - size = njs_strlen(attr->name); + name.length -= njs_length("$attr$"); + name.start += njs_length("$attr$"); - if (name.length != (size + njs_length("$attr$")) - || njs_strncmp(&name.start[njs_length("$attr$")], - attr->name, size) != 0) - { - continue; - } - - content = (const u_char *) attr->children->content; - - return njs_vm_value_string_create(vm, retval, content, - njs_strlen(content)); - } + return njs_xml_node_attr_handler(vm, current, &name, setval, + retval); } if (name.length > njs_length("$tag$") && njs_strncmp(&name.start[1], "tag$", njs_length("tag$")) == 0) { - for (node = current->children; node != NULL; node = node->next) { - if (node->type != XML_ELEMENT_NODE) { - continue; - } - - size = njs_strlen(node->name); + name.length -= njs_length("$tag$"); + name.start += njs_length("$tag$"); - if (name.length != (size + njs_length("$tag$")) - || njs_strncmp(&name.start[njs_length("$tag$")], - node->name, size) != 0) - { - continue; - } - - return njs_vm_external_create(vm, retval, njs_xml_node_proto_id, - node, 0); - } + return njs_xml_node_tag_handler(vm, current, &name, setval, retval); } if (name.length >= njs_length("$tags$") && njs_strncmp(&name.start[1], "tags$", njs_length("tags$")) == 0) { - ret = njs_vm_array_alloc(vm, retval, 2); - if (njs_slow_path(ret != NJS_OK)) { - return NJS_ERROR; - } - - for (node = current->children; node != NULL; node = node->next) { - if (node->type != XML_ELEMENT_NODE) { - continue; - } - - size = njs_strlen(node->name); + name.length -= njs_length("$tags$"); + name.start += njs_length("$tags$"); - if (name.length > njs_length("$tags$") - && (name.length != (size + njs_length("$tags$")) - || njs_strncmp(&name.start[njs_length("$tags$")], - node->name, size) != 0)) - { - continue; - } - - push = njs_vm_array_push(vm, retval); - if (njs_slow_path(push == NULL)) { - return NJS_ERROR; - } - - ret = njs_vm_external_create(vm, push, njs_xml_node_proto_id, - node, 0); - if (njs_slow_path(ret != NJS_OK)) { - return NJS_ERROR; - } - } - - return NJS_OK; + return njs_xml_node_tags_handler(vm, current, &name, setval, + retval); } } - for (node = current->children; node != NULL; node = node->next) { - if (node->type != XML_ELEMENT_NODE) { - continue; - } + return njs_xml_node_tag_handler(vm, current, &name, setval, retval); +} + - size = njs_strlen(node->name); +static njs_int_t +njs_xml_node_ext_add_child(njs_vm_t *vm, njs_value_t *args, + njs_uint_t nargs, njs_index_t unused) +{ + xmlNode *current, *node, *copy; + njs_int_t ret; - if (name.length != size - || njs_strncmp(name.start, node->name, size) != 0) - { - continue; - } + current = njs_vm_external(vm, njs_xml_node_proto_id, njs_argument(args, 0)); + if (njs_slow_path(current == NULL)) { + njs_vm_error(vm, "\"this\" is not a XMLNode object"); + return NJS_ERROR; + } - return njs_vm_external_create(vm, retval, njs_xml_node_proto_id, - node, 0); + copy = xmlDocCopyNode(current, current->doc, 1); + if (njs_slow_path(copy == NULL)) { + njs_vm_error(vm, "xmlDocCopyNode() failed"); + return NJS_ERROR; + } + + node = njs_xml_external_node(vm, njs_arg(args, nargs, 1)); + if (njs_slow_path(node == NULL)) { + njs_vm_error(vm, "node is not a XMLNode object"); + goto error; } - njs_value_undefined_set(retval); + node = xmlDocCopyNode(node, current->doc, 1); + if (njs_slow_path(node == NULL)) { + njs_vm_error(vm, "xmlDocCopyNode() failed"); + goto error; + } + + node = xmlAddChild(copy, node); + if (njs_slow_path(node == NULL)) { + njs_vm_error(vm, "xmlAddChild() failed"); + goto error; + } - return NJS_DECLINED; + ret = xmlReconciliateNs(current->doc, copy); + if (njs_slow_path(ret == -1)) { + njs_vm_error(vm, "xmlReconciliateNs() failed"); + return NJS_ERROR; + } + + njs_value_undefined_set(njs_vm_retval(vm)); + + return njs_xml_replace_node(vm, current, copy); + +error: + + if (node != NULL) { + xmlFreeNode(node); + } + + xmlFreeNode(copy); + + return NJS_ERROR; } @@ -725,12 +835,133 @@ njs_xml_node_ext_parent(njs_vm_t *vm, nj static njs_int_t +njs_xml_node_ext_remove_attribute(njs_vm_t *vm, njs_value_t *args, + njs_uint_t nargs, njs_index_t unused) +{ + return njs_xml_node_ext_set_attribute(vm, args, nargs, 1); +} + + +static njs_int_t +njs_xml_node_ext_remove_all_attributes(njs_vm_t *vm, + njs_value_t *args, njs_uint_t nargs, njs_index_t unused) +{ + xmlNode *current; + + current = njs_vm_external(vm, njs_xml_node_proto_id, njs_argument(args, 0)); + if (njs_slow_path(current == NULL)) { + njs_vm_error(vm, "\"this\" is not a XMLNode object"); + return NJS_ERROR; + } + + if (current->properties != NULL) { + xmlFreePropList(current->properties); + current->properties = NULL; + } + + njs_value_undefined_set(njs_vm_retval(vm)); + + return NJS_OK; +} + + +static njs_int_t +njs_xml_node_ext_remove_children(njs_vm_t *vm, njs_value_t *args, + njs_uint_t nargs, njs_index_t unused) +{ + xmlNode *current, *copy; + njs_str_t name; + njs_value_t *selector; + + current = njs_vm_external(vm, njs_xml_node_proto_id, njs_argument(args, 0)); + if (njs_slow_path(current == NULL)) { + njs_vm_error(vm, "\"this\" is not a XMLNode object"); + return NJS_ERROR; + } + + selector = njs_arg(args, nargs, 1); + + njs_value_undefined_set(njs_vm_retval(vm)); + + if (!njs_value_is_null_or_undefined(selector)) { + if (njs_slow_path(!njs_value_is_string(selector))) { + njs_vm_error(vm, "selector is not a string"); + return NJS_ERROR; + } + + njs_value_string_get(selector, &name); + + return njs_xml_node_tag_remove(vm, current, &name); + } + + /* all. */ + + copy = xmlDocCopyNode(current, current->doc, 1); + if (njs_slow_path(copy == NULL)) { + njs_vm_error(vm, "xmlDocCopyNode() failed"); + return NJS_ERROR; + } + + if (copy->children != NULL) { + xmlFreeNodeList(copy->children); + copy->children = NULL; + } + + return njs_xml_replace_node(vm, current, copy); +} + + +static njs_int_t +njs_xml_node_ext_remove_text(njs_vm_t *vm, njs_value_t *args, + njs_uint_t nargs, njs_index_t unused) +{ + return njs_xml_node_ext_text(vm, NULL, njs_argument(args, 0), NULL, NULL); +} + + +static njs_int_t +njs_xml_node_ext_set_attribute(njs_vm_t *vm, njs_value_t *args, + njs_uint_t nargs, njs_index_t remove) +{ + xmlNode *current; + njs_str_t str; + njs_value_t *name; + + current = njs_vm_external(vm, njs_xml_node_proto_id, njs_argument(args, 0)); + if (njs_slow_path(current == NULL)) { + njs_vm_error(vm, "\"this\" is not a XMLNode object"); + return NJS_ERROR; + } + + name = njs_arg(args, nargs, 1); + + if (njs_slow_path(!njs_value_is_string(name))) { + njs_vm_error(vm, "name is not a string"); + return NJS_ERROR; + } + + njs_value_string_get(name, &str); + + return njs_xml_node_attr_handler(vm, current, &str, njs_arg(args, nargs, 2), + !remove ? njs_vm_retval(vm) : NULL); +} + + +static njs_int_t +njs_xml_node_ext_set_text(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, + njs_index_t unused) +{ + return njs_xml_node_ext_text(vm, NULL, njs_argument(args, 0), + njs_arg(args, nargs, 1), njs_vm_retval(vm)); +} + + +static njs_int_t njs_xml_node_ext_tags(njs_vm_t *vm, njs_object_prop_t *prop, njs_value_t *value, njs_value_t *setval, njs_value_t *retval) { - xmlNode *node, *current; - njs_int_t ret; - njs_value_t *push; + xmlNode *current; + njs_str_t name; current = njs_vm_external(vm, njs_xml_node_proto_id, value); if (njs_slow_path(current == NULL || current->children == NULL)) { @@ -738,38 +969,21 @@ njs_xml_node_ext_tags(njs_vm_t *vm, njs_ return NJS_DECLINED; } - ret = njs_vm_array_alloc(vm, retval, 2); - if (njs_slow_path(ret != NJS_OK)) { - return NJS_ERROR; - } - - for (node = current->children; node != NULL; node = node->next) { - if (node->type != XML_ELEMENT_NODE) { - continue; - } + name.start = NULL; + name.length = 0; - push = njs_vm_array_push(vm, retval); - if (njs_slow_path(push == NULL)) { - return NJS_ERROR; - } - - ret = njs_vm_external_create(vm, push, njs_xml_node_proto_id, node, 0); - if (njs_slow_path(ret != NJS_OK)) { - return NJS_ERROR; - } - } - - return NJS_OK; + return njs_xml_node_tags_handler(vm, current, &name, setval, retval); } static njs_int_t -njs_xml_node_ext_text(njs_vm_t *vm, njs_object_prop_t *prop, njs_value_t *value, - njs_value_t *setval, njs_value_t *retval) +njs_xml_node_ext_text(njs_vm_t *vm, njs_object_prop_t *unused, + njs_value_t *value, njs_value_t *setval, njs_value_t *retval) { - xmlNode *current, *node; + u_char *text; + xmlNode *current, *copy; njs_int_t ret; - njs_chb_t chain; + njs_str_t content, enc; current = njs_vm_external(vm, njs_xml_node_proto_id, value); if (njs_slow_path(current == NULL)) { @@ -777,21 +991,551 @@ njs_xml_node_ext_text(njs_vm_t *vm, njs_ return NJS_DECLINED; } - njs_chb_init(&chain, njs_vm_memory_pool(vm)); + if (retval != NULL && setval == NULL) { + text = xmlNodeGetContent(current); + ret = njs_vm_value_string_create(vm, retval, text, njs_strlen(text)); + + xmlFree(text); + + return ret; + } + + /* set or delete. */ + + enc.start = NULL; + enc.length = 0; + + if (retval != NULL + && (setval != NULL && !njs_value_is_null_or_undefined(setval))) + { + if (njs_slow_path(!njs_value_is_string(setval))) { + njs_vm_error(vm, "setval is not a string"); + return NJS_ERROR; + } + + njs_value_string_get(setval, &content); + + ret = njs_xml_encode_special_chars(vm, &content, &enc); + if (njs_slow_path(ret != NJS_OK)) { + return NJS_ERROR; + } + } + + copy = xmlDocCopyNode(current, current->doc, 1); + if (njs_slow_path(copy == NULL)) { + njs_vm_error(vm, "xmlDocCopyNode() failed"); + return NJS_ERROR; + } + + xmlNodeSetContentLen(copy, enc.start, enc.length); + + if (retval != NULL) { + njs_value_undefined_set(retval); + } + + return njs_xml_replace_node(vm, current, copy); +} + + +static njs_int_t +njs_xml_node_attr_handler(njs_vm_t *vm, xmlNode *current, njs_str_t *name, + njs_value_t *setval, njs_value_t *retval) +{ + size_t size; + njs_int_t ret; + xmlAttr *attr; + const u_char *content, *value; + u_char name_buf[512], value_buf[1024]; + + if (retval != NULL && setval == NULL) { + /* get. */ + + for (attr = current->properties; attr != NULL; attr = attr->next) { + if (attr->type != XML_ATTRIBUTE_NODE) { + continue; + } + + size = njs_strlen(attr->name); + + if (name->length != size + || njs_strncmp(name->start, attr->name, size) != 0) + { + continue; + } + + if (attr->children != NULL + && attr->children->next == NULL + && attr->children->type == XML_TEXT_NODE) + { + content = (const u_char *) attr->children->content; - for (node = current->children; node != NULL; node = node->next) { - if (node->type != XML_TEXT_NODE) { + return njs_vm_value_string_create(vm, retval, content, + njs_strlen(content)); + } + } + + njs_value_undefined_set(retval); + + return NJS_DECLINED; + } + + /* set or delete. */ + + ret = njs_xml_str_to_c_string(vm, name, &name_buf[0], sizeof(name_buf)); + if (njs_slow_path(ret != NJS_OK)) { + return NJS_ERROR; + } + + ret = xmlValidateQName(&name_buf[0], 0); + if (njs_slow_path(ret != 0)) { + njs_vm_error(vm, "attribute name \"%V\" is not valid", name); + return NJS_ERROR; + } + + if (retval == NULL + || (setval != NULL && njs_value_is_null_or_undefined(setval))) + { + /* delete. */ + + attr = xmlHasProp(current, &name_buf[0]); + + if (attr != NULL) { + xmlRemoveProp(attr); + } + + return NJS_OK; + } + + value = njs_xml_value_to_c_string(vm, setval, &value_buf[0], + sizeof(value_buf)); + if (njs_slow_path(value == NULL)) { + return NJS_ERROR; + } + + attr = xmlSetProp(current, &name_buf[0], value); + if (njs_slow_path(attr == NULL)) { + njs_vm_error(vm, "xmlSetProp() failed"); + return NJS_ERROR; + } + + njs_value_undefined_set(retval); + + return NJS_OK; +} + + +static njs_int_t +njs_xml_node_tag_remove(njs_vm_t *vm, xmlNode *current, njs_str_t *name) +{ + size_t size; + xmlNode *node, *next, *copy; + njs_int_t ret; + + copy = xmlDocCopyNode(current, current->doc, 1); + if (njs_slow_path(copy == NULL)) { + njs_vm_error(vm, "xmlDocCopyNode() failed"); + return NJS_ERROR; + } + + for (node = copy->children; node != NULL; node = next) { + next = node->next; + + if (node->type != XML_ELEMENT_NODE) { + continue; + } + + size = njs_strlen(node->name); + + if (name->length != size + || njs_strncmp(name->start, node->name, size) != 0) + { continue; } - njs_chb_append(&chain, node->content, njs_strlen(node->content)); + ret = njs_xml_replace_node(vm, node, NULL); + if (njs_slow_path(ret != NJS_OK)) { + xmlFreeNode(copy); + return NJS_ERROR; + } + } + + return njs_xml_replace_node(vm, current, copy); +} + + +static njs_int_t +njs_xml_node_tag_handler(njs_vm_t *vm, xmlNode *current, njs_str_t *name, + njs_value_t *setval, njs_value_t *retval) +{ + size_t size; + xmlNode *node; + + if (retval != NULL && setval == NULL) { + + /* get. */ + + for (node = current->children; node != NULL; node = node->next) { + if (node->type != XML_ELEMENT_NODE) { + continue; + } + + size = njs_strlen(node->name); + + if (name->length != size + || njs_strncmp(name->start, node->name, size) != 0) + { + continue; + } + + return njs_vm_external_create(vm, retval, njs_xml_node_proto_id, + node, 0); + } + + njs_value_undefined_set(retval); + + return NJS_DECLINED; + } + + if (retval != NULL) { + njs_vm_error(vm, "XMLNode.$tag$xxx is not assignable, use addChild() or" + "node.$tags = [node1, node2, ..] syntax"); + return NJS_ERROR; + } + + /* delete. */ + + return njs_xml_node_tag_remove(vm, current, name); +} + + +static njs_int_t +njs_xml_node_tags_handler(njs_vm_t *vm, xmlNode *current, njs_str_t *name, + njs_value_t *setval, njs_value_t *retval) +{ + size_t size; + int64_t i, length; + xmlNode *node, *copy; + njs_int_t ret; + njs_value_t *push; + njs_opaque_value_t *start; + + if (retval != NULL && setval == NULL) { + + /* get. */ + + ret = njs_vm_array_alloc(vm, retval, 2); + if (njs_slow_path(ret != NJS_OK)) { + return NJS_ERROR; + } + + for (node = current->children; node != NULL; node = node->next) { + if (node->type != XML_ELEMENT_NODE) { + continue; + } + + size = njs_strlen(node->name); + + if (name->length > 0 + && (name->length != size + || njs_strncmp(name->start, node->name, size) != 0)) + { + continue; + } + + push = njs_vm_array_push(vm, retval); + if (njs_slow_path(push == NULL)) { + return NJS_ERROR; + } + + ret = njs_vm_external_create(vm, push, njs_xml_node_proto_id, + node, 0); + if (njs_slow_path(ret != NJS_OK)) { + return NJS_ERROR; + } + } + + return NJS_OK; + } + + if (name->length > 0) { + njs_vm_error(vm, "XMLNode $tags$xxx is not assignable, use addChild() " + "or node.$tags = [node1, node2, ..] syntax"); + return NJS_ERROR; + } + + /* set or delete. */ + + copy = xmlDocCopyNode(current, current->doc, 1); + if (njs_slow_path(copy == NULL)) { + njs_vm_error(vm, "xmlDocCopyNode() failed"); + return NJS_ERROR; + } + + if (copy->children != NULL) { + xmlFreeNodeList(copy->children); + copy->children = NULL; + } + + if (retval == NULL) { + /* delete. */ + return njs_xml_replace_node(vm, current, copy); + } + + if (!njs_value_is_array(setval)) { + njs_vm_error(vm, "setval is not an array"); + goto error; + } + + start = (njs_opaque_value_t *) njs_vm_array_start(vm, setval); + if (njs_slow_path(start == NULL)) { + goto error; + } + + (void) njs_vm_array_length(vm, setval, &length); + + for (i = 0; i < length; i++) { + node = njs_xml_external_node(vm, njs_value_arg(start++)); + if (njs_slow_path(node == NULL)) { + njs_vm_error(vm, "setval[%D] is not a XMLNode object", i); + goto error; + } + + node = xmlDocCopyNode(node, current->doc, 1); + if (njs_slow_path(node == NULL)) { + njs_vm_error(vm, "xmlDocCopyNode() failed"); + goto error; + } + + node = xmlAddChild(copy, node); + if (njs_slow_path(node == NULL)) { + njs_vm_error(vm, "xmlAddChild() failed"); + xmlFreeNode(node); + goto error; + } + + ret = xmlReconciliateNs(current->doc, copy); + if (njs_slow_path(ret == -1)) { + njs_vm_error(vm, "xmlReconciliateNs() failed"); + return NJS_ERROR; + } } - ret = njs_vm_value_string_create_chb(vm, retval, &chain); + njs_value_undefined_set(retval); + + return njs_xml_replace_node(vm, current, copy); + +error: + + xmlFreeNode(copy); + + return NJS_ERROR; +} + + +static xmlNode * +njs_xml_external_node(njs_vm_t *vm, njs_value_t *value) +{ + xmlNode *current; + njs_xml_doc_t *tree; + + current = njs_vm_external(vm, njs_xml_node_proto_id, value); + if (njs_slow_path(current == NULL)) { + tree = njs_vm_external(vm, njs_xml_doc_proto_id, value); + if (njs_slow_path(tree == NULL)) { + njs_vm_error(vm, "\"this\" is not a XMLNode object"); + return NULL; + } + + current = xmlDocGetRootElement(tree->doc); + if (njs_slow_path(current == NULL)) { + njs_vm_error(vm, "\"this\" is not a XMLNode object"); + return NULL; + } + } + + return current; +} + + +static njs_int_t +njs_xml_str_to_c_string(njs_vm_t *vm, njs_str_t *str, u_char *dst, + size_t size) +{ + u_char *p; + + if (njs_slow_path(str->length > size - njs_length("\0"))) { + njs_vm_error(vm, "njs_xml_str_to_c_string() very long string, " + "length >= %uz", size - njs_length("\0")); + return NJS_ERROR; + } + + p = njs_cpymem(dst, str->start, str->length); + *p = '\0'; + + return NJS_OK; +} From arut at nginx.com Thu Feb 23 11:49:14 2023 From: arut at nginx.com (=?iso-8859-1?q?Roman_Arutyunyan?=) Date: Thu, 23 Feb 2023 15:49:14 +0400 Subject: [PATCH] QUIC: optimized sending stream response Message-ID: <6d471753917c083b4044.1677152954@arut-laptop> # HG changeset patch # User Roman Arutyunyan # Date 1677152799 -14400 # Thu Feb 23 15:46:39 2023 +0400 # Branch quic # Node ID 6d471753917c083b4044f73557f9af8d91c20d36 # Parent 3c33d39a51d334d99fcc7d2b45e8d8190c431492 QUIC: optimized sending stream response. When a stream is created by client, it's often the case that nginx will send immediate response on that stream. An example is HTTP/3 request stream, which in most cases quickly replies with at least HTTP headers. QUIC stream init handlers are called from a posted event. Output QUIC frames are also sent to client from a posted event, called the push event. If the push event is posted before the stream init event, then output produced by stream may trigger sending an extra UDP datagram. To address this, push event is now re-posted when a new stream init event is posted. An example is handling 0-RTT packets. Client typically sends an init packet coalesced with a 0-RTT packet. Previously, nginx replied with a padded CRYPTO datagram, followed by a 1-RTT stream reply datagram. Now CRYPTO and STREAM packets are coalesced in one reply datagram, which saves bandwidth. diff --git a/src/event/quic/ngx_event_quic_streams.c b/src/event/quic/ngx_event_quic_streams.c --- a/src/event/quic/ngx_event_quic_streams.c +++ b/src/event/quic/ngx_event_quic_streams.c @@ -472,6 +472,17 @@ ngx_quic_get_stream(ngx_connection_t *c, if (qc->streams.initialized) { ngx_post_event(rev, &ngx_posted_events); + + if (qc->push.posted) { + /* + * It's highly likely the posted stream will produce output + * immediately. By postponing the push event, we coalesce + * the stream output with queued frames in one UDP datagram. + */ + + ngx_delete_posted_event(&qc->push); + ngx_post_event(&qc->push, &ngx_posted_events); + } } } From pluknet at nginx.com Thu Feb 23 12:56:49 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Thu, 23 Feb 2023 16:56:49 +0400 Subject: [PATCH] QUIC: improved ssl_reject_handshake error logging In-Reply-To: <20230221115805.j7aefptanzhkhyfh@N00W24XTQX> References: <60b9d9ef878cc87d8a18.1676907424@enoparse.local> <20230221115805.j7aefptanzhkhyfh@N00W24XTQX> Message-ID: <2A10ECAC-7D43-4AB8-9D85-B9381750ADB9@nginx.com> > On 21 Feb 2023, at 15:58, Roman Arutyunyan wrote: > > On Mon, Feb 20, 2023 at 07:37:04PM +0400, Sergey Kandaurov wrote: >> # HG changeset patch >> # User Sergey Kandaurov >> # Date 1676907370 -14400 >> # Mon Feb 20 19:36:10 2023 +0400 >> # Branch quic >> # Node ID 60b9d9ef878cc87d8a18e968b4f9122534a7deb8 >> # Parent ed403c1fe1f060ee362155bb00c43aaab356e30f >> QUIC: improved ssl_reject_handshake error logging. >> >> The check follows the ngx_ssl_handshake() change in 59e1c73fe02b. >> >> diff --git a/src/event/quic/ngx_event_quic_ssl.c b/src/event/quic/ngx_event_quic_ssl.c >> --- a/src/event/quic/ngx_event_quic_ssl.c >> +++ b/src/event/quic/ngx_event_quic_ssl.c >> @@ -422,8 +422,17 @@ ngx_quic_crypto_input(ngx_connection_t * >> sslerr); >> >> if (sslerr != SSL_ERROR_WANT_READ) { >> + >> + qc->error_reason = "handshake failed"; >> + >> + if (c->ssl->handshake_rejected) { >> + ngx_connection_error(c, 0, "handshake rejected"); >> + ERR_clear_error(); >> + >> + return NGX_ERROR; >> + } >> + >> ngx_ssl_error(NGX_LOG_ERR, c->log, 0, "SSL_do_handshake() failed"); >> - qc->error_reason = "handshake failed"; >> return NGX_ERROR; >> } >> } > > Looks ok For the record, the patch was slightly simplified after cb7dc35ed428. Pushed in 639fa6723700. -- Sergey Kandaurov From pluknet at nginx.com Thu Feb 23 12:58:01 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Thu, 23 Feb 2023 16:58:01 +0400 Subject: QUIC: add error code for handshake failed. In-Reply-To: <20230222131546.2sgyga72jnp7gsiv@N00W24XTQX> References: <5b0eee35.4de7.1865a7130cb.Coremail.jiuzhoucui@163.com> <9CFEFC7C-19E1-4AFD-8CE5-2D55AAC02AA8@nginx.com> <170ee7d0.234e.1865dce18a2.Coremail.jiuzhoucui@163.com> <1787FA6E-24CB-4CCE-8A5D-3EFBA7BD9AEC@nginx.com> <6EE82B90-09DA-4C49-96B6-EBCC0B963122@nginx.com> <20230222131546.2sgyga72jnp7gsiv@N00W24XTQX> Message-ID: <1B4B2ED8-C873-4D1C-A397-54D3DEC175FC@nginx.com> > On 22 Feb 2023, at 17:15, Roman Arutyunyan wrote: > > On Tue, Feb 21, 2023 at 05:54:42PM +0400, Sergey Kandaurov wrote: > > [..] > >> # HG changeset patch >> # User Sergey Kandaurov >> # Date 1676987626 -14400 >> # Tue Feb 21 17:53:46 2023 +0400 >> # Branch quic >> # Node ID e012c3999592fc935bfc786a232903567c512bfe >> # Parent baada8d864cbedc7557b879f89eee5c412f3ca85 >> QUIC: moved "handshake failed" reason to send_alert. >> >> A QUIC handshake may fail for a variety of reasons, which breaks down into >> several cases, in order of precedence: > >> - generally, the error code is reported with the send_alert callback >> - else, our own QUIC checks may set specific error code / reason phrase >> - as a last resort, handshake may fail for some reason, which falls back >> to send INTERNAL_ERROR in the CONNECTION_CLOSE frame. >> Now, in the first case, both error code and generic phrase are set in the >> send_alert callback, to preserve a specific reason phrase from overriding. >> Additionally, absent error code / reason phrase are now converted to just >> INTERNAL_ERROR, without reason phrase set. > i> >> Reported by Jiuzhou Cui. > > I suggest the following commit message: > > QUIC: moved "handshake failed" reason to send_alert. > > A QUIC handshake may fail for a variety of reasons, which breaks down into > several cases, in order of precedence: > > - a handshake error which leads to a send_alert call > - an error triggered by add_handshake_data callback > - internal errors (allocation etc) > > Previously, in the first case, only error code was set in send_alert callback. > Now "handshake failed" is set there as a reason. In the second case, error > code and reason are set by add_handshake_data. In the last case, setting > error reason is now removed. Returning NGX_ERROR will lead to closing the > connection with just INTERNAL_ERROR. > > Reported by Jiuzhou Cui. Thanks, applied with minor adjustments. https://hg.nginx.org/nginx-quic/rev/cb7dc35ed428 -- Sergey Kandaurov From mdounin at mdounin.ru Thu Feb 23 17:48:39 2023 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 23 Feb 2023 20:48:39 +0300 Subject: [PATCH 4 of 4] Win32: OpenSSL compilation for x64 targets with MSVC In-Reply-To: <2AD9CC4B-5466-4DC4-AEAF-136F0512085E@nginx.com> References: <2AD9CC4B-5466-4DC4-AEAF-136F0512085E@nginx.com> Message-ID: Hello! On Fri, Feb 10, 2023 at 06:11:48PM +0400, Sergey Kandaurov wrote: > > On 20 Dec 2022, at 17:30, Maxim Dounin wrote: > > > > # HG changeset patch > > # User Maxim Dounin > > # Date 1671542914 -10800 > > # Tue Dec 20 16:28:34 2022 +0300 > > # Node ID e5a75718823d5ec365703275f3efa87d0b63f8c4 > > # Parent 6606ed21a7091b060ebec0d082876ddbbbe0ea79 > > Win32: OpenSSL compilation for x64 targets with MSVC. > > > > To ensure proper target selection the NGX_MACHINE variable is now set > > based on the MSVC compiler output, and the OpenSSL target is set based > > on it. > > > > This is not important as long as "no-asm" is used (as in misc/GNUmakefile > > and win32 build instructions), but might be beneficial if someone is trying > > to build OpenSSL with assembler code. > > I was not able to test x64 case, but looking at the change it's all good. Pushed to http://mdounin.ru/hg/nginx. [...] -- Maxim Dounin http://mdounin.ru/ From mdounin at mdounin.ru Thu Feb 23 18:46:03 2023 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 23 Feb 2023 21:46:03 +0300 Subject: [PATCH 11 of 12] Win32: fixed ngx_fs_bsize() for symlinks In-Reply-To: <5F74999F-228E-43AB-B059-879964530E55@nginx.com> References: <5F74999F-228E-43AB-B059-879964530E55@nginx.com> Message-ID: Hello! On Wed, Feb 22, 2023 at 08:01:15PM +0400, Sergey Kandaurov wrote: > > On 19 Feb 2023, at 21:23, Maxim Dounin wrote: > > > > Hello! > > > > On Fri, Feb 17, 2023 at 07:17:02PM +0400, Sergey Kandaurov wrote: > > > >>> On 13 Jan 2023, at 01:35, Maxim Dounin wrote: > >>> > >>> # HG changeset patch > >>> # User Maxim Dounin > >>> # Date 1673549010 -10800 > >>> # Thu Jan 12 21:43:30 2023 +0300 > >>> # Node ID be7eb9ec28dcbfdfd2e850befc8d051c0e4d46fd > >>> # Parent e62c8e9724ba68a698a2c3613edca73fe4e1c4ae > >>> Win32: fixed ngx_fs_bsize() for symlinks. > >>> > >>> Just a drive letter might not correctly represent file system being used, > >>> notably when using symlinks (as created by "mklink /d"). As such, instead > >>> of calling GetDiskFreeSpace() with just a drive letter, we now always > >>> use GetDiskFreeSpace() with full path. > >>> > >>> diff -r e62c8e9724ba -r be7eb9ec28dc src/os/win32/ngx_files.c > >>> --- a/src/os/win32/ngx_files.c Thu Jan 12 21:43:14 2023 +0300 > >>> +++ b/src/os/win32/ngx_files.c Thu Jan 12 21:43:30 2023 +0300 > >>> @@ -955,14 +955,8 @@ ngx_directio_off(ngx_fd_t fd) > >>> size_t > >>> ngx_fs_bsize(u_char *name) > >>> { > >>> - u_char root[4]; > >>> u_long sc, bs, nfree, ncl; > >>> > >>> - if (name[2] == ':') { > >>> - ngx_cpystrn(root, name, 4); > >>> - name = root; > >>> - } > >>> - > >> > >> BTW, I wonder how this condition could be true. > >> Specifically, what name should represent in order to match. > >> I'm happy that it's leaving though. > > > > I tend to think that this actually never worked, and the original > > intention was to test name[1] instead. > > > > Updated commit log: > > > > : Win32: removed attempt to use a drive letter in ngx_fs_bsize(). > > : > > : Just a drive letter might not correctly represent file system being used, > > : notably when using symlinks (as created by "mklink /d"). As such, instead > > : of trying to call GetDiskFreeSpace() with just a drive letter, we now always > > : use GetDiskFreeSpace() with full path. > > : > > : Further, it looks like the code to use just a drive letter never worked, > > : since it tried to test name[2] instead of name[1] to be ':'. > > > > [...] > > > > Looks fine, thanks. Thanks for the review, pushed to http://mdounin.ru/hg/nginx. -- Maxim Dounin http://mdounin.ru/ From pluknet at nginx.com Fri Feb 24 10:32:36 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Fri, 24 Feb 2023 10:32:36 +0000 Subject: [nginx] Win32: removed unneeded wildcard in NGX_CC_NAME test for msvc. Message-ID: details: https://hg.nginx.org/nginx/rev/9bcc5cc94ff4 branches: changeset: 8126:9bcc5cc94ff4 user: Maxim Dounin date: Thu Feb 23 18:15:53 2023 +0300 description: Win32: removed unneeded wildcard in NGX_CC_NAME test for msvc. Wildcards for msvc in NGX_CC_NAME tests are not needed since 78f8ac479735. diffstat: auto/cc/conf | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff -r cffaf3f2eec8 -r 9bcc5cc94ff4 auto/cc/conf --- a/auto/cc/conf Thu Feb 02 23:38:48 2023 +0300 +++ b/auto/cc/conf Thu Feb 23 18:15:53 2023 +0300 @@ -117,7 +117,7 @@ else . auto/cc/acc ;; - msvc*) + msvc) # MSVC++ 6.0 SP2, MSVC++ Toolkit 2003 . auto/cc/msvc From pluknet at nginx.com Fri Feb 24 10:32:39 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Fri, 24 Feb 2023 10:32:39 +0000 Subject: [nginx] Win32: handling of localized MSVC cl output. Message-ID: details: https://hg.nginx.org/nginx/rev/17b3efb45b17 branches: changeset: 8127:17b3efb45b17 user: Maxim Dounin date: Thu Feb 23 18:15:57 2023 +0300 description: Win32: handling of localized MSVC cl output. Output examples in English, Russian, and Spanish: Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 16.00.30319.01 for 80x86 Оптимизирующий 32-разрядный компилятор Microsoft (R) C/C++ версии 16.00.30319.01 для 80x86 Compilador de optimización de C/C++ de Microsoft (R) versión 16.00.30319.01 para x64 Since most of the words are translated, instead of looking for the words "Compiler Version" we now search for "C/C++" and the version number. diffstat: auto/cc/msvc | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diffs (14 lines): diff -r 9bcc5cc94ff4 -r 17b3efb45b17 auto/cc/msvc --- a/auto/cc/msvc Thu Feb 23 18:15:53 2023 +0300 +++ b/auto/cc/msvc Thu Feb 23 18:15:57 2023 +0300 @@ -11,8 +11,8 @@ # MSVC 2015 (14.0) cl 19.00 -NGX_MSVC_VER=`$NGX_WINE $CC 2>&1 | grep 'Compiler Version' 2>&1 \ - | sed -e 's/^.* Version \(.*\)/\1/'` +NGX_MSVC_VER=`$NGX_WINE $CC 2>&1 | grep 'C/C++.* [0-9][0-9]*\.[0-9]' 2>&1 \ + | sed -e 's/^.* \([0-9][0-9]*\.[0-9].*\)/\1/'` echo " + cl version: $NGX_MSVC_VER" From pluknet at nginx.com Fri Feb 24 10:32:42 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Fri, 24 Feb 2023 10:32:42 +0000 Subject: [nginx] Win32: i386 now assumed when crossbuilding (ticket #2416). Message-ID: details: https://hg.nginx.org/nginx/rev/79c04253bc43 branches: changeset: 8128:79c04253bc43 user: Maxim Dounin date: Thu Feb 23 18:15:59 2023 +0300 description: Win32: i386 now assumed when crossbuilding (ticket #2416). Previously, NGX_MACHINE was not set when crossbuilding, resulting in NGX_ALIGNMENT=16 being used in 32-bit builds (if not explicitly set to a correct value). This in turn might result in memory corruption in ngx_palloc() (as there are no usable aligned allocator on Windows, and normal malloc() is used instead, which provides 8 byte alignment on 32-bit platforms). To fix this, now i386 machine is set when crossbuilding, so nginx won't assume strict alignment requirements. diffstat: auto/configure | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diffs (11 lines): diff -r 17b3efb45b17 -r 79c04253bc43 auto/configure --- a/auto/configure Thu Feb 23 18:15:57 2023 +0300 +++ b/auto/configure Thu Feb 23 18:15:59 2023 +0300 @@ -44,6 +44,7 @@ if test -z "$NGX_PLATFORM"; then else echo "building for $NGX_PLATFORM" NGX_SYSTEM=$NGX_PLATFORM + NGX_MACHINE=i386 fi . auto/cc/conf From pluknet at nginx.com Fri Feb 24 10:32:45 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Fri, 24 Feb 2023 10:32:45 +0000 Subject: [nginx] Win32: OpenSSL compilation for x64 targets with MSVC. Message-ID: details: https://hg.nginx.org/nginx/rev/3c4d81ea1338 branches: changeset: 8129:3c4d81ea1338 user: Maxim Dounin date: Thu Feb 23 18:16:08 2023 +0300 description: Win32: OpenSSL compilation for x64 targets with MSVC. To ensure proper target selection the NGX_MACHINE variable is now set based on the MSVC compiler output, and the OpenSSL target is set based on it. This is not important as long as "no-asm" is used (as in misc/GNUmakefile and win32 build instructions), but might be beneficial if someone is trying to build OpenSSL with assembler code. diffstat: auto/cc/msvc | 15 +++++++++++++++ auto/lib/openssl/make | 15 ++++++++++++++- auto/lib/openssl/makefile.msvc | 2 +- 3 files changed, 30 insertions(+), 2 deletions(-) diffs (66 lines): diff -r 79c04253bc43 -r 3c4d81ea1338 auto/cc/msvc --- a/auto/cc/msvc Thu Feb 23 18:15:59 2023 +0300 +++ b/auto/cc/msvc Thu Feb 23 18:16:08 2023 +0300 @@ -22,6 +22,21 @@ have=NGX_COMPILER value="\"cl $NGX_MSVC_ ngx_msvc_ver=`echo $NGX_MSVC_VER | sed -e 's/^\([0-9]*\).*/\1/'` +# detect x64 builds + +case "$NGX_MSVC_VER" in + + *x64) + NGX_MACHINE=amd64 + ;; + + *) + NGX_MACHINE=i386 + ;; + +esac + + # optimizations # maximize speed, equivalent to -Og -Oi -Ot -Oy -Ob2 -Gs -GF -Gy diff -r 79c04253bc43 -r 3c4d81ea1338 auto/lib/openssl/make --- a/auto/lib/openssl/make Thu Feb 23 18:15:59 2023 +0300 +++ b/auto/lib/openssl/make Thu Feb 23 18:16:08 2023 +0300 @@ -7,11 +7,24 @@ case "$CC" in cl) + case "$NGX_MACHINE" in + + amd64) + OPENSSL_TARGET=VC-WIN64A + ;; + + *) + OPENSSL_TARGET=VC-WIN32 + ;; + + esac + cat << END >> $NGX_MAKEFILE $OPENSSL/openssl/include/openssl/ssl.h: $NGX_MAKEFILE \$(MAKE) -f auto/lib/openssl/makefile.msvc \ - OPENSSL="$OPENSSL" OPENSSL_OPT="$OPENSSL_OPT" + OPENSSL="$OPENSSL" OPENSSL_OPT="$OPENSSL_OPT" \ + OPENSSL_TARGET="$OPENSSL_TARGET" END diff -r 79c04253bc43 -r 3c4d81ea1338 auto/lib/openssl/makefile.msvc --- a/auto/lib/openssl/makefile.msvc Thu Feb 23 18:15:59 2023 +0300 +++ b/auto/lib/openssl/makefile.msvc Thu Feb 23 18:16:08 2023 +0300 @@ -6,7 +6,7 @@ all: cd $(OPENSSL) - perl Configure VC-WIN32 no-shared no-threads \ + perl Configure $(OPENSSL_TARGET) no-shared no-threads \ --prefix="%cd%/openssl" \ --openssldir="%cd%/openssl/ssl" \ $(OPENSSL_OPT) From pluknet at nginx.com Fri Feb 24 10:32:48 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Fri, 24 Feb 2023 10:32:48 +0000 Subject: [nginx] Win32: non-ASCII names support in autoindex (ticket #458). Message-ID: details: https://hg.nginx.org/nginx/rev/b0a06c50c1b4 branches: changeset: 8130:b0a06c50c1b4 user: Maxim Dounin date: Thu Feb 23 20:49:39 2023 +0300 description: Win32: non-ASCII names support in autoindex (ticket #458). Notably, ngx_open_dir() now supports opening directories with non-ASCII characters, and directory entries returned by ngx_read_dir() are properly converted to UTF-8. diffstat: src/os/win32/ngx_files.c | 273 ++++++++++++++++++++++++++++++++++++++++++---- src/os/win32/ngx_files.h | 10 +- 2 files changed, 253 insertions(+), 30 deletions(-) diffs (379 lines): diff -r 3c4d81ea1338 -r b0a06c50c1b4 src/os/win32/ngx_files.c --- a/src/os/win32/ngx_files.c Thu Feb 23 18:16:08 2023 +0300 +++ b/src/os/win32/ngx_files.c Thu Feb 23 20:49:39 2023 +0300 @@ -13,7 +13,11 @@ static ngx_int_t ngx_win32_check_filename(u_char *name, u_short *u, size_t len); -static u_short *ngx_utf8_to_utf16(u_short *utf16, u_char *utf8, size_t *len); +static u_short *ngx_utf8_to_utf16(u_short *utf16, u_char *utf8, size_t *len, + size_t reserved); +static u_char *ngx_utf16_to_utf8(u_char *utf8, u_short *utf16, size_t *len, + size_t *allocated); +uint32_t ngx_utf16_decode(u_short **u, size_t n); /* FILE_FLAG_BACKUP_SEMANTICS allows to obtain a handle to a directory */ @@ -28,7 +32,7 @@ ngx_open_file(u_char *name, u_long mode, u_short utf16[NGX_UTF16_BUFLEN]; len = NGX_UTF16_BUFLEN; - u = ngx_utf8_to_utf16(utf16, name, &len); + u = ngx_utf8_to_utf16(utf16, name, &len, 0); if (u == NULL) { return INVALID_HANDLE_VALUE; @@ -269,7 +273,7 @@ ngx_file_info(u_char *file, ngx_file_inf len = NGX_UTF16_BUFLEN; - u = ngx_utf8_to_utf16(utf16, file, &len); + u = ngx_utf8_to_utf16(utf16, file, &len, 0); if (u == NULL) { return NGX_FILE_ERROR; @@ -427,49 +431,51 @@ ngx_realpath(u_char *path, u_char *resol ngx_int_t ngx_open_dir(ngx_str_t *name, ngx_dir_t *dir) { - u_char *pattern, *p; + size_t len; + u_short *u, *p; ngx_err_t err; + u_short utf16[NGX_UTF16_BUFLEN]; - pattern = malloc(name->len + 3); - if (pattern == NULL) { + len = NGX_UTF16_BUFLEN - 2; + u = ngx_utf8_to_utf16(utf16, name->data, &len, 2); + + if (u == NULL) { return NGX_ERROR; } - p = ngx_cpymem(pattern, name->data, name->len); + if (ngx_win32_check_filename(name->data, u, len) != NGX_OK) { + goto failed; + } + + p = &u[len - 1]; *p++ = '/'; *p++ = '*'; *p = '\0'; - dir->dir = FindFirstFile((const char *) pattern, &dir->finddata); + dir->dir = FindFirstFileW(u, &dir->finddata); if (dir->dir == INVALID_HANDLE_VALUE) { - err = ngx_errno; - ngx_free(pattern); - ngx_set_errno(err); - return NGX_ERROR; + goto failed; } - ngx_free(pattern); + if (u != utf16) { + ngx_free(u); + } dir->valid_info = 1; dir->ready = 1; + dir->name = NULL; + dir->allocated = 0; return NGX_OK; -} +failed: -ngx_int_t -ngx_read_dir(ngx_dir_t *dir) -{ - if (dir->ready) { - dir->ready = 0; - return NGX_OK; - } - - if (FindNextFile(dir->dir, &dir->finddata) != 0) { - dir->type = 1; - return NGX_OK; + if (u != utf16) { + err = ngx_errno; + ngx_free(u); + ngx_set_errno(err); } return NGX_ERROR; @@ -477,8 +483,57 @@ ngx_read_dir(ngx_dir_t *dir) ngx_int_t +ngx_read_dir(ngx_dir_t *dir) +{ + u_char *name; + size_t len, allocated; + + if (dir->ready) { + dir->ready = 0; + goto convert; + } + + if (FindNextFileW(dir->dir, &dir->finddata) != 0) { + dir->type = 1; + goto convert; + } + + return NGX_ERROR; + +convert: + + name = dir->name; + len = dir->allocated; + + name = ngx_utf16_to_utf8(name, dir->finddata.cFileName, &len, &allocated); + + if (name == NULL) { + return NGX_ERROR; + } + + if (name != dir->name) { + + if (dir->name) { + ngx_free(dir->name); + } + + dir->name = name; + dir->allocated = allocated; + } + + dir->namelen = len - 1; + + return NGX_OK; +} + + +ngx_int_t ngx_close_dir(ngx_dir_t *dir) { + if (dir->name) { + ngx_free(dir->name); + } + if (FindClose(dir->dir) == 0) { return NGX_ERROR; } @@ -816,7 +871,7 @@ failed: static u_short * -ngx_utf8_to_utf16(u_short *utf16, u_char *utf8, size_t *len) +ngx_utf8_to_utf16(u_short *utf16, u_char *utf8, size_t *len, size_t reserved) { u_char *p; u_short *u, *last; @@ -865,7 +920,7 @@ ngx_utf8_to_utf16(u_short *utf16, u_char /* the given buffer is not enough, allocate a new one */ - u = malloc(((p - utf8) + ngx_strlen(p) + 1) * sizeof(u_short)); + u = malloc(((p - utf8) + ngx_strlen(p) + 1 + reserved) * sizeof(u_short)); if (u == NULL) { return NULL; } @@ -910,3 +965,167 @@ ngx_utf8_to_utf16(u_short *utf16, u_char /* unreachable */ } + + +static u_char * +ngx_utf16_to_utf8(u_char *utf8, u_short *utf16, size_t *len, size_t *allocated) +{ + u_char *p, *last; + u_short *u, *j; + uint32_t n; + + u = utf16; + p = utf8; + last = utf8 + *len; + + while (p < last) { + + if (*u < 0x80) { + *p++ = (u_char) *u; + + if (*u == 0) { + *len = p - utf8; + return utf8; + } + + u++; + + continue; + } + + if (p >= last - 4) { + *len = p - utf8; + break; + } + + n = ngx_utf16_decode(&u, 2); + + if (n > 0x10ffff) { + ngx_set_errno(NGX_EILSEQ); + return NULL; + } + + if (n >= 0x10000) { + *p++ = (u_char) (0xf0 + (n >> 18)); + *p++ = (u_char) (0x80 + ((n >> 12) & 0x3f)); + *p++ = (u_char) (0x80 + ((n >> 6) & 0x3f)); + *p++ = (u_char) (0x80 + (n & 0x3f)); + continue; + } + + if (n >= 0x0800) { + *p++ = (u_char) (0xe0 + (n >> 12)); + *p++ = (u_char) (0x80 + ((n >> 6) & 0x3f)); + *p++ = (u_char) (0x80 + (n & 0x3f)); + continue; + } + + *p++ = (u_char) (0xc0 + (n >> 6)); + *p++ = (u_char) (0x80 + (n & 0x3f)); + } + + /* the given buffer is not enough, allocate a new one */ + + for (j = u; *j; j++) { /* void */ } + + p = malloc((j - utf16) * 4 + 1); + if (p == NULL) { + return NULL; + } + + if (allocated) { + *allocated = (j - utf16) * 4 + 1; + } + + ngx_memcpy(p, utf8, *len); + + utf8 = p; + p += *len; + + for ( ;; ) { + + if (*u < 0x80) { + *p++ = (u_char) *u; + + if (*u == 0) { + *len = p - utf8; + return utf8; + } + + u++; + + continue; + } + + n = ngx_utf16_decode(&u, 2); + + if (n > 0x10ffff) { + ngx_free(utf8); + ngx_set_errno(NGX_EILSEQ); + return NULL; + } + + if (n >= 0x10000) { + *p++ = (u_char) (0xf0 + (n >> 18)); + *p++ = (u_char) (0x80 + ((n >> 12) & 0x3f)); + *p++ = (u_char) (0x80 + ((n >> 6) & 0x3f)); + *p++ = (u_char) (0x80 + (n & 0x3f)); + continue; + } + + if (n >= 0x0800) { + *p++ = (u_char) (0xe0 + (n >> 12)); + *p++ = (u_char) (0x80 + ((n >> 6) & 0x3f)); + *p++ = (u_char) (0x80 + (n & 0x3f)); + continue; + } + + *p++ = (u_char) (0xc0 + (n >> 6)); + *p++ = (u_char) (0x80 + (n & 0x3f)); + } + + /* unreachable */ +} + + +/* + * ngx_utf16_decode() decodes one or two UTF-16 code units + * the return values: + * 0x80 - 0x10ffff valid character + * 0x110000 - 0xfffffffd invalid sequence + * 0xfffffffe incomplete sequence + * 0xffffffff error + */ + +uint32_t +ngx_utf16_decode(u_short **u, size_t n) +{ + uint32_t k, m; + + k = **u; + + if (k < 0xd800 || k > 0xdfff) { + (*u)++; + return k; + } + + if (k > 0xdbff) { + (*u)++; + return 0xffffffff; + } + + if (n < 2) { + return 0xfffffffe; + } + + (*u)++; + + m = *(*u)++; + + if (m < 0xdc00 || m > 0xdfff) { + return 0xffffffff; + + } + + return 0x10000 + ((k - 0xd800) << 10) + (m - 0xdc00); +} diff -r 3c4d81ea1338 -r b0a06c50c1b4 src/os/win32/ngx_files.h --- a/src/os/win32/ngx_files.h Thu Feb 23 18:16:08 2023 +0300 +++ b/src/os/win32/ngx_files.h Thu Feb 23 20:49:39 2023 +0300 @@ -30,7 +30,11 @@ typedef struct { typedef struct { HANDLE dir; - WIN32_FIND_DATA finddata; + WIN32_FIND_DATAW finddata; + + u_char *name; + size_t namelen; + size_t allocated; unsigned valid_info:1; unsigned type:1; @@ -205,8 +209,8 @@ ngx_int_t ngx_close_dir(ngx_dir_t *dir); #define ngx_dir_access(a) (a) -#define ngx_de_name(dir) ((u_char *) (dir)->finddata.cFileName) -#define ngx_de_namelen(dir) ngx_strlen((dir)->finddata.cFileName) +#define ngx_de_name(dir) (dir)->name +#define ngx_de_namelen(dir) (dir)->namelen ngx_int_t ngx_de_info(u_char *name, ngx_dir_t *dir); #define ngx_de_info_n "dummy()" From pluknet at nginx.com Fri Feb 24 10:32:51 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Fri, 24 Feb 2023 10:32:51 +0000 Subject: [nginx] Win32: non-ASCII names support in "include" with wildcards. Message-ID: details: https://hg.nginx.org/nginx/rev/751f79bd802c branches: changeset: 8131:751f79bd802c user: Maxim Dounin date: Thu Feb 23 20:49:41 2023 +0300 description: Win32: non-ASCII names support in "include" with wildcards. Notably, ngx_open_glob() now supports opening directories with non-ASCII characters, and pathnames returned by ngx_read_glob() are converted to UTF-8. diffstat: src/os/win32/ngx_files.c | 96 +++++++++++++++++++++++++++++++---------------- src/os/win32/ngx_files.h | 2 +- 2 files changed, 64 insertions(+), 34 deletions(-) diffs (174 lines): diff -r b0a06c50c1b4 -r 751f79bd802c src/os/win32/ngx_files.c --- a/src/os/win32/ngx_files.c Thu Feb 23 20:49:39 2023 +0300 +++ b/src/os/win32/ngx_files.c Thu Feb 23 20:49:41 2023 +0300 @@ -10,6 +10,7 @@ #define NGX_UTF16_BUFLEN 256 +#define NGX_UTF8_BUFLEN 512 static ngx_int_t ngx_win32_check_filename(u_char *name, u_short *u, size_t len); @@ -547,14 +548,27 @@ ngx_open_glob(ngx_glob_t *gl) { u_char *p; size_t len; + u_short *u; ngx_err_t err; + u_short utf16[NGX_UTF16_BUFLEN]; - gl->dir = FindFirstFile((const char *) gl->pattern, &gl->finddata); + len = NGX_UTF16_BUFLEN; + u = ngx_utf8_to_utf16(utf16, gl->pattern, &len, 0); + + if (u == NULL) { + return NGX_ERROR; + } + + gl->dir = FindFirstFileW(u, &gl->finddata); if (gl->dir == INVALID_HANDLE_VALUE) { err = ngx_errno; + if (u != utf16) { + ngx_free(u); + } + if ((err == ERROR_FILE_NOT_FOUND || err == ERROR_PATH_NOT_FOUND) && gl->test) { @@ -562,6 +576,8 @@ ngx_open_glob(ngx_glob_t *gl) return NGX_OK; } + ngx_set_errno(err); + return NGX_ERROR; } @@ -571,18 +587,10 @@ ngx_open_glob(ngx_glob_t *gl) } } - len = ngx_strlen(gl->finddata.cFileName); - gl->name.len = gl->last + len; - - gl->name.data = ngx_alloc(gl->name.len + 1, gl->log); - if (gl->name.data == NULL) { - return NGX_ERROR; + if (u != utf16) { + ngx_free(u); } - ngx_memcpy(gl->name.data, gl->pattern, gl->last); - ngx_cpystrn(gl->name.data + gl->last, (u_char *) gl->finddata.cFileName, - len + 1); - gl->ready = 1; return NGX_OK; @@ -592,40 +600,25 @@ ngx_open_glob(ngx_glob_t *gl) ngx_int_t ngx_read_glob(ngx_glob_t *gl, ngx_str_t *name) { - size_t len; - ngx_err_t err; + u_char *p; + size_t len; + ngx_err_t err; + u_char utf8[NGX_UTF8_BUFLEN]; if (gl->no_match) { return NGX_DONE; } if (gl->ready) { - *name = gl->name; - gl->ready = 0; - return NGX_OK; + goto convert; } ngx_free(gl->name.data); gl->name.data = NULL; - if (FindNextFile(gl->dir, &gl->finddata) != 0) { - - len = ngx_strlen(gl->finddata.cFileName); - gl->name.len = gl->last + len; - - gl->name.data = ngx_alloc(gl->name.len + 1, gl->log); - if (gl->name.data == NULL) { - return NGX_ERROR; - } - - ngx_memcpy(gl->name.data, gl->pattern, gl->last); - ngx_cpystrn(gl->name.data + gl->last, (u_char *) gl->finddata.cFileName, - len + 1); - - *name = gl->name; - - return NGX_OK; + if (FindNextFileW(gl->dir, &gl->finddata) != 0) { + goto convert; } err = ngx_errno; @@ -638,6 +631,43 @@ ngx_read_glob(ngx_glob_t *gl, ngx_str_t "FindNextFile(%s) failed", gl->pattern); return NGX_ERROR; + +convert: + + len = NGX_UTF8_BUFLEN; + p = ngx_utf16_to_utf8(utf8, gl->finddata.cFileName, &len, NULL); + + if (p == NULL) { + return NGX_ERROR; + } + + gl->name.len = gl->last + len - 1; + + gl->name.data = ngx_alloc(gl->name.len + 1, gl->log); + if (gl->name.data == NULL) { + goto failed; + } + + ngx_memcpy(gl->name.data, gl->pattern, gl->last); + ngx_cpystrn(gl->name.data + gl->last, p, len); + + if (p != utf8) { + ngx_free(p); + } + + *name = gl->name; + + return NGX_OK; + +failed: + + if (p != utf8) { + err = ngx_errno; + ngx_free(p); + ngx_set_errno(err); + } + + return NGX_ERROR; } diff -r b0a06c50c1b4 -r 751f79bd802c src/os/win32/ngx_files.h --- a/src/os/win32/ngx_files.h Thu Feb 23 20:49:39 2023 +0300 +++ b/src/os/win32/ngx_files.h Thu Feb 23 20:49:41 2023 +0300 @@ -44,7 +44,7 @@ typedef struct { typedef struct { HANDLE dir; - WIN32_FIND_DATA finddata; + WIN32_FIND_DATAW finddata; unsigned ready:1; unsigned test:1; From pluknet at nginx.com Fri Feb 24 10:32:53 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Fri, 24 Feb 2023 10:32:53 +0000 Subject: [nginx] Win32: non-ASCII directory names support in ngx_getcwd(). Message-ID: details: https://hg.nginx.org/nginx/rev/8ea2e052feb4 branches: changeset: 8132:8ea2e052feb4 user: Maxim Dounin date: Thu Feb 23 20:49:44 2023 +0300 description: Win32: non-ASCII directory names support in ngx_getcwd(). This makes it possible to start nginx without a prefix explicitly set in a directory with non-ASCII characters in it. diffstat: src/os/win32/ngx_files.c | 34 ++++++++++++++++++++++++++++++++++ src/os/win32/ngx_files.h | 6 +++++- 2 files changed, 39 insertions(+), 1 deletions(-) diffs (61 lines): diff -r 751f79bd802c -r 8ea2e052feb4 src/os/win32/ngx_files.c --- a/src/os/win32/ngx_files.c Thu Feb 23 20:49:41 2023 +0300 +++ b/src/os/win32/ngx_files.c Thu Feb 23 20:49:44 2023 +0300 @@ -429,6 +429,40 @@ ngx_realpath(u_char *path, u_char *resol } +size_t +ngx_getcwd(u_char *buf, size_t size) +{ + u_char *p; + size_t n; + u_short utf16[NGX_MAX_PATH]; + + n = GetCurrentDirectoryW(NGX_MAX_PATH, utf16); + + if (n == 0) { + return 0; + } + + if (n > NGX_MAX_PATH) { + ngx_set_errno(ERROR_INSUFFICIENT_BUFFER); + return 0; + } + + p = ngx_utf16_to_utf8(buf, utf16, &size, NULL); + + if (p == NULL) { + return 0; + } + + if (p != buf) { + ngx_free(p); + ngx_set_errno(ERROR_INSUFFICIENT_BUFFER); + return 0; + } + + return size - 1; +} + + ngx_int_t ngx_open_dir(ngx_str_t *name, ngx_dir_t *dir) { diff -r 751f79bd802c -r 8ea2e052feb4 src/os/win32/ngx_files.h --- a/src/os/win32/ngx_files.h Thu Feb 23 20:49:41 2023 +0300 +++ b/src/os/win32/ngx_files.h Thu Feb 23 20:49:44 2023 +0300 @@ -178,8 +178,12 @@ void ngx_close_file_mapping(ngx_file_map u_char *ngx_realpath(u_char *path, u_char *resolved); #define ngx_realpath_n "" -#define ngx_getcwd(buf, size) GetCurrentDirectory(size, (char *) buf) + + +size_t ngx_getcwd(u_char *buf, size_t size); #define ngx_getcwd_n "GetCurrentDirectory()" + + #define ngx_path_separator(c) ((c) == '/' || (c) == '\\') #define NGX_HAVE_MAX_PATH 1 From pluknet at nginx.com Fri Feb 24 10:32:56 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Fri, 24 Feb 2023 10:32:56 +0000 Subject: [nginx] Win32: non-ASCII directory names support in ngx_create_dir(). Message-ID: details: https://hg.nginx.org/nginx/rev/e0f385521c79 branches: changeset: 8133:e0f385521c79 user: Maxim Dounin date: Thu Feb 23 20:49:45 2023 +0300 description: Win32: non-ASCII directory names support in ngx_create_dir(). This makes it possible to create directories under prefix with non-ASCII characters, as well as makes it possible to create directories with non-ASCII characters when using the dav module (ticket #1433). To ensure that the dav module operations are restricted similarly to other file operations (in particular, short names are not allowed), the ngx_win32_check_filename() function is used. It improved to support checking of just dirname, and now can be used to check paths when creating files or directories. diffstat: src/os/win32/ngx_files.c | 80 ++++++++++++++++++++++++++++++++++++++++++----- src/os/win32/ngx_files.h | 2 +- 2 files changed, 72 insertions(+), 10 deletions(-) diffs (197 lines): diff -r 8ea2e052feb4 -r e0f385521c79 src/os/win32/ngx_files.c --- a/src/os/win32/ngx_files.c Thu Feb 23 20:49:44 2023 +0300 +++ b/src/os/win32/ngx_files.c Thu Feb 23 20:49:45 2023 +0300 @@ -12,8 +12,8 @@ #define NGX_UTF16_BUFLEN 256 #define NGX_UTF8_BUFLEN 512 -static ngx_int_t ngx_win32_check_filename(u_char *name, u_short *u, - size_t len); +static ngx_int_t ngx_win32_check_filename(u_short *u, size_t len, + ngx_uint_t dirname); static u_short *ngx_utf8_to_utf16(u_short *utf16, u_char *utf8, size_t *len, size_t reserved); static u_char *ngx_utf16_to_utf8(u_char *utf8, u_short *utf16, size_t *len, @@ -42,7 +42,7 @@ ngx_open_file(u_char *name, u_long mode, fd = INVALID_HANDLE_VALUE; if (create == NGX_FILE_OPEN - && ngx_win32_check_filename(name, u, len) != NGX_OK) + && ngx_win32_check_filename(u, len, 0) != NGX_OK) { goto failed; } @@ -282,7 +282,7 @@ ngx_file_info(u_char *file, ngx_file_inf rc = NGX_FILE_ERROR; - if (ngx_win32_check_filename(file, u, len) != NGX_OK) { + if (ngx_win32_check_filename(u, len, 0) != NGX_OK) { goto failed; } @@ -478,7 +478,7 @@ ngx_open_dir(ngx_str_t *name, ngx_dir_t return NGX_ERROR; } - if (ngx_win32_check_filename(name->data, u, len) != NGX_OK) { + if (ngx_win32_check_filename(u, len, 0) != NGX_OK) { goto failed; } @@ -578,6 +578,42 @@ ngx_close_dir(ngx_dir_t *dir) ngx_int_t +ngx_create_dir(u_char *name, ngx_uint_t access) +{ + long rc; + size_t len; + u_short *u; + ngx_err_t err; + u_short utf16[NGX_UTF16_BUFLEN]; + + len = NGX_UTF16_BUFLEN; + u = ngx_utf8_to_utf16(utf16, name, &len, 0); + + if (u == NULL) { + return NGX_FILE_ERROR; + } + + rc = NGX_FILE_ERROR; + + if (ngx_win32_check_filename(u, len, 1) != NGX_OK) { + goto failed; + } + + rc = CreateDirectoryW(u, NULL); + +failed: + + if (u != utf16) { + err = ngx_errno; + ngx_free(u); + ngx_set_errno(err); + } + + return rc; +} + + +ngx_int_t ngx_open_glob(ngx_glob_t *gl) { u_char *p; @@ -791,11 +827,10 @@ ngx_fs_available(u_char *name) static ngx_int_t -ngx_win32_check_filename(u_char *name, u_short *u, size_t len) +ngx_win32_check_filename(u_short *u, size_t len, ngx_uint_t dirname) { - u_char *p, ch; u_long n; - u_short *lu; + u_short *lu, *p, *slash, ch; ngx_err_t err; enum { sw_start = 0, @@ -808,9 +843,14 @@ ngx_win32_check_filename(u_char *name, u /* check for NTFS streams (":"), trailing dots and spaces */ lu = NULL; + slash = NULL; state = sw_start; - for (p = name; *p; p++) { +#if (NGX_SUPPRESS_WARN) + ch = 0; +#endif + + for (p = u; *p; p++) { ch = *p; switch (state) { @@ -824,6 +864,7 @@ ngx_win32_check_filename(u_char *name, u if (ch == '/' || ch == '\\') { state = sw_after_slash; + slash = p; } break; @@ -842,6 +883,7 @@ ngx_win32_check_filename(u_char *name, u if (ch == '/' || ch == '\\') { state = sw_after_slash; + slash = p; break; } @@ -869,6 +911,7 @@ ngx_win32_check_filename(u_char *name, u if (ch == '/' || ch == '\\') { state = sw_after_slash; + slash = p; break; } @@ -897,6 +940,12 @@ ngx_win32_check_filename(u_char *name, u goto invalid; } + if (dirname && slash) { + ch = *slash; + *slash = '\0'; + len = slash - u + 1; + } + /* check if long name match */ lu = malloc(len * 2); @@ -907,6 +956,11 @@ ngx_win32_check_filename(u_char *name, u n = GetLongPathNameW(u, lu, len); if (n == 0) { + + if (dirname && slash && ngx_errno == NGX_ENOENT) { + ngx_set_errno(NGX_ENOPATH); + } + goto failed; } @@ -914,6 +968,10 @@ ngx_win32_check_filename(u_char *name, u goto invalid; } + if (dirname && slash) { + *slash = ch; + } + ngx_free(lu); return NGX_OK; @@ -924,6 +982,10 @@ invalid: failed: + if (dirname && slash) { + *slash = ch; + } + if (lu) { err = ngx_errno; ngx_free(lu); diff -r 8ea2e052feb4 -r e0f385521c79 src/os/win32/ngx_files.h --- a/src/os/win32/ngx_files.h Thu Feb 23 20:49:44 2023 +0300 +++ b/src/os/win32/ngx_files.h Thu Feb 23 20:49:45 2023 +0300 @@ -202,7 +202,7 @@ ngx_int_t ngx_close_dir(ngx_dir_t *dir); #define ngx_close_dir_n "FindClose()" -#define ngx_create_dir(name, access) CreateDirectory((const char *) name, NULL) +ngx_int_t ngx_create_dir(u_char *name, ngx_uint_t access); #define ngx_create_dir_n "CreateDirectory()" From pluknet at nginx.com Fri Feb 24 10:32:59 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Fri, 24 Feb 2023 10:32:59 +0000 Subject: [nginx] Win32: non-ASCII directory names support in ngx_delete_dir(). Message-ID: details: https://hg.nginx.org/nginx/rev/7d60e4584d9e branches: changeset: 8134:7d60e4584d9e user: Maxim Dounin date: Thu Feb 23 20:49:47 2023 +0300 description: Win32: non-ASCII directory names support in ngx_delete_dir(). This makes it possible to delete directories with non-ASCII characters when using the dav module (ticket #1433). diffstat: src/os/win32/ngx_files.c | 36 ++++++++++++++++++++++++++++++++++++ src/os/win32/ngx_files.h | 2 +- 2 files changed, 37 insertions(+), 1 deletions(-) diffs (58 lines): diff -r e0f385521c79 -r 7d60e4584d9e src/os/win32/ngx_files.c --- a/src/os/win32/ngx_files.c Thu Feb 23 20:49:45 2023 +0300 +++ b/src/os/win32/ngx_files.c Thu Feb 23 20:49:47 2023 +0300 @@ -614,6 +614,42 @@ failed: ngx_int_t +ngx_delete_dir(u_char *name) +{ + long rc; + size_t len; + u_short *u; + ngx_err_t err; + u_short utf16[NGX_UTF16_BUFLEN]; + + len = NGX_UTF16_BUFLEN; + u = ngx_utf8_to_utf16(utf16, name, &len, 0); + + if (u == NULL) { + return NGX_FILE_ERROR; + } + + rc = NGX_FILE_ERROR; + + if (ngx_win32_check_filename(u, len, 0) != NGX_OK) { + goto failed; + } + + rc = RemoveDirectoryW(u); + +failed: + + if (u != utf16) { + err = ngx_errno; + ngx_free(u); + ngx_set_errno(err); + } + + return rc; +} + + +ngx_int_t ngx_open_glob(ngx_glob_t *gl) { u_char *p; diff -r e0f385521c79 -r 7d60e4584d9e src/os/win32/ngx_files.h --- a/src/os/win32/ngx_files.h Thu Feb 23 20:49:45 2023 +0300 +++ b/src/os/win32/ngx_files.h Thu Feb 23 20:49:47 2023 +0300 @@ -206,7 +206,7 @@ ngx_int_t ngx_create_dir(u_char *name, n #define ngx_create_dir_n "CreateDirectory()" -#define ngx_delete_dir(name) RemoveDirectory((const char *) name) +ngx_int_t ngx_delete_dir(u_char *name); #define ngx_delete_dir_n "RemoveDirectory()" From pluknet at nginx.com Fri Feb 24 10:33:02 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Fri, 24 Feb 2023 10:33:02 +0000 Subject: [nginx] Win32: reworked ngx_win32_rename_file() to check errors. Message-ID: details: https://hg.nginx.org/nginx/rev/8880fe0b193c branches: changeset: 8135:8880fe0b193c user: Maxim Dounin date: Thu Feb 23 20:49:50 2023 +0300 description: Win32: reworked ngx_win32_rename_file() to check errors. Previously, ngx_win32_rename_file() retried on all errors returned by MoveFile() to a temporary name. It only make sense, however, to retry when the destination file already exists, similarly to the condition when ngx_win32_rename_file() is called. Retrying on other errors is meaningless and might result in an infinite loop. diffstat: src/os/win32/ngx_files.c | 12 ++++++++++-- 1 files changed, 10 insertions(+), 2 deletions(-) diffs (31 lines): diff -r 7d60e4584d9e -r 8880fe0b193c src/os/win32/ngx_files.c --- a/src/os/win32/ngx_files.c Thu Feb 23 20:49:47 2023 +0300 +++ b/src/os/win32/ngx_files.c Thu Feb 23 20:49:50 2023 +0300 @@ -236,10 +236,16 @@ ngx_win32_rename_file(ngx_str_t *from, n break; } - collision = 1; + err = ngx_errno; - ngx_log_error(NGX_LOG_CRIT, log, ngx_errno, + if (err == NGX_EEXIST || err == NGX_EEXIST_FILE) { + collision = 1; + continue; + } + + ngx_log_error(NGX_LOG_CRIT, log, err, "MoveFile() \"%s\" to \"%s\" failed", to->data, name); + goto failed; } if (MoveFile((const char *) from->data, (const char *) to->data) == 0) { @@ -254,6 +260,8 @@ ngx_win32_rename_file(ngx_str_t *from, n "DeleteFile() \"%s\" failed", name); } +failed: + /* mutex_unlock() */ ngx_free(name); From pluknet at nginx.com Fri Feb 24 10:33:05 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Fri, 24 Feb 2023 10:33:05 +0000 Subject: [nginx] Win32: reworked ngx_win32_rename_file() to use nginx wrappers. Message-ID: details: https://hg.nginx.org/nginx/rev/37a184966ab3 branches: changeset: 8136:37a184966ab3 user: Maxim Dounin date: Thu Feb 23 20:49:52 2023 +0300 description: Win32: reworked ngx_win32_rename_file() to use nginx wrappers. This ensures that ngx_win32_rename_file() will support non-ASCII names when supported by the wrappers. Notably, this is used by PUT requests in the dav module when overwriting existing files with non-ASCII names (ticket #1433). diffstat: src/os/win32/ngx_files.c | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diffs (29 lines): diff -r 8880fe0b193c -r 37a184966ab3 src/os/win32/ngx_files.c --- a/src/os/win32/ngx_files.c Thu Feb 23 20:49:50 2023 +0300 +++ b/src/os/win32/ngx_files.c Thu Feb 23 20:49:52 2023 +0300 @@ -232,7 +232,7 @@ ngx_win32_rename_file(ngx_str_t *from, n ngx_sprintf(name + to->len, ".%0muA.DELETE%Z", num); - if (MoveFile((const char *) to->data, (const char *) name) != 0) { + if (ngx_rename_file(to->data, name) != NGX_FILE_ERROR) { break; } @@ -248,14 +248,14 @@ ngx_win32_rename_file(ngx_str_t *from, n goto failed; } - if (MoveFile((const char *) from->data, (const char *) to->data) == 0) { + if (ngx_rename_file(from->data, to->data) == NGX_FILE_ERROR) { err = ngx_errno; } else { err = 0; } - if (DeleteFile((const char *) name) == 0) { + if (ngx_delete_file(name) == NGX_FILE_ERROR) { ngx_log_error(NGX_LOG_CRIT, log, ngx_errno, "DeleteFile() \"%s\" failed", name); } From pluknet at nginx.com Fri Feb 24 10:33:08 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Fri, 24 Feb 2023 10:33:08 +0000 Subject: [nginx] Win32: non-ASCII names support in ngx_delete_file(). Message-ID: details: https://hg.nginx.org/nginx/rev/4b06186670ed branches: changeset: 8137:4b06186670ed user: Maxim Dounin date: Thu Feb 23 20:49:54 2023 +0300 description: Win32: non-ASCII names support in ngx_delete_file(). This makes it possible to delete files with non-ASCII characters when using the dav module (ticket #1433). diffstat: src/os/win32/ngx_files.c | 36 ++++++++++++++++++++++++++++++++++++ src/os/win32/ngx_files.h | 2 +- 2 files changed, 37 insertions(+), 1 deletions(-) diffs (58 lines): diff -r 37a184966ab3 -r 4b06186670ed src/os/win32/ngx_files.c --- a/src/os/win32/ngx_files.c Thu Feb 23 20:49:52 2023 +0300 +++ b/src/os/win32/ngx_files.c Thu Feb 23 20:49:54 2023 +0300 @@ -207,6 +207,42 @@ ngx_write_console(ngx_fd_t fd, void *buf } +ngx_int_t +ngx_delete_file(u_char *name) +{ + long rc; + size_t len; + u_short *u; + ngx_err_t err; + u_short utf16[NGX_UTF16_BUFLEN]; + + len = NGX_UTF16_BUFLEN; + u = ngx_utf8_to_utf16(utf16, name, &len, 0); + + if (u == NULL) { + return NGX_FILE_ERROR; + } + + rc = NGX_FILE_ERROR; + + if (ngx_win32_check_filename(u, len, 0) != NGX_OK) { + goto failed; + } + + rc = DeleteFileW(u); + +failed: + + if (u != utf16) { + err = ngx_errno; + ngx_free(u); + ngx_set_errno(err); + } + + return rc; +} + + ngx_err_t ngx_win32_rename_file(ngx_str_t *from, ngx_str_t *to, ngx_log_t *log) { diff -r 37a184966ab3 -r 4b06186670ed src/os/win32/ngx_files.h --- a/src/os/win32/ngx_files.h Thu Feb 23 20:49:52 2023 +0300 +++ b/src/os/win32/ngx_files.h Thu Feb 23 20:49:54 2023 +0300 @@ -123,7 +123,7 @@ ssize_t ngx_write_console(ngx_fd_t fd, v #define NGX_LINEFEED CRLF -#define ngx_delete_file(name) DeleteFile((const char *) name) +ngx_int_t ngx_delete_file(u_char *name); #define ngx_delete_file_n "DeleteFile()" From pluknet at nginx.com Fri Feb 24 10:33:11 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Fri, 24 Feb 2023 10:33:11 +0000 Subject: [nginx] Win32: non-ASCII names support in ngx_rename_file(). Message-ID: details: https://hg.nginx.org/nginx/rev/96d894b38667 branches: changeset: 8138:96d894b38667 user: Maxim Dounin date: Thu Feb 23 20:49:55 2023 +0300 description: Win32: non-ASCII names support in ngx_rename_file(). This makes it possible to upload files with non-ASCII characters when using the dav module (ticket #1433). diffstat: src/os/win32/ngx_files.c | 55 ++++++++++++++++++++++++++++++++++++++++++++++++ src/os/win32/ngx_files.h | 2 +- 2 files changed, 56 insertions(+), 1 deletions(-) diffs (77 lines): diff -r 4b06186670ed -r 96d894b38667 src/os/win32/ngx_files.c --- a/src/os/win32/ngx_files.c Thu Feb 23 20:49:54 2023 +0300 +++ b/src/os/win32/ngx_files.c Thu Feb 23 20:49:55 2023 +0300 @@ -243,6 +243,61 @@ failed: } +ngx_int_t +ngx_rename_file(u_char *from, u_char *to) +{ + long rc; + size_t len; + u_short *fu, *tu; + ngx_err_t err; + u_short utf16f[NGX_UTF16_BUFLEN]; + u_short utf16t[NGX_UTF16_BUFLEN]; + + len = NGX_UTF16_BUFLEN; + fu = ngx_utf8_to_utf16(utf16f, from, &len, 0); + + if (fu == NULL) { + return NGX_FILE_ERROR; + } + + rc = NGX_FILE_ERROR; + tu = NULL; + + if (ngx_win32_check_filename(fu, len, 0) != NGX_OK) { + goto failed; + } + + len = NGX_UTF16_BUFLEN; + tu = ngx_utf8_to_utf16(utf16t, to, &len, 0); + + if (tu == NULL) { + goto failed; + } + + if (ngx_win32_check_filename(tu, len, 1) != NGX_OK) { + goto failed; + } + + rc = MoveFileW(fu, tu); + +failed: + + if (fu != utf16f) { + err = ngx_errno; + ngx_free(fu); + ngx_set_errno(err); + } + + if (tu && tu != utf16t) { + err = ngx_errno; + ngx_free(tu); + ngx_set_errno(err); + } + + return rc; +} + + ngx_err_t ngx_win32_rename_file(ngx_str_t *from, ngx_str_t *to, ngx_log_t *log) { diff -r 4b06186670ed -r 96d894b38667 src/os/win32/ngx_files.h --- a/src/os/win32/ngx_files.h Thu Feb 23 20:49:54 2023 +0300 +++ b/src/os/win32/ngx_files.h Thu Feb 23 20:49:55 2023 +0300 @@ -127,7 +127,7 @@ ngx_int_t ngx_delete_file(u_char *name); #define ngx_delete_file_n "DeleteFile()" -#define ngx_rename_file(o, n) MoveFile((const char *) o, (const char *) n) +ngx_int_t ngx_rename_file(u_char *from, u_char *to); #define ngx_rename_file_n "MoveFile()" ngx_err_t ngx_win32_rename_file(ngx_str_t *from, ngx_str_t *to, ngx_log_t *log); From pluknet at nginx.com Fri Feb 24 10:33:14 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Fri, 24 Feb 2023 10:33:14 +0000 Subject: [nginx] Win32: non-ASCII names support in ngx_open_tempfile(). Message-ID: details: https://hg.nginx.org/nginx/rev/e818ed227735 branches: changeset: 8139:e818ed227735 user: Maxim Dounin date: Thu Feb 23 20:49:57 2023 +0300 description: Win32: non-ASCII names support in ngx_open_tempfile(). This makes it possible to use temporary directories with non-ASCII characters, either explicitly or via a prefix with non-ASCII characters in it. diffstat: src/os/win32/ngx_files.c | 35 +++++++++++++++++++++++++++++++++++ src/os/win32/ngx_files.h | 12 ++---------- 2 files changed, 37 insertions(+), 10 deletions(-) diffs (67 lines): diff -r 96d894b38667 -r e818ed227735 src/os/win32/ngx_files.c --- a/src/os/win32/ngx_files.c Thu Feb 23 20:49:55 2023 +0300 +++ b/src/os/win32/ngx_files.c Thu Feb 23 20:49:57 2023 +0300 @@ -63,6 +63,41 @@ failed: } +ngx_fd_t +ngx_open_tempfile(u_char *name, ngx_uint_t persistent, ngx_uint_t access) +{ + size_t len; + u_short *u; + ngx_fd_t fd; + ngx_err_t err; + u_short utf16[NGX_UTF16_BUFLEN]; + + len = NGX_UTF16_BUFLEN; + u = ngx_utf8_to_utf16(utf16, name, &len, 0); + + if (u == NULL) { + return INVALID_HANDLE_VALUE; + } + + fd = CreateFileW(u, + GENERIC_READ|GENERIC_WRITE, + FILE_SHARE_READ|FILE_SHARE_WRITE|FILE_SHARE_DELETE, + NULL, + CREATE_NEW, + persistent ? 0: + FILE_ATTRIBUTE_TEMPORARY|FILE_FLAG_DELETE_ON_CLOSE, + NULL); + + if (u != utf16) { + err = ngx_errno; + ngx_free(u); + ngx_set_errno(err); + } + + return fd; +} + + ssize_t ngx_read_file(ngx_file_t *file, u_char *buf, size_t size, off_t offset) { diff -r 96d894b38667 -r e818ed227735 src/os/win32/ngx_files.h --- a/src/os/win32/ngx_files.h Thu Feb 23 20:49:55 2023 +0300 +++ b/src/os/win32/ngx_files.h Thu Feb 23 20:49:57 2023 +0300 @@ -90,16 +90,8 @@ ngx_fd_t ngx_open_file(u_char *name, u_l #define NGX_FILE_OWNER_ACCESS 0 -#define ngx_open_tempfile(name, persistent, access) \ - CreateFile((const char *) name, \ - GENERIC_READ|GENERIC_WRITE, \ - FILE_SHARE_READ|FILE_SHARE_WRITE|FILE_SHARE_DELETE, \ - NULL, \ - CREATE_NEW, \ - persistent ? 0: \ - FILE_ATTRIBUTE_TEMPORARY|FILE_FLAG_DELETE_ON_CLOSE, \ - NULL); - +ngx_fd_t ngx_open_tempfile(u_char *name, ngx_uint_t persistent, + ngx_uint_t access); #define ngx_open_tempfile_n "CreateFile()" From pluknet at nginx.com Fri Feb 24 10:33:16 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Fri, 24 Feb 2023 10:33:16 +0000 Subject: [nginx] Win32: removed attempt to use a drive letter in ngx_fs_bsize(). Message-ID: details: https://hg.nginx.org/nginx/rev/c33eb93f9c7a branches: changeset: 8140:c33eb93f9c7a user: Maxim Dounin date: Thu Feb 23 20:50:00 2023 +0300 description: Win32: removed attempt to use a drive letter in ngx_fs_bsize(). Just a drive letter might not correctly represent file system being used, notably when using symlinks (as created by "mklink /d"). As such, instead of trying to call GetDiskFreeSpace() with just a drive letter, we now always use GetDiskFreeSpace() with full path. Further, it looks like the code to use just a drive letter never worked, since it tried to test name[2] instead of name[1] to be ':'. diffstat: src/os/win32/ngx_files.c | 6 ------ 1 files changed, 0 insertions(+), 6 deletions(-) diffs (18 lines): diff -r e818ed227735 -r c33eb93f9c7a src/os/win32/ngx_files.c --- a/src/os/win32/ngx_files.c Thu Feb 23 20:49:57 2023 +0300 +++ b/src/os/win32/ngx_files.c Thu Feb 23 20:50:00 2023 +0300 @@ -967,14 +967,8 @@ ngx_directio_off(ngx_fd_t fd) size_t ngx_fs_bsize(u_char *name) { - u_char root[4]; u_long sc, bs, nfree, ncl; - if (name[2] == ':') { - ngx_cpystrn(root, name, 4); - name = root; - } - if (GetDiskFreeSpace((const char *) name, &sc, &bs, &nfree, &ncl) == 0) { return 512; } From pluknet at nginx.com Fri Feb 24 10:33:19 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Fri, 24 Feb 2023 10:33:19 +0000 Subject: [nginx] Win32: non-ASCII names in ngx_fs_bsize(), ngx_fs_available(). Message-ID: details: https://hg.nginx.org/nginx/rev/2acb00b9b5ff branches: changeset: 8141:2acb00b9b5ff user: Maxim Dounin date: Thu Feb 23 20:50:03 2023 +0300 description: Win32: non-ASCII names in ngx_fs_bsize(), ngx_fs_available(). This fixes potentially incorrect cache size calculations and non-working "min_free" when using cache in directories with non-ASCII names. diffstat: src/os/win32/ngx_files.c | 46 ++++++++++++++++++++++++++++++++++++++++++---- 1 files changed, 42 insertions(+), 4 deletions(-) diffs (71 lines): diff -r c33eb93f9c7a -r 2acb00b9b5ff src/os/win32/ngx_files.c --- a/src/os/win32/ngx_files.c Thu Feb 23 20:50:00 2023 +0300 +++ b/src/os/win32/ngx_files.c Thu Feb 23 20:50:03 2023 +0300 @@ -967,12 +967,31 @@ ngx_directio_off(ngx_fd_t fd) size_t ngx_fs_bsize(u_char *name) { - u_long sc, bs, nfree, ncl; + u_long sc, bs, nfree, ncl; + size_t len; + u_short *u; + u_short utf16[NGX_UTF16_BUFLEN]; + + len = NGX_UTF16_BUFLEN; + u = ngx_utf8_to_utf16(utf16, name, &len, 0); + + if (u == NULL) { + return 512; + } - if (GetDiskFreeSpace((const char *) name, &sc, &bs, &nfree, &ncl) == 0) { + if (GetDiskFreeSpaceW(u, &sc, &bs, &nfree, &ncl) == 0) { + + if (u != utf16) { + ngx_free(u); + } + return 512; } + if (u != utf16) { + ngx_free(u); + } + return sc * bs; } @@ -980,12 +999,31 @@ ngx_fs_bsize(u_char *name) off_t ngx_fs_available(u_char *name) { - ULARGE_INTEGER navail; + size_t len; + u_short *u; + ULARGE_INTEGER navail; + u_short utf16[NGX_UTF16_BUFLEN]; + + len = NGX_UTF16_BUFLEN; + u = ngx_utf8_to_utf16(utf16, name, &len, 0); + + if (u == NULL) { + return NGX_MAX_OFF_T_VALUE; + } - if (GetDiskFreeSpaceEx((const char *) name, &navail, NULL, NULL) == 0) { + if (GetDiskFreeSpaceExW(u, &navail, NULL, NULL) == 0) { + + if (u != utf16) { + ngx_free(u); + } + return NGX_MAX_OFF_T_VALUE; } + if (u != utf16) { + ngx_free(u); + } + return (off_t) navail.QuadPart; } From pluknet at nginx.com Fri Feb 24 10:40:17 2023 From: pluknet at nginx.com (Sergey Kandaurov) Date: Fri, 24 Feb 2023 14:40:17 +0400 Subject: [PATCH 11 of 12] Win32: fixed ngx_fs_bsize() for symlinks In-Reply-To: References: <5F74999F-228E-43AB-B059-879964530E55@nginx.com> Message-ID: <5EC16E2F-2F2D-4049-A734-6C734B7AE541@nginx.com> > On 23 Feb 2023, at 22:46, Maxim Dounin wrote: > > Hello! > > On Wed, Feb 22, 2023 at 08:01:15PM +0400, Sergey Kandaurov wrote: > >>> On 19 Feb 2023, at 21:23, Maxim Dounin wrote: >>> >>> Hello! >>> >>> On Fri, Feb 17, 2023 at 07:17:02PM +0400, Sergey Kandaurov wrote: >>> >>>>> On 13 Jan 2023, at 01:35, Maxim Dounin wrote: >>>>> >>>>> # HG changeset patch >>>>> # User Maxim Dounin >>>>> # Date 1673549010 -10800 >>>>> # Thu Jan 12 21:43:30 2023 +0300 >>>>> # Node ID be7eb9ec28dcbfdfd2e850befc8d051c0e4d46fd >>>>> # Parent e62c8e9724ba68a698a2c3613edca73fe4e1c4ae >>>>> Win32: fixed ngx_fs_bsize() for symlinks. >>>>> >>>>> Just a drive letter might not correctly represent file system being used, >>>>> notably when using symlinks (as created by "mklink /d"). As such, instead >>>>> of calling GetDiskFreeSpace() with just a drive letter, we now always >>>>> use GetDiskFreeSpace() with full path. >>>>> >>>>> diff -r e62c8e9724ba -r be7eb9ec28dc src/os/win32/ngx_files.c >>>>> --- a/src/os/win32/ngx_files.c Thu Jan 12 21:43:14 2023 +0300 >>>>> +++ b/src/os/win32/ngx_files.c Thu Jan 12 21:43:30 2023 +0300 >>>>> @@ -955,14 +955,8 @@ ngx_directio_off(ngx_fd_t fd) >>>>> size_t >>>>> ngx_fs_bsize(u_char *name) >>>>> { >>>>> - u_char root[4]; >>>>> u_long sc, bs, nfree, ncl; >>>>> >>>>> - if (name[2] == ':') { >>>>> - ngx_cpystrn(root, name, 4); >>>>> - name = root; >>>>> - } >>>>> - >>>> >>>> BTW, I wonder how this condition could be true. >>>> Specifically, what name should represent in order to match. >>>> I'm happy that it's leaving though. >>> >>> I tend to think that this actually never worked, and the original >>> intention was to test name[1] instead. >>> >>> Updated commit log: >>> >>> : Win32: removed attempt to use a drive letter in ngx_fs_bsize(). >>> : >>> : Just a drive letter might not correctly represent file system being used, >>> : notably when using symlinks (as created by "mklink /d"). As such, instead >>> : of trying to call GetDiskFreeSpace() with just a drive letter, we now always >>> : use GetDiskFreeSpace() with full path. >>> : >>> : Further, it looks like the code to use just a drive letter never worked, >>> : since it tried to test name[2] instead of name[1] to be ':'. >>> >>> [...] >>> >> >> Looks fine, thanks. > > Thanks for the review, pushed to http://mdounin.ru/hg/nginx. Tests for autoindex and dav modules for your consideration. Largely based on existing tests. # HG changeset patch # User Sergey Kandaurov # Date 1677235010 -14400 # Fri Feb 24 14:36:50 2023 +0400 # Node ID 0e2ad324773111b2b2798f6926ce58a7ba93b30e # Parent 4d44ea5ccca73813f557bda1fb4d024aaa4c9eea Tests: win32 autoindex tests. diff --git a/autoindex_win32.t b/autoindex_win32.t new file mode 100644 --- /dev/null +++ b/autoindex_win32.t @@ -0,0 +1,119 @@ +#!/usr/bin/perl + +# (C) Sergey Kandaurov +# (C) Nginx, Inc. + +# Tests for autoindex module. + +############################################################################### + +use warnings; +use strict; + +use Test::More; + +use Encode qw/ encode /; + +BEGIN { use FindBin; chdir($FindBin::Bin); } + +use lib 'lib'; +use Test::Nginx; + +############################################################################### + +select STDERR; $| = 1; +select STDOUT; $| = 1; + +eval { require Win32API::File; }; +plan(skip_all => 'Win32API::File not installed') if $@; + +my $t = Test::Nginx->new()->has(qw/http autoindex charset/)->plan(9) + ->write_file_expand('nginx.conf', <<'EOF'); + +%%TEST_GLOBALS%% + +daemon off; + +events { +} + +http { + %%TEST_GLOBALS_HTTP%% + + server { + listen 127.0.0.1:8080; + server_name localhost; + + location / { + autoindex on; + charset utf-8; + } + } +} + +EOF + +my $d = $t->testdir(); + +mkdir("$d/test-dir"); +$t->write_file('test-file', ''); + +my $file = "$d/test-file-" . ("\x{043c}\x{0438}") x 3; +win32_write_file($file, ''); + +my $dir = "$d/test-dir-" . ("\x{043c}\x{0438}") x 3; +win32_mkdir($dir); + +my $subfile = "$dir/test-subfile-" . ("\x{043c}\x{0438}") x 3; +win32_write_file($subfile, ''); + +$t->run(); + +############################################################################### + +my $r = http_get('/'); + +like($r, qr!href="test-file"!ms, 'file'); +like($r, qr!href="test-dir/"!ms, 'directory'); + +TODO: { +local $TODO = 'not yet' unless $t->has_version('1.23.4'); + +like($r, qr!href="test-file-(%d0%bc%d0%b8){3}"!msi, 'utf file link'); +like($r, qr!test-file-(\xd0\xbc\xd0\xb8){3}!ms, 'utf file name'); + +like($r, qr!href="test-dir-(%d0%bc%d0%b8){3}/"!msi, 'utf dir link'); +like($r, qr!test-dir-(\xd0\xbc\xd0\xb8){3}/!ms, 'utf dir name'); + +$r = http_get('/test-dir-' . "\xd0\xbc\xd0\xb8" x 3 . '/'); + +like($r, qr!Index of /test-dir-(\xd0\xbc\xd0\xb8){3}/!msi, 'utf subdir index'); + +like($r, qr!href="test-subfile-(%d0%bc%d0%b8){3}"!msi, 'utf subdir link'); +like($r, qr!test-subfile-(\xd0\xbc\xd0\xb8){3}!msi, 'utf subdir name'); + +} + +############################################################################### + +sub win32_mkdir { + my ($name) = @_; + + mkdir("$d/test-dir-tmp"); + Win32API::File::MoveFileW(encode("UTF-16LE","$d/test-dir-tmp\0"), + encode("UTF-16LE", $name . "\0")) or die "$^E"; +} + +sub win32_write_file { + my ($name, $data) = @_; + + my $h = Win32API::File::CreateFileW(encode("UTF-16LE", $name . "\0"), + Win32API::File::FILE_READ_DATA() + | Win32API::File::FILE_WRITE_DATA(), 0, [], + Win32API::File::CREATE_NEW(), 0, []) or die $^E; + + Win32API::File::WriteFile($h, $data, 0, [], []) or die $^E; + Win32API::File::CloseHandle($h) or die $^E; +} + +############################################################################### # HG changeset patch # User Sergey Kandaurov # Date 1677235049 -14400 # Fri Feb 24 14:37:29 2023 +0400 # Node ID 9378b768e0668dbe080d301818b394e71a2bb84d # Parent 0e2ad324773111b2b2798f6926ce58a7ba93b30e Tests: dav tests with UTF-8. diff --git a/dav_utf8.t b/dav_utf8.t new file mode 100644 --- /dev/null +++ b/dav_utf8.t @@ -0,0 +1,229 @@ +#!/usr/bin/perl + +# (C) Maxim Dounin +# (C) Sergey Kandaurov +# (C) Nginx, Inc. + +# Tests for nginx dav module with utf8 encoded names. + +############################################################################### + +use warnings; +use strict; + +use Test::More; + +use Encode qw/ encode /; + +BEGIN { use FindBin; chdir($FindBin::Bin); } + +use lib 'lib'; +use Test::Nginx; + +############################################################################### + +select STDERR; $| = 1; +select STDOUT; $| = 1; + +eval { require Win32API::File; }; +plan(skip_all => 'Win32API::File not installed') if $@ && $^O eq 'MSWin32'; + +my $t = Test::Nginx->new()->has(qw/http dav/)->plan(18); + +$t->write_file_expand('nginx.conf', <<'EOF'); + +%%TEST_GLOBALS%% + +daemon off; + +events { +} + +http { + %%TEST_GLOBALS_HTTP%% + + server { + listen 127.0.0.1:8080; + server_name localhost; + + absolute_redirect off; + + location / { + dav_methods PUT DELETE MKCOL COPY MOVE; + } + + location /i/ { + alias %%TESTDIR%%/; + dav_methods PUT DELETE MKCOL COPY MOVE; + } + } +} + +EOF + +$t->run(); + +############################################################################### + +local $TODO = 'not yet' unless $t->has_version('1.23.4'); + +my $r; + +my $file = "file-%D0%BC%D0%B8"; +my $file_path = "file-\x{043c}\x{0438}"; + +$r = http(<testdir() . '/' . $file_path), 10, 'put file size'); + +$r = http(<testdir() . '/' . $file_path), 0, 'put file again size'); + +$r = http(<testdir() . '/' . $file_path), 'file deleted'); + +$r = http(<testdir() . '/' . $file_path), 10, + 'put file extra data size'); + +TODO: { +local $TODO = 'not yet' unless $t->has_version('1.21.0'); + +$r = http(<testdir() . "/$file_path-moved escape"), 10, + 'file copied unescaped'); + +my $dir_path = "dir-\x{043c}\x{0438}"; +my $dir = "dir-%D0%BC%D0%B8"; + +# 201 replies contain body, response should indicate it's empty + +$r = http(< details: https://hg.nginx.org/njs/rev/0f5b52a42f97 branches: changeset: 2050:0f5b52a42f97 user: Dmitry Volyntsev date: Mon Feb 27 18:16:01 2023 -0800 description: Stream: added missed linking with libxml2 for the dynamic module. The issue was introduced in 99b9f83e4d4d (0.7.10). diffstat: nginx/config | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff -r 3891f338e2c9 -r 0f5b52a42f97 nginx/config --- a/nginx/config Wed Feb 22 19:13:08 2023 -0800 +++ b/nginx/config Mon Feb 27 18:16:01 2023 -0800 @@ -29,7 +29,7 @@ if [ $STREAM != NO ]; then ngx_module_incs="$ngx_addon_dir/../src $ngx_addon_dir/../build" ngx_module_deps="$ngx_addon_dir/../build/libnjs.a $NJS_DEPS" ngx_module_srcs="$ngx_addon_dir/ngx_stream_js_module.c $NJS_SRCS" - ngx_module_libs="PCRE OPENSSL $ngx_addon_dir/../build/libnjs.a -lm" + ngx_module_libs="PCRE OPENSSL LIBXSLT $ngx_addon_dir/../build/libnjs.a -lm" . auto/module fi From v.zhestikov at f5.com Tue Feb 28 02:41:38 2023 From: v.zhestikov at f5.com (Vadim Zhestikov) Date: Tue, 28 Feb 2023 02:41:38 +0000 Subject: [njs] Parser: fixed the detection of await in arguments. Message-ID: details: https://hg.nginx.org/njs/rev/82e673ec5f19 branches: changeset: 2051:82e673ec5f19 user: Vadim Zhestikov date: Mon Feb 27 18:39:44 2023 -0800 description: Parser: fixed the detection of await in arguments. This fixes #619 issue on Github. diffstat: src/njs_parser.c | 9 +++------ src/test/njs_unit_test.c | 3 +++ 2 files changed, 6 insertions(+), 6 deletions(-) diffs (37 lines): diff -r 0f5b52a42f97 -r 82e673ec5f19 src/njs_parser.c --- a/src/njs_parser.c Mon Feb 27 18:16:01 2023 -0800 +++ b/src/njs_parser.c Mon Feb 27 18:39:44 2023 -0800 @@ -3565,17 +3565,14 @@ njs_parser_await(njs_parser_t *parser, n njs_queue_link_t *current) { njs_parser_node_t *node; - njs_parser_scope_t *scope; - - scope = njs_function_scope(parser->scope); - - if (!scope->async) { + + if (!njs_function_scope(parser->scope)->async) { njs_parser_syntax_error(parser, "await is only valid in async functions"); return NJS_ERROR; } - if (scope->in_args) { + if (parser->scope->in_args) { njs_parser_syntax_error(parser, "await in arguments not supported"); return NJS_ERROR; } diff -r 0f5b52a42f97 -r 82e673ec5f19 src/test/njs_unit_test.c --- a/src/test/njs_unit_test.c Mon Feb 27 18:16:01 2023 -0800 +++ b/src/test/njs_unit_test.c Mon Feb 27 18:39:44 2023 -0800 @@ -19824,6 +19824,9 @@ static njs_unit_test_t njs_test[] = "(async function() {f('Number: ' + await 111)})"), njs_str("SyntaxError: await in arguments not supported in 1") }, + { njs_str("async function f1() {try {f(await f1)} catch(e) {}}"), + njs_str("SyntaxError: await in arguments not supported in 1") }, + { njs_str("async function af() {await encrypt({},}"), njs_str("SyntaxError: Unexpected token \"}\" in 1") }, From xeioex at nginx.com Tue Feb 28 03:09:12 2023 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Tue, 28 Feb 2023 03:09:12 +0000 Subject: [njs] Unit tests: replacing njs_vm_retval_string() with njs_vm_retval_dump(). Message-ID: details: https://hg.nginx.org/njs/rev/e92881f2b395 branches: changeset: 2052:e92881f2b395 user: Dmitry Volyntsev date: Mon Feb 27 19:05:03 2023 -0800 description: Unit tests: replacing njs_vm_retval_string() with njs_vm_retval_dump(). The change allows to cover CLI behaviour more closely. diffstat: src/test/njs_unit_test.c | 20 ++++++++++---------- 1 files changed, 10 insertions(+), 10 deletions(-) diffs (71 lines): diff -r 82e673ec5f19 -r e92881f2b395 src/test/njs_unit_test.c --- a/src/test/njs_unit_test.c Mon Feb 27 18:39:44 2023 -0800 +++ b/src/test/njs_unit_test.c Mon Feb 27 19:05:03 2023 -0800 @@ -22823,12 +22823,12 @@ static njs_unit_test_t njs_shell_test[] { njs_str("var a = \"aa\\naa\"" ENTER "a" ENTER), - njs_str("aa\naa") }, + njs_str("'aa\\naa'") }, { njs_str("var a = 3" ENTER "var a = 'str'" ENTER "a" ENTER), - njs_str("str") }, + njs_str("'str'") }, { njs_str("var a = 2" ENTER "a *= 2" ENTER @@ -22862,7 +22862,7 @@ static njs_unit_test_t njs_shell_test[] "case 0: a += '0';" "case 1: a += '1';" "}; a" ENTER), - njs_str("A") }, + njs_str("'A'") }, { njs_str("var a = 0; try { a = 5 }" "catch (e) { a = 9 } finally { a++ } a" ENTER), @@ -22879,7 +22879,7 @@ static njs_unit_test_t njs_shell_test[] { njs_str("Number.prototype.test = 'test'" ENTER "Number.prototype.test" ENTER), - njs_str("test") }, + njs_str("'test'") }, { njs_str("function f(a) {return a}" ENTER "function f(a) {return a}; f(2)" ENTER), @@ -22929,11 +22929,12 @@ static njs_unit_test_t njs_shell_test[] "function(){}()" ENTER), njs_str("SyntaxError: Unexpected token \"(\" in 1") }, - /* Exception in njs_vm_retval_string() */ + { njs_str("var o = { toString: function() { return [1] } }; o" ENTER), + njs_str("{\n toString: [Function: toString]\n}") }, { njs_str("var o = { toString: function() { return [1] } }" ENTER - "o" ENTER), - njs_str("TypeError: Cannot convert object to primitive value") }, + "o.toString()" ENTER), + njs_str("[\n 1\n]") }, }; @@ -23630,8 +23631,8 @@ njs_interactive_test(njs_unit_test_t tes } } - if (njs_vm_retval_string(vm, &s) != NJS_OK) { - njs_printf("njs_vm_retval_string() failed\n"); + if (njs_vm_retval_dump(vm, &s, 0) != NJS_OK) { + njs_printf("njs_vm_retval_dump() failed\n"); goto done; } @@ -24956,7 +24957,6 @@ static njs_test_suite_t njs_suites[] = { .externals = 1, .repeat = 1, .unsafe = 1 }, njs_shell_test, njs_nitems(njs_shell_test), - njs_interactive_test }, { njs_str("backtraces"), From xeioex at nginx.com Tue Feb 28 06:15:41 2023 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Tue, 28 Feb 2023 06:15:41 +0000 Subject: [njs] Fixed Error() instance dumping when "name" prop is not primitive. Message-ID: details: https://hg.nginx.org/njs/rev/1aa137411b09 branches: changeset: 2053:1aa137411b09 user: Dmitry Volyntsev date: Mon Feb 27 22:14:34 2023 -0800 description: Fixed Error() instance dumping when "name" prop is not primitive. Previously, njs_error_to_string2() might be invoked with error argument pointing to vm->retval. When "name" prop was not primitive vm->retval might be overwritten. As a result error pointer might be referencing a primitive value. In turn the second call of njs_object_property() received an invalid object pointer because it expects only object value types. The fix is to ensure that error object pointer is never overwritten. This closes #615 issue on Github. diffstat: src/njs_array.c | 5 +++-- src/njs_date.c | 5 +++-- src/njs_error.c | 14 ++++++++++++-- src/njs_json.c | 18 +++++++++++------- src/njs_object.h | 2 +- src/njs_object_prop.c | 26 ++++++++++++++------------ src/njs_value.c | 2 +- src/test/njs_unit_test.c | 3 +++ 8 files changed, 48 insertions(+), 27 deletions(-) diffs (254 lines): diff -r e92881f2b395 -r 1aa137411b09 src/njs_array.c --- a/src/njs_array.c Mon Feb 27 19:05:03 2023 -0800 +++ b/src/njs_array.c Mon Feb 27 22:14:34 2023 -0800 @@ -1404,10 +1404,11 @@ njs_array_prototype_to_string(njs_vm_t * static const njs_value_t join_string = njs_string("join"); - if (njs_is_object(&args[0])) { + if (njs_is_object(njs_argument(args, 0))) { njs_object_property_init(&lhq, &join_string, NJS_JOIN_HASH); - ret = njs_object_property(vm, &args[0], &lhq, &value); + ret = njs_object_property(vm, njs_object(njs_argument(args, 0)), &lhq, + &value); if (njs_slow_path(ret == NJS_ERROR)) { return ret; diff -r e92881f2b395 -r 1aa137411b09 src/njs_date.c --- a/src/njs_date.c Mon Feb 27 19:05:03 2023 -0800 +++ b/src/njs_date.c Mon Feb 27 22:14:34 2023 -0800 @@ -1385,10 +1385,11 @@ njs_date_prototype_to_json(njs_vm_t *vm, static const njs_value_t to_iso_string = njs_string("toISOString"); - if (njs_is_object(&args[0])) { + if (njs_is_object(njs_argument(args, 0))) { njs_object_property_init(&lhq, &to_iso_string, NJS_TO_ISO_STRING_HASH); - ret = njs_object_property(vm, &args[0], &lhq, &value); + ret = njs_object_property(vm, njs_object(njs_argument(args, 0)), &lhq, + &value); if (njs_slow_path(ret == NJS_ERROR)) { return ret; diff -r e92881f2b395 -r 1aa137411b09 src/njs_error.c --- a/src/njs_error.c Mon Feb 27 19:05:03 2023 -0800 +++ b/src/njs_error.c Mon Feb 27 22:14:34 2023 -0800 @@ -574,6 +574,7 @@ njs_error_to_string2(njs_vm_t *vm, njs_v size_t length; u_char *p; njs_int_t ret; + njs_object_t *error_object; njs_value_t value1, value2; njs_value_t *name_value, *message_value; njs_string_prop_t name, message; @@ -581,6 +582,8 @@ njs_error_to_string2(njs_vm_t *vm, njs_v static const njs_value_t default_name = njs_string("Error"); + njs_assert(njs_is_object(error)); + if (want_stack) { ret = njs_error_stack(vm, njs_value_arg(error), retval); if (njs_slow_path(ret == NJS_ERROR)) { @@ -592,9 +595,11 @@ njs_error_to_string2(njs_vm_t *vm, njs_v } } + error_object = njs_object(error); + njs_object_property_init(&lhq, &njs_string_name, NJS_NAME_HASH); - ret = njs_object_property(vm, error, &lhq, &value1); + ret = njs_object_property(vm, error_object, &lhq, &value1); if (njs_slow_path(ret == NJS_ERROR)) { return ret; @@ -616,7 +621,7 @@ njs_error_to_string2(njs_vm_t *vm, njs_v lhq.key_hash = NJS_MESSAGE_HASH; lhq.key = njs_str_value("message"); - ret = njs_object_property(vm, error, &lhq, &value2); + ret = njs_object_property(vm, error_object, &lhq, &value2); if (njs_slow_path(ret == NJS_ERROR)) { return ret; @@ -686,6 +691,11 @@ njs_error_prototype_to_string(njs_vm_t * njs_int_t njs_error_to_string(njs_vm_t *vm, njs_value_t *retval, const njs_value_t *error) { + if (njs_slow_path(!njs_is_object(error))) { + njs_type_error(vm, "\"error\" is not an object"); + return NJS_ERROR; + } + return njs_error_to_string2(vm, retval, error, 1); } diff -r e92881f2b395 -r 1aa137411b09 src/njs_json.c --- a/src/njs_json.c Mon Feb 27 19:05:03 2023 -0800 +++ b/src/njs_json.c Mon Feb 27 22:14:34 2023 -0800 @@ -1252,15 +1252,19 @@ njs_object_to_json_function(njs_vm_t *vm static const njs_value_t to_json_string = njs_string("toJSON"); - njs_object_property_init(&lhq, &to_json_string, NJS_TO_JSON_HASH); - - ret = njs_object_property(vm, value, &lhq, &retval); - - if (njs_slow_path(ret == NJS_ERROR)) { - return NULL; + if (njs_is_object(value)) { + njs_object_property_init(&lhq, &to_json_string, NJS_TO_JSON_HASH); + + ret = njs_object_property(vm, njs_object(value), &lhq, &retval); + + if (njs_slow_path(ret == NJS_ERROR)) { + return NULL; + } + + return njs_is_function(&retval) ? njs_function(&retval) : NULL; } - return njs_is_function(&retval) ? njs_function(&retval) : NULL; + return NULL; } diff -r e92881f2b395 -r 1aa137411b09 src/njs_object.h --- a/src/njs_object.h Mon Feb 27 19:05:03 2023 -0800 +++ b/src/njs_object.h Mon Feb 27 22:14:34 2023 -0800 @@ -98,7 +98,7 @@ njs_int_t njs_prop_private_copy(njs_vm_t njs_object_t *proto); njs_object_prop_t *njs_object_prop_alloc(njs_vm_t *vm, const njs_value_t *name, const njs_value_t *value, uint8_t attributes); -njs_int_t njs_object_property(njs_vm_t *vm, const njs_value_t *value, +njs_int_t njs_object_property(njs_vm_t *vm, njs_object_t *object, njs_lvlhsh_query_t *lhq, njs_value_t *retval); njs_object_prop_t *njs_object_property_add(njs_vm_t *vm, njs_value_t *object, njs_value_t *key, njs_bool_t replace); diff -r e92881f2b395 -r 1aa137411b09 src/njs_object_prop.c --- a/src/njs_object_prop.c Mon Feb 27 19:05:03 2023 -0800 +++ b/src/njs_object_prop.c Mon Feb 27 22:14:34 2023 -0800 @@ -91,15 +91,13 @@ njs_object_prop_alloc2(njs_vm_t *vm, con njs_int_t -njs_object_property(njs_vm_t *vm, const njs_value_t *value, - njs_lvlhsh_query_t *lhq, njs_value_t *retval) +njs_object_property(njs_vm_t *vm, njs_object_t *object, njs_lvlhsh_query_t *lhq, + njs_value_t *retval) { njs_int_t ret; - njs_object_t *object; + njs_value_t value; njs_object_prop_t *prop; - object = njs_object(value); - do { ret = njs_lvlhsh_find(&object->hash, lhq); @@ -135,7 +133,9 @@ found: return NJS_OK; } - return njs_function_apply(vm, njs_prop_getter(prop), value, 1, retval); + njs_set_object(&value, object); + + return njs_function_apply(vm, njs_prop_getter(prop), &value, 1, retval); } @@ -720,6 +720,7 @@ njs_descriptor_prop(njs_vm_t *vm, const njs_int_t ret; njs_bool_t data, accessor; njs_value_t value; + njs_object_t *desc_object; njs_function_t *getter, *setter; njs_object_prop_t *prop; njs_lvlhsh_query_t lhq; @@ -741,10 +742,11 @@ njs_descriptor_prop(njs_vm_t *vm, const accessor = 0; getter = NJS_PROP_PTR_UNSET; setter = NJS_PROP_PTR_UNSET; + desc_object = njs_object(desc); njs_object_property_init(&lhq, &get_string, NJS_GET_HASH); - ret = njs_object_property(vm, desc, &lhq, &value); + ret = njs_object_property(vm, desc_object, &lhq, &value); if (njs_slow_path(ret == NJS_ERROR)) { return NULL; } @@ -762,7 +764,7 @@ njs_descriptor_prop(njs_vm_t *vm, const lhq.key = njs_str_value("set"); lhq.key_hash = NJS_SET_HASH; - ret = njs_object_property(vm, desc, &lhq, &value); + ret = njs_object_property(vm, desc_object, &lhq, &value); if (njs_slow_path(ret == NJS_ERROR)) { return NULL; } @@ -780,7 +782,7 @@ njs_descriptor_prop(njs_vm_t *vm, const lhq.key = njs_str_value("value"); lhq.key_hash = NJS_VALUE_HASH; - ret = njs_object_property(vm, desc, &lhq, &value); + ret = njs_object_property(vm, desc_object, &lhq, &value); if (njs_slow_path(ret == NJS_ERROR)) { return NULL; } @@ -793,7 +795,7 @@ njs_descriptor_prop(njs_vm_t *vm, const lhq.key = njs_str_value("writable"); lhq.key_hash = NJS_WRITABABLE_HASH; - ret = njs_object_property(vm, desc, &lhq, &value); + ret = njs_object_property(vm, desc_object, &lhq, &value); if (njs_slow_path(ret == NJS_ERROR)) { return NULL; } @@ -812,7 +814,7 @@ njs_descriptor_prop(njs_vm_t *vm, const lhq.key = njs_str_value("enumerable"); lhq.key_hash = NJS_ENUMERABLE_HASH; - ret = njs_object_property(vm, desc, &lhq, &value); + ret = njs_object_property(vm, desc_object, &lhq, &value); if (njs_slow_path(ret == NJS_ERROR)) { return NULL; } @@ -824,7 +826,7 @@ njs_descriptor_prop(njs_vm_t *vm, const lhq.key = njs_str_value("configurable"); lhq.key_hash = NJS_CONFIGURABLE_HASH; - ret = njs_object_property(vm, desc, &lhq, &value); + ret = njs_object_property(vm, desc_object, &lhq, &value); if (njs_slow_path(ret == NJS_ERROR)) { return NULL; } diff -r e92881f2b395 -r 1aa137411b09 src/njs_value.c --- a/src/njs_value.c Mon Feb 27 19:05:03 2023 -0800 +++ b/src/njs_value.c Mon Feb 27 22:14:34 2023 -0800 @@ -157,7 +157,7 @@ njs_value_to_primitive(njs_vm_t *vm, njs lhq.key_hash = hashes[hint]; lhq.key = names[hint]; - ret = njs_object_property(vm, value, &lhq, &method); + ret = njs_object_property(vm, njs_object(value), &lhq, &method); if (njs_slow_path(ret == NJS_ERROR)) { return ret; diff -r e92881f2b395 -r 1aa137411b09 src/test/njs_unit_test.c --- a/src/test/njs_unit_test.c Mon Feb 27 19:05:03 2023 -0800 +++ b/src/test/njs_unit_test.c Mon Feb 27 22:14:34 2023 -0800 @@ -22849,6 +22849,9 @@ static njs_unit_test_t njs_shell_test[] "sq(function () { return 3 })" ENTER), njs_str("9") }, + { njs_str("var e = Error(); e.name = {}; e" ENTER), + njs_str("[object Object]") }, + /* Temporary indexes */ { njs_str("var a = [1,2,3], i; for (i in a) {Object.seal({});}" ENTER), From xeioex at nginx.com Tue Feb 28 06:15:43 2023 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Tue, 28 Feb 2023 06:15:43 +0000 Subject: [njs] Fixed array instance with a getter property dumping. Message-ID: details: https://hg.nginx.org/njs/rev/e4cef2c70d7c branches: changeset: 2054:e4cef2c70d7c user: Dmitry Volyntsev date: Mon Feb 27 22:14:36 2023 -0800 description: Fixed array instance with a getter property dumping. This closes #618 issue on Github. diffstat: src/njs_json.c | 40 +++++++++++++++++++--------------------- src/test/njs_unit_test.c | 3 +++ 2 files changed, 22 insertions(+), 21 deletions(-) diffs (68 lines): diff -r 1aa137411b09 -r e4cef2c70d7c src/njs_json.c --- a/src/njs_json.c Mon Feb 27 22:14:34 2023 -0800 +++ b/src/njs_json.c Mon Feb 27 22:14:36 2023 -0800 @@ -2085,32 +2085,30 @@ njs_vm_value_dump(njs_vm_t *vm, njs_str_ val = njs_prop_value(prop); - if (!state->fast_array) { - if (prop->type == NJS_PROPERTY_HANDLER) { - pq.scratch = *prop; - prop = &pq.scratch; - ret = njs_prop_handler(prop)(vm, prop, &state->value, NULL, - njs_prop_value(prop)); - - if (njs_slow_path(ret == NJS_ERROR)) { - return ret; - } - - val = njs_prop_value(prop); + if (prop->type == NJS_PROPERTY_HANDLER) { + pq.scratch = *prop; + prop = &pq.scratch; + ret = njs_prop_handler(prop)(vm, prop, &state->value, NULL, + njs_prop_value(prop)); + + if (njs_slow_path(ret == NJS_ERROR)) { + return ret; } - if (njs_is_accessor_descriptor(prop)) { - if (njs_prop_getter(prop) != NULL) { - if (njs_prop_setter(prop) != NULL) { - val = njs_value_arg(&string_get_set); - - } else { - val = njs_value_arg(&string_get); - } + val = njs_prop_value(prop); + } + + if (njs_is_accessor_descriptor(prop)) { + if (njs_prop_getter(prop) != NULL) { + if (njs_prop_setter(prop) != NULL) { + val = njs_value_arg(&string_get_set); } else { - val = njs_value_arg(&string_set); + val = njs_value_arg(&string_get); } + + } else { + val = njs_value_arg(&string_set); } } diff -r 1aa137411b09 -r e4cef2c70d7c src/test/njs_unit_test.c --- a/src/test/njs_unit_test.c Mon Feb 27 22:14:34 2023 -0800 +++ b/src/test/njs_unit_test.c Mon Feb 27 22:14:36 2023 -0800 @@ -22852,6 +22852,9 @@ static njs_unit_test_t njs_shell_test[] { njs_str("var e = Error(); e.name = {}; e" ENTER), njs_str("[object Object]") }, + { njs_str("var a = []; Object.defineProperty(a, 'b', {enumerable: true, get: Object}); a" ENTER), + njs_str("[\n b: '[Getter]'\n]") }, + /* Temporary indexes */ { njs_str("var a = [1,2,3], i; for (i in a) {Object.seal({});}" ENTER), From xeioex at nginx.com Tue Feb 28 06:15:45 2023 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Tue, 28 Feb 2023 06:15:45 +0000 Subject: [njs] Fixed njs_object_property() with NJS_WHITEOUT properties. Message-ID: details: https://hg.nginx.org/njs/rev/a79b6a75cfab branches: changeset: 2055:a79b6a75cfab user: Dmitry Volyntsev date: Mon Feb 27 22:14:36 2023 -0800 description: Fixed njs_object_property() with NJS_WHITEOUT properties. Previosly, an error object dumping might result in invalid pointer dereference when 'name' or 'message' property of accessor descriptor type was added and removed before. The fix is to properly handle NJS_WHITEOUT properties. This fixes #617 issue on Github. diffstat: src/njs_object_prop.c | 6 +++++- src/njs_value.c | 9 ++++++--- src/test/njs_unit_test.c | 10 ++++++++++ 3 files changed, 21 insertions(+), 4 deletions(-) diffs (59 lines): diff -r e4cef2c70d7c -r a79b6a75cfab src/njs_object_prop.c --- a/src/njs_object_prop.c Mon Feb 27 22:14:36 2023 -0800 +++ b/src/njs_object_prop.c Mon Feb 27 22:14:36 2023 -0800 @@ -102,7 +102,11 @@ njs_object_property(njs_vm_t *vm, njs_ob ret = njs_lvlhsh_find(&object->hash, lhq); if (njs_fast_path(ret == NJS_OK)) { - goto found; + prop = lhq->value; + + if (prop->type != NJS_WHITEOUT) { + goto found; + } } ret = njs_lvlhsh_find(&object->shared_hash, lhq); diff -r e4cef2c70d7c -r a79b6a75cfab src/njs_value.c --- a/src/njs_value.c Mon Feb 27 22:14:36 2023 -0800 +++ b/src/njs_value.c Mon Feb 27 22:14:36 2023 -0800 @@ -1487,13 +1487,16 @@ slow_path: return NJS_ERROR; } - /* GC: release value. */ if (removed != NULL) { - njs_value_assign(removed, njs_prop_value(prop)); + if (njs_is_valid(njs_prop_value(prop))) { + njs_value_assign(removed, njs_prop_value(prop)); + + } else { + njs_set_undefined(removed); + } } prop->type = NJS_WHITEOUT; - njs_set_invalid(njs_prop_value(prop)); return NJS_OK; } diff -r e4cef2c70d7c -r a79b6a75cfab src/test/njs_unit_test.c --- a/src/test/njs_unit_test.c Mon Feb 27 22:14:36 2023 -0800 +++ b/src/test/njs_unit_test.c Mon Feb 27 22:14:36 2023 -0800 @@ -22855,6 +22855,16 @@ static njs_unit_test_t njs_shell_test[] { njs_str("var a = []; Object.defineProperty(a, 'b', {enumerable: true, get: Object}); a" ENTER), njs_str("[\n b: '[Getter]'\n]") }, + { njs_str("var e = Error()" ENTER + "Object.defineProperty(e, 'message', { configurable: true, set: Object })" ENTER + "delete e.message; e" ENTER), + njs_str("Error") }, + + { njs_str("var e = Error()" ENTER + "Object.defineProperty(e, 'message', { configurable: true, get(){ return 'foo'} })" ENTER + "e" ENTER), + njs_str("Error: foo") }, + /* Temporary indexes */ { njs_str("var a = [1,2,3], i; for (i in a) {Object.seal({});}" ENTER), From xeioex at nginx.com Tue Feb 28 06:15:47 2023 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Tue, 28 Feb 2023 06:15:47 +0000 Subject: [njs] Fixed a func instance dumping with "name" as getter. Message-ID: details: https://hg.nginx.org/njs/rev/293fe42c5e1c branches: changeset: 2056:293fe42c5e1c user: Dmitry Volyntsev date: Mon Feb 27 22:14:36 2023 -0800 description: Fixed a func instance dumping with "name" as getter. After njs_value_property() call the value argument might be overwritten. This is similar to #615. diffstat: src/njs_json.c | 13 +++++++------ src/test/njs_unit_test.c | 5 +++++ 2 files changed, 12 insertions(+), 6 deletions(-) diffs (45 lines): diff -r a79b6a75cfab -r 293fe42c5e1c src/njs_json.c --- a/src/njs_json.c Mon Feb 27 22:14:36 2023 -0800 +++ b/src/njs_json.c Mon Feb 27 22:14:36 2023 -0800 @@ -1771,6 +1771,13 @@ njs_dump_terminal(njs_json_stringify_t * break; case NJS_FUNCTION: + if (njs_function(value)->native) { + str = njs_str_value("native"); + + } else { + str = njs_str_value(""); + } + ret = njs_value_property(stringify->vm, value, njs_value_arg(&name_string), &tag); if (njs_slow_path(ret == NJS_ERROR)) { @@ -1779,12 +1786,6 @@ njs_dump_terminal(njs_json_stringify_t * if (njs_is_string(&tag)) { njs_string_get(&tag, &str); - - } else if (njs_function(value)->native) { - str = njs_str_value("native"); - - } else { - str = njs_str_value(""); } if (str.length != 0) { diff -r a79b6a75cfab -r 293fe42c5e1c src/test/njs_unit_test.c --- a/src/test/njs_unit_test.c Mon Feb 27 22:14:36 2023 -0800 +++ b/src/test/njs_unit_test.c Mon Feb 27 22:14:36 2023 -0800 @@ -22865,6 +22865,11 @@ static njs_unit_test_t njs_shell_test[] "e" ENTER), njs_str("Error: foo") }, + { njs_str("function f() {};" ENTER + "Object.defineProperty(f, 'name', { get() {void(0)} })" ENTER + "f" ENTER), + njs_str("[Function]") }, + /* Temporary indexes */ { njs_str("var a = [1,2,3], i; for (i in a) {Object.seal({});}" ENTER), From artem.konev at nginx.com Tue Feb 28 18:09:11 2023 From: artem.konev at nginx.com (=?iso-8859-1?q?Artem_Konev?=) Date: Tue, 28 Feb 2023 18:09:11 +0000 Subject: [PATCH] Added info about the Unit 1.29.1 release Message-ID: <9c4971870962c46ef54d.1677607751@gp194lpycg.station> xml/index.xml | 7 +++++++ 1 files changed, 7 insertions(+), 0 deletions(-) # HG changeset patch # User Artem Konev # Date 1677607644 0 # Tue Feb 28 18:07:24 2023 +0000 # Node ID 9c4971870962c46ef54d4c9dc986adc0de1a7afb # Parent b274d289798d5cc5667c689d3f18cad22b1ca4ae Added info about the Unit 1.29.1 release. diff --git a/xml/index.xml b/xml/index.xml --- a/xml/index.xml +++ b/xml/index.xml @@ -7,6 +7,13 @@ + + +unit-1.29.1 bugfix version has been +released. + + + nginx-quic packages From maxim at nginx.com Tue Feb 28 19:17:47 2023 From: maxim at nginx.com (Maxim Konovalov) Date: Tue, 28 Feb 2023 11:17:47 -0800 Subject: [PATCH] Added info about the Unit 1.29.1 release In-Reply-To: <9c4971870962c46ef54d.1677607751@gp194lpycg.station> References: <9c4971870962c46ef54d.1677607751@gp194lpycg.station> Message-ID: <6c453f72-b108-d18b-471a-8c5d2f1849a1@nginx.com> On 28.02.2023 10:09, Artem Konev wrote: > xml/index.xml | 7 +++++++ > 1 files changed, 7 insertions(+), 0 deletions(-) > > > # HG changeset patch > # User Artem Konev > # Date 1677607644 0 > # Tue Feb 28 18:07:24 2023 +0000 > # Node ID 9c4971870962c46ef54d4c9dc986adc0de1a7afb > # Parent b274d289798d5cc5667c689d3f18cad22b1ca4ae > Added info about the Unit 1.29.1 release. > > diff --git a/xml/index.xml b/xml/index.xml > --- a/xml/index.xml > +++ b/xml/index.xml > @@ -7,6 +7,13 @@ > > > > + > + > +unit-1.29.1 bugfix version has been > +released. > + > + > + > > > nginx-quic packages Looks good. -- Maxim Konovalov