From xeioex at nginx.com Fri Mar 1 04:54:02 2024 From: xeioex at nginx.com (=?utf-8?q?Dmitry_Volyntsev?=) Date: Fri, 01 Mar 2024 04:54:02 +0000 Subject: [njs] Shell: revert -t option back to preserve backward compatibility. Message-ID: details: https://hg.nginx.org/njs/rev/d7a0bbcba46e branches: changeset: 2294:d7a0bbcba46e user: Dmitry Volyntsev date: Thu Feb 29 18:56:13 2024 -0800 description: Shell: revert -t option back to preserve backward compatibility. The issue was introduced in cb3e068a511c. diffstat: external/njs_shell.c | 16 ++++++++++++++++ 1 files changed, 16 insertions(+), 0 deletions(-) diffs (26 lines): diff -r 49417e2749e0 -r d7a0bbcba46e external/njs_shell.c --- a/external/njs_shell.c Tue Feb 27 23:25:11 2024 -0800 +++ b/external/njs_shell.c Thu Feb 29 18:56:13 2024 -0800 @@ -726,6 +726,22 @@ njs_options_parse(njs_opts_t *opts, int opts->sandbox = 1; break; + case 't': + if (++i < argc) { + if (strcmp(argv[i], "module") == 0) { + opts->module = 1; + + } else if (strcmp(argv[i], "script") != 0) { + njs_stderror("option \"-t\" unexpected source type: %s\n", + argv[i]); + return NJS_ERROR; + } + + break; + } + + njs_stderror("option \"-t\" requires source type\n"); + return NJS_ERROR; case 'v': case 'V': opts->version = 1; From arut at nginx.com Mon Mar 4 14:46:23 2024 From: arut at nginx.com (=?iso-8859-1?q?Roman_Arutyunyan?=) Date: Mon, 04 Mar 2024 18:46:23 +0400 Subject: [PATCH] Fixed 413 custom error page for HTTP/2 and HTTP/3 (ticket #2609) Message-ID: <3b0be477ab7246caba4c.1709563583@arut-laptop> # HG changeset patch # User Roman Arutyunyan # Date 1709563405 -14400 # Mon Mar 04 18:43:25 2024 +0400 # Node ID 3b0be477ab7246caba4c5152286b8be520ee0418 # Parent 44da04c2d4db94ad4eefa84b299e07c5fa4a00b9 Fixed 413 custom error page for HTTP/2 and HTTP/3 (ticket #2609). Previously an attempt to return a custom 413 error page for these protocols resulted in the standard 413 page (if recursive_error_pages was off) or otherwise internal redirection cycle followed by the 500 error. Discarding request body for HTTP/1 starts by setting r->discard_body which indicates the body is currently being discarded. If and when the entire body is read and discarded, the flag is cleared and r->headers_in.content_length_n is set to zero. Both r->discard_body and r->headers_in.content_length_n prevent nginx from re-generating 413 error after internal redirect in ngx_http_core_find_config_phase(). However the above does not work for HTTP/2 and HTTP/3. Discarding request body for these protocols does not affect the above mentioned fields, which is why there's no protection against re-generating the 413 error. The fix is to assign zero to r->headers_in.content_length_n much like in HTTP/1 case after the body is entirely read and discarded, except for these protocols no active discard is needed. diff --git a/src/http/ngx_http_request_body.c b/src/http/ngx_http_request_body.c --- a/src/http/ngx_http_request_body.c +++ b/src/http/ngx_http_request_body.c @@ -640,12 +640,14 @@ ngx_http_discard_request_body(ngx_http_r #if (NGX_HTTP_V2) if (r->stream) { r->stream->skip_data = 1; + r->headers_in.content_length_n = 0; return NGX_OK; } #endif #if (NGX_HTTP_V3) if (r->http_version == NGX_HTTP_VERSION_30) { + r->headers_in.content_length_n = 0; return NGX_OK; } #endif From xeioex at nginx.com Tue Mar 5 17:23:59 2024 From: xeioex at nginx.com (=?utf-8?q?Dmitry_Volyntsev?=) Date: Tue, 05 Mar 2024 17:23:59 +0000 Subject: [njs] HTTP: allowing to set Server header. Message-ID: details: https://hg.nginx.org/njs/rev/eb01434865d7 branches: changeset: 2295:eb01434865d7 user: Dmitry Volyntsev date: Thu Feb 29 20:56:56 2024 -0800 description: HTTP: allowing to set Server header. diffstat: nginx/ngx_http_js_module.c | 38 ++++++++++++++++++++++++++++++++++++++ nginx/t/js_headers.t | 22 ++++++++++++++++++++-- 2 files changed, 58 insertions(+), 2 deletions(-) diffs (140 lines): diff -r d7a0bbcba46e -r eb01434865d7 nginx/ngx_http_js_module.c --- a/nginx/ngx_http_js_module.c Thu Feb 29 18:56:13 2024 -0800 +++ b/nginx/ngx_http_js_module.c Thu Feb 29 20:56:56 2024 -0800 @@ -149,6 +149,9 @@ static njs_int_t ngx_http_js_last_modifi static njs_int_t ngx_http_js_location122(njs_vm_t *vm, ngx_http_request_t *r, ngx_list_t *headers, njs_str_t *name, njs_value_t *setval, njs_value_t *retval); +static njs_int_t ngx_http_js_server122(njs_vm_t *vm, ngx_http_request_t *r, + ngx_list_t *headers, njs_str_t *name, njs_value_t *setval, + njs_value_t *retval); #endif static njs_int_t ngx_http_js_ext_keys_header_out(njs_vm_t *vm, njs_value_t *value, njs_value_t *keys); @@ -255,6 +258,9 @@ static njs_int_t ngx_http_js_last_modifi static njs_int_t ngx_http_js_location(njs_vm_t *vm, ngx_http_request_t *r, unsigned flags, njs_str_t *name, njs_value_t *setval, njs_value_t *retval); +static njs_int_t ngx_http_js_server(njs_vm_t *vm, ngx_http_request_t *r, + unsigned flags, njs_str_t *name, njs_value_t *setval, + njs_value_t *retval); static ngx_pool_t *ngx_http_js_pool(njs_vm_t *vm, ngx_http_request_t *r); static ngx_resolver_t *ngx_http_js_resolver(njs_vm_t *vm, @@ -1617,6 +1623,7 @@ ngx_http_js_ext_header_out(njs_vm_t *vm, { njs_str("Expires"), ngx_http_js_header_single }, { njs_str("Last-Modified"), ngx_http_js_last_modified122 }, { njs_str("Location"), ngx_http_js_location122 }, + { njs_str("Server"), ngx_http_js_server122 }, { njs_str("Set-Cookie"), ngx_http_js_header_array }, { njs_str("Retry-After"), ngx_http_js_header_single }, { njs_str(""), ngx_http_js_header_generic }, @@ -1630,6 +1637,7 @@ ngx_http_js_ext_header_out(njs_vm_t *vm, { njs_str("Expires"), NJS_HEADER_SINGLE, ngx_http_js_header_out }, { njs_str("Last-Modified"), 0, ngx_http_js_last_modified }, { njs_str("Location"), 0, ngx_http_js_location }, + { njs_str("Server"), 0, ngx_http_js_server }, { njs_str("Set-Cookie"), NJS_HEADER_ARRAY, ngx_http_js_header_out }, { njs_str("Retry-After"), NJS_HEADER_SINGLE, ngx_http_js_header_out }, { njs_str(""), 0, ngx_http_js_header_out }, @@ -2065,6 +2073,14 @@ ngx_http_js_location122(njs_vm_t *vm, ng { return ngx_http_js_location(vm, r, 0, v, setval, retval); } + + +static njs_int_t +ngx_http_js_server122(njs_vm_t *vm, ngx_http_request_t *r, + ngx_list_t *headers, njs_str_t *v, njs_value_t *setval, njs_value_t *retval) +{ + return ngx_http_js_server(vm, r, 0, v, setval, retval); +} #endif @@ -4160,6 +4176,28 @@ ngx_http_js_location(njs_vm_t *vm, ngx_h } +static njs_int_t +ngx_http_js_server(njs_vm_t *vm, ngx_http_request_t *r, unsigned flags, + njs_str_t *v, njs_value_t *setval, njs_value_t *retval) +{ + njs_int_t rc; + ngx_table_elt_t *h; + + rc = ngx_http_js_header_out_special(vm, r, v, setval, retval, &h); + if (rc == NJS_ERROR) { + return NJS_ERROR; + } + + if (setval != NULL || retval == NULL) { + r->headers_out.server = h; + } + + return NJS_OK; + + return NJS_OK; +} + + static void ngx_http_js_periodic_handler(ngx_event_t *ev) { diff -r d7a0bbcba46e -r eb01434865d7 nginx/t/js_headers.t --- a/nginx/t/js_headers.t Thu Feb 29 18:56:13 2024 -0800 +++ b/nginx/t/js_headers.t Thu Feb 29 20:56:56 2024 -0800 @@ -160,6 +160,10 @@ http { js_content test.copy_subrequest_hdrs; } + location /server { + js_content test.server; + } + location = /subrequest { internal; @@ -435,6 +439,11 @@ EOF r.return(200, resp.responseText); } + function server(r) { + r.headersOut['Server'] = 'Foo'; + r.return(200); + } + function subrequest(r) { r.headersOut['A'] = 'a'; r.headersOut['Content-Encoding'] = 'ce'; @@ -456,12 +465,12 @@ EOF hdr_out, raw_hdr_out, hdr_out_array, hdr_out_single, hdr_out_set_cookie, ihdr_out, hdr_out_special_set, copy_subrequest_hdrs, subrequest, date, last_modified, - location, location_sr}; + location, location_sr, server}; EOF -$t->try_run('no njs')->plan(46); +$t->try_run('no njs')->plan(49); ############################################################################### @@ -620,6 +629,15 @@ like(http_get('/last_modified'), } +TODO: { +local $TODO = 'not yet' unless has_version('0.8.4'); + +like(http_get('/date'), qr/Server: nginx/, 'normal server'); +like(http_get('/server'), qr/Server: Foo/, 'set server'); +unlike(http_get('/server'), qr/Server: nginx/, 'set server 2'); + +} + ############################################################################### sub has_version { From osa at FreeBSD.org.ru Tue Mar 5 18:15:31 2024 From: osa at FreeBSD.org.ru (=?iso-8859-1?q?Sergey_A=2E_Osokin?=) Date: Tue, 05 Mar 2024 21:15:31 +0300 Subject: [PATCH] HTTP: remove needless code Message-ID: <00de871f1e5e0936e240.1709662531@mp2.macomnet.net> # HG changeset patch # User Sergey A. Osokin # Date 1709662441 -10800 # Tue Mar 05 21:14:01 2024 +0300 # Node ID 00de871f1e5e0936e2401aaec863bf7040de8608 # Parent eb01434865d7b698f036f199084308726c2827ea HTTP: remove needless code. diff -r eb01434865d7 -r 00de871f1e5e nginx/ngx_http_js_module.c --- a/nginx/ngx_http_js_module.c Thu Feb 29 20:56:56 2024 -0800 +++ b/nginx/ngx_http_js_module.c Tue Mar 05 21:14:01 2024 +0300 @@ -4193,8 +4193,6 @@ } return NJS_OK; - - return NJS_OK; } From xeioex at nginx.com Tue Mar 5 20:13:02 2024 From: xeioex at nginx.com (=?utf-8?q?Dmitry_Volyntsev?=) Date: Tue, 05 Mar 2024 20:13:02 +0000 Subject: [njs] Fixed typo introduced in eb01434865d7. Message-ID: details: https://hg.nginx.org/njs/rev/8309b884e265 branches: changeset: 2296:8309b884e265 user: Dmitry Volyntsev date: Tue Mar 05 11:43:49 2024 -0800 description: Fixed typo introduced in eb01434865d7. diffstat: nginx/ngx_http_js_module.c | 2 -- 1 files changed, 0 insertions(+), 2 deletions(-) diffs (12 lines): diff -r eb01434865d7 -r 8309b884e265 nginx/ngx_http_js_module.c --- a/nginx/ngx_http_js_module.c Thu Feb 29 20:56:56 2024 -0800 +++ b/nginx/ngx_http_js_module.c Tue Mar 05 11:43:49 2024 -0800 @@ -4193,8 +4193,6 @@ ngx_http_js_server(njs_vm_t *vm, ngx_htt } return NJS_OK; - - return NJS_OK; } From mdounin at mdounin.ru Tue Mar 5 20:28:26 2024 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 5 Mar 2024 23:28:26 +0300 Subject: [PATCH] Fixed 413 custom error page for HTTP/2 and HTTP/3 (ticket #2609) In-Reply-To: <3b0be477ab7246caba4c.1709563583@arut-laptop> References: <3b0be477ab7246caba4c.1709563583@arut-laptop> Message-ID: Hello! On Mon, Mar 04, 2024 at 06:46:23PM +0400, Roman Arutyunyan wrote: > # HG changeset patch > # User Roman Arutyunyan > # Date 1709563405 -14400 > # Mon Mar 04 18:43:25 2024 +0400 > # Node ID 3b0be477ab7246caba4c5152286b8be520ee0418 > # Parent 44da04c2d4db94ad4eefa84b299e07c5fa4a00b9 > Fixed 413 custom error page for HTTP/2 and HTTP/3 (ticket #2609). > > Previously an attempt to return a custom 413 error page for these protocols > resulted in the standard 413 page (if recursive_error_pages was off) or > otherwise internal redirection cycle followed by the 500 error. > > Discarding request body for HTTP/1 starts by setting r->discard_body which > indicates the body is currently being discarded. If and when the entire body > is read and discarded, the flag is cleared and r->headers_in.content_length_n > is set to zero. Both r->discard_body and r->headers_in.content_length_n > prevent nginx from re-generating 413 error after internal redirect in > ngx_http_core_find_config_phase(). > > However the above does not work for HTTP/2 and HTTP/3. Discarding request > body for these protocols does not affect the above mentioned fields, which is > why there's no protection against re-generating the 413 error. The fix is to > assign zero to r->headers_in.content_length_n much like in HTTP/1 case after > the body is entirely read and discarded, except for these protocols no active > discard is needed. > > diff --git a/src/http/ngx_http_request_body.c b/src/http/ngx_http_request_body.c > --- a/src/http/ngx_http_request_body.c > +++ b/src/http/ngx_http_request_body.c > @@ -640,12 +640,14 @@ ngx_http_discard_request_body(ngx_http_r > #if (NGX_HTTP_V2) > if (r->stream) { > r->stream->skip_data = 1; > + r->headers_in.content_length_n = 0; > return NGX_OK; > } > #endif > > #if (NGX_HTTP_V3) > if (r->http_version == NGX_HTTP_VERSION_30) { > + r->headers_in.content_length_n = 0; > return NGX_OK; > } > #endif The patch is wrong, see here: https://trac.nginx.org/nginx/ticket/1152#comment:6 The issue is in my TODO list. Once properly fixed, you'll be able to merge the fix from freenginx. Alternatively, consider submitting patches to the nginx-devel at freenginx.org list for proper review. -- Maxim Dounin http://mdounin.ru/ From xeioex at nginx.com Wed Mar 6 01:33:28 2024 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Tue, 5 Mar 2024 17:33:28 -0800 Subject: [PATCH] HTTP: remove needless code In-Reply-To: <00de871f1e5e0936e240.1709662531@mp2.macomnet.net> References: <00de871f1e5e0936e240.1709662531@mp2.macomnet.net> Message-ID: <02a86240-4bd3-44ea-82bf-da5f69129bf3@nginx.com> On 3/5/24 10:15 AM, Sergey A. Osokin wrote: > # HG changeset patch > # User Sergey A. Osokin > # Date 1709662441 -10800 > # Tue Mar 05 21:14:01 2024 +0300 > # Node ID 00de871f1e5e0936e2401aaec863bf7040de8608 > # Parent eb01434865d7b698f036f199084308726c2827ea > HTTP: remove needless code. > > diff -r eb01434865d7 -r 00de871f1e5e nginx/ngx_http_js_module.c > --- a/nginx/ngx_http_js_module.c Thu Feb 29 20:56:56 2024 -0800 > +++ b/nginx/ngx_http_js_module.c Tue Mar 05 21:14:01 2024 +0300 > @@ -4193,8 +4193,6 @@ > } > > return NJS_OK; > - > - return NJS_OK; > } > Thank you for catching the issue. It is already fixed here https://hg.nginx.org/njs/rev/8309b884e265. > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > https://mailman.nginx.org/mailman/listinfo/nginx-devel From arut at nginx.com Wed Mar 6 06:32:29 2024 From: arut at nginx.com (Roman Arutyunyan) Date: Wed, 6 Mar 2024 10:32:29 +0400 Subject: [PATCH] Fixed 413 custom error page for HTTP/2 and HTTP/3 (ticket #2609) In-Reply-To: References: <3b0be477ab7246caba4c.1709563583@arut-laptop> Message-ID: <8D368D6E-BF5F-4C04-9C58-9B391A565898@nginx.com> Hi Maxim, > On 6 Mar 2024, at 12:28 AM, Maxim Dounin wrote: > > Hello! > > On Mon, Mar 04, 2024 at 06:46:23PM +0400, Roman Arutyunyan wrote: > >> # HG changeset patch >> # User Roman Arutyunyan >> # Date 1709563405 -14400 >> # Mon Mar 04 18:43:25 2024 +0400 >> # Node ID 3b0be477ab7246caba4c5152286b8be520ee0418 >> # Parent 44da04c2d4db94ad4eefa84b299e07c5fa4a00b9 >> Fixed 413 custom error page for HTTP/2 and HTTP/3 (ticket #2609). >> >> Previously an attempt to return a custom 413 error page for these protocols >> resulted in the standard 413 page (if recursive_error_pages was off) or >> otherwise internal redirection cycle followed by the 500 error. >> >> Discarding request body for HTTP/1 starts by setting r->discard_body which >> indicates the body is currently being discarded. If and when the entire body >> is read and discarded, the flag is cleared and r->headers_in.content_length_n >> is set to zero. Both r->discard_body and r->headers_in.content_length_n >> prevent nginx from re-generating 413 error after internal redirect in >> ngx_http_core_find_config_phase(). >> >> However the above does not work for HTTP/2 and HTTP/3. Discarding request >> body for these protocols does not affect the above mentioned fields, which is >> why there's no protection against re-generating the 413 error. The fix is to >> assign zero to r->headers_in.content_length_n much like in HTTP/1 case after >> the body is entirely read and discarded, except for these protocols no active >> discard is needed. >> >> diff --git a/src/http/ngx_http_request_body.c b/src/http/ngx_http_request_body.c >> --- a/src/http/ngx_http_request_body.c >> +++ b/src/http/ngx_http_request_body.c >> @@ -640,12 +640,14 @@ ngx_http_discard_request_body(ngx_http_r >> #if (NGX_HTTP_V2) >> if (r->stream) { >> r->stream->skip_data = 1; >> + r->headers_in.content_length_n = 0; >> return NGX_OK; >> } >> #endif >> >> #if (NGX_HTTP_V3) >> if (r->http_version == NGX_HTTP_VERSION_30) { >> + r->headers_in.content_length_n = 0; >> return NGX_OK; >> } >> #endif > > The patch is wrong, see here: > > https://trac.nginx.org/nginx/ticket/1152#comment:6 Thanks for your kind comment, I read that before. The patch fixes exactly what it fixes. Accessing the request body after it was discarded (for example from a 413 custom handler) is a sign of misconfiguration. Indeed for a misconfigured nginx there is a problem with a HTTP/1 body currently being discarded as well as other inconsistencies. This may or may not be fixed in the future, but anyway it's a separate issue, which does not exist in a properly configured nginx. > The issue is in my TODO list. Once properly fixed, you'll be able > to merge the fix from freenginx. > > Alternatively, consider submitting patches to > the nginx-devel at freenginx.org list for proper review. We understand your hard feelings about leaving the project. Hope you'll be ok. > -- > Maxim Dounin > http://mdounin.ru/ > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > https://mailman.nginx.org/mailman/listinfo/nginx-devel ---- Roman Arutyunyan arut at nginx.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From arut at nginx.com Wed Mar 6 12:55:42 2024 From: arut at nginx.com (Roman Arutyunyan) Date: Wed, 6 Mar 2024 16:55:42 +0400 Subject: [PATCH] Configure: add support for Homebrew on Apple Silicon In-Reply-To: References: Message-ID: <20240306125542.gxdehvg7i7d6ggsn@N00W24XTQX> Hi Piotr, On Wed, Feb 28, 2024 at 01:24:26AM +0000, Piotr Sikora via nginx-devel wrote: > # HG changeset patch > # User Piotr Sikora > # Date 1708977643 0 > # Mon Feb 26 20:00:43 2024 +0000 > # Branch patch017 > # Node ID dd95daa55cf6131a7e845edd6ad3b429bcef6f98 > # Parent bb99cbe3a343ae581d2369b990aee66e69679ca2 > Configure: add support for Homebrew on Apple Silicon. > > Signed-off-by: Piotr Sikora > > diff -r bb99cbe3a343 -r dd95daa55cf6 auto/lib/geoip/conf > --- a/auto/lib/geoip/conf Mon Feb 26 20:00:42 2024 +0000 > +++ b/auto/lib/geoip/conf Mon Feb 26 20:00:43 2024 +0000 > @@ -64,6 +64,23 @@ > fi > > > +if [ $ngx_found = no ]; then > + > + # Homebrew on Apple Silicon > + > + ngx_feature="GeoIP library in /opt/homebrew/" > + ngx_feature_path="/opt/homebrew/include" > + > + if [ $NGX_RPATH = YES ]; then > + ngx_feature_libs="-R/opt/homebrew/lib -L/opt/homebrew/lib -lGeoIP" > + else > + ngx_feature_libs="-L/opt/homebrew/lib -lGeoIP" > + fi > + > + . auto/feature > +fi > + > + > if [ $ngx_found = yes ]; then > > CORE_INCS="$CORE_INCS $ngx_feature_path" > diff -r bb99cbe3a343 -r dd95daa55cf6 auto/lib/google-perftools/conf > --- a/auto/lib/google-perftools/conf Mon Feb 26 20:00:42 2024 +0000 > +++ b/auto/lib/google-perftools/conf Mon Feb 26 20:00:43 2024 +0000 > @@ -46,6 +46,22 @@ > fi > > > +if [ $ngx_found = no ]; then > + > + # Homebrew on Apple Silicon > + > + ngx_feature="Google perftools in /opt/homebrew/" > + > + if [ $NGX_RPATH = YES ]; then > + ngx_feature_libs="-R/opt/homebrew/lib -L/opt/homebrew/lib -lprofiler" > + else > + ngx_feature_libs="-L/opt/homebrew/lib -lprofiler" > + fi > + > + . auto/feature > +fi > + > + > if [ $ngx_found = yes ]; then > CORE_LIBS="$CORE_LIBS $ngx_feature_libs" > > diff -r bb99cbe3a343 -r dd95daa55cf6 auto/lib/libgd/conf > --- a/auto/lib/libgd/conf Mon Feb 26 20:00:42 2024 +0000 > +++ b/auto/lib/libgd/conf Mon Feb 26 20:00:43 2024 +0000 > @@ -65,6 +65,23 @@ > fi > > > +if [ $ngx_found = no ]; then > + > + # Homebrew on Apple Silicon > + > + ngx_feature="GD library in /opt/homebrew/" > + ngx_feature_path="/opt/homebrew/include" > + > + if [ $NGX_RPATH = YES ]; then > + ngx_feature_libs="-R/opt/homebrew/lib -L/opt/homebrew/lib -lgd" > + else > + ngx_feature_libs="-L/opt/homebrew/lib -lgd" > + fi > + > + . auto/feature > +fi > + > + > if [ $ngx_found = yes ]; then > > CORE_INCS="$CORE_INCS $ngx_feature_path" > diff -r bb99cbe3a343 -r dd95daa55cf6 auto/lib/openssl/conf > --- a/auto/lib/openssl/conf Mon Feb 26 20:00:42 2024 +0000 > +++ b/auto/lib/openssl/conf Mon Feb 26 20:00:43 2024 +0000 > @@ -122,6 +122,24 @@ > . auto/feature > fi > > + if [ $ngx_found = no ]; then > + > + # Homebrew on Apple Silicon > + > + ngx_feature="OpenSSL library in /opt/homebrew/" > + ngx_feature_path="/opt/homebrew/include" > + > + if [ $NGX_RPATH = YES ]; then > + ngx_feature_libs="-R/opt/homebrew/lib -L/opt/homebrew/lib -lssl -lcrypto" > + else > + ngx_feature_libs="-L/opt/homebrew/lib -lssl -lcrypto" > + fi > + > + ngx_feature_libs="$ngx_feature_libs $NGX_LIBDL $NGX_LIBPTHREAD" > + > + . auto/feature > + fi > + > if [ $ngx_found = yes ]; then > have=NGX_SSL . auto/have > CORE_INCS="$CORE_INCS $ngx_feature_path" > diff -r bb99cbe3a343 -r dd95daa55cf6 auto/lib/pcre/conf > --- a/auto/lib/pcre/conf Mon Feb 26 20:00:42 2024 +0000 > +++ b/auto/lib/pcre/conf Mon Feb 26 20:00:43 2024 +0000 > @@ -182,6 +182,22 @@ > . auto/feature > fi > > + if [ $ngx_found = no ]; then > + > + # Homebrew on Apple Silicon > + > + ngx_feature="PCRE library in /opt/homebrew/" > + ngx_feature_path="/opt/homebrew/include" > + > + if [ $NGX_RPATH = YES ]; then > + ngx_feature_libs="-R/opt/homebrew/lib -L/opt/homebrew/lib -lpcre" > + else > + ngx_feature_libs="-L/opt/homebrew/lib -lpcre" > + fi > + > + . auto/feature > + fi > + > if [ $ngx_found = yes ]; then > CORE_INCS="$CORE_INCS $ngx_feature_path" > CORE_LIBS="$CORE_LIBS $ngx_feature_libs" Thanks for the patch. Looks good to me. For the record: Among the libraries we search MacPorts for, there are xml/xslt-related libraries as well, which are not included in this patch. Modern MacOS however includes those libraries by default, and older versions do not support Apple Silicon. -- Roman Arutyunyan From pluknet at nginx.com Wed Mar 6 14:50:26 2024 From: pluknet at nginx.com (Sergey Kandaurov) Date: Wed, 6 Mar 2024 18:50:26 +0400 Subject: [PATCH] Avoiding mixed socket families in PROXY protocol v1 (ticket #2594) In-Reply-To: <20240222151726.3dzpvvanswdqhbkh@N00W24XTQX> References: <2f12c929527b2337c15e.1705920594@arut-laptop> <20240122154801.ycda4ie442ipzw6n@N00W24XTQX> <20240221132920.chmms5v3aekvmc2i@N00W24XTQX> <20240222151726.3dzpvvanswdqhbkh@N00W24XTQX> Message-ID: On Thu, Feb 22, 2024 at 07:17:26PM +0400, Roman Arutyunyan wrote: > Hi, > > On Thu, Feb 22, 2024 at 01:59:25AM +0000, J Carter wrote: > > Hello Roman, > > > > On Wed, 21 Feb 2024 17:29:52 +0400 > > Roman Arutyunyan wrote: > > > > > Hi, > > > > > > > > On Wed, Jan 24, 2024 at 12:03:06AM +0300, Maxim Dounin wrote: > > > > [..] > > > > Also, as suggested, using the server address as obtained via PROXY > > > > protocol from the client might be a better solution as long as the > > > > client address was set via PROXY protocol (regardless of whether > > > > address families match or not), and what users expect from the > > > > "proty_protocol on;" when chaining stream proxies in the first > > > > place. > > > > > Checking whether the address used in PROXY writer is in fact the address > > > that was passed in the PROXY header, is complicated. This will either require > > > setting a flag when PROXY address is set by realip, which is ugly. > > > Another approach is checking if the client address written to a PROXY header > > > matches the client address in the received PROXY header. However since > > > currently PROXY protocol addresses are stored as text, and not all addresses > > > have unique text repersentations, this approach would require refactoring all > > > PROXY protocol code + realip modules to switch from text to sockaddr. Checking for the realip context should do the trick, it is created once the client address was successfully substituted. We could then parse the text repr of server address in the PROXY header and use it in PROXY writer. This will allow not to break API. Something like this should work called from the stream proxy module, it allows to preserve an original server address in the PROXY header keeping both addresses unmutated (which seems to be the best way to solve this problem). : extern ngx_module_t ngx_stream_realip_module; : : if (ngx_stream_get_module_ctx(s, ngx_stream_realip_module)) { : : local_sockaddr = c->local_sockaddr; : local_socklen = c->local_socklen; : : if (ngx_parse_addr(c->pool, &addr, c->proxy_protocol->dst_addr.data, : c->proxy_protocol->dst_addr.len) : != NGX_OK) : { : ngx_stream_proxy_finalize(s, NGX_STREAM_INTERNAL_SERVER_ERROR); : return; : } : : ngx_inet_set_port(addr.sockaddr, c->proxy_protocol->dst_port); : : c->local_sockaddr = addr.sockaddr; : c->local_socklen = addr.socklen; : } : : # call ngx_proxy_protocol_write(), other existing stuff : : if (ngx_stream_get_module_ctx(s, ngx_stream_realip_module)) { : c->local_sockaddr = local_sockaddr; : c->local_socklen = local_socklen; : } Can't say I am excited from this code, though. Another way is stick to the INADDR_ANY plan removing the server address part from the PROXY header if not applicable. > > > > > > I suggest that we follow the first plan (INADDR_ANY etc). I agree in general, but see below. > > > > > > > [...] > > > > > > Updated patch attached. > > > > # HG changeset patch > # User Roman Arutyunyan > # Date 1708522464 -14400 > # Wed Feb 21 17:34:24 2024 +0400 > # Node ID 2d9bb7b49d64576fa29a673133129f16de3cfbbe > # Parent 44da04c2d4db94ad4eefa84b299e07c5fa4a00b9 > Avoiding mixed socket families in PROXY protocol v1 (ticket #2010). > > When using realip module, remote and local addresses of a connection can belong > to different address families. This previously resulted in generating PROXY > protocol headers like this: > > PROXY TCP4 127.0.0.1 unix:/tmp/nginx1.sock 55544 0 Well, UNIX-domain socket addresses are somewhat rather exsotic, especially in the PROXY protocol. I'd use an example with IPv4/IPv6. > > The PROXY protocol v1 specification does not allow mixed families. The change > substitutes server address with zero address in this case: > > PROXY TCP4 127.0.0.1 0.0.0.0 55544 0 > > As an alternative, "PROXY UNKNOWN" header could be used, which unlike this > header does not contain any useful information about the client. > > Also, the above mentioned format for unix socket address is not specified in > PROXY protocol v1 and is a by-product of internal nginx representation of it. > The change eliminates such addresses from PROXY protocol headers as well. > > diff --git a/src/core/ngx_proxy_protocol.c b/src/core/ngx_proxy_protocol.c > --- a/src/core/ngx_proxy_protocol.c > +++ b/src/core/ngx_proxy_protocol.c > @@ -279,7 +279,13 @@ ngx_proxy_protocol_read_port(u_char *p, > u_char * > ngx_proxy_protocol_write(ngx_connection_t *c, u_char *buf, u_char *last) > { > - ngx_uint_t port, lport; > + socklen_t local_socklen; > + ngx_uint_t port, lport; > + struct sockaddr *local_sockaddr; > + struct sockaddr_in sin; > +#if (NGX_HAVE_INET6) > + struct sockaddr_in6 sin6; > +#endif > > if (last - buf < NGX_PROXY_PROTOCOL_V1_MAX_HEADER) { > ngx_log_error(NGX_LOG_ALERT, c->log, 0, > @@ -312,11 +318,35 @@ ngx_proxy_protocol_write(ngx_connection_ > > *buf++ = ' '; > > - buf += ngx_sock_ntop(c->local_sockaddr, c->local_socklen, buf, last - buf, > - 0); > + if (c->sockaddr->sa_family == c->local_sockaddr->sa_family) { > + local_sockaddr = c->local_sockaddr; > + local_socklen = c->local_socklen; > + > + } else { > + switch (c->sockaddr->sa_family) { > + > +#if (NGX_HAVE_INET6) > + case AF_INET6: > + ngx_memzero(&sin6, sizeof(struct sockaddr_in6)); > + sin6.sin6_family = AF_INET6; > + local_sockaddr = (struct sockaddr *) &sin6; > + local_socklen = sizeof(struct sockaddr_in6); > + break; > +#endif > + > + default: /* AF_INET */ > + ngx_memzero(&sin, sizeof(struct sockaddr)); Note that although wildcard adresses are typically implemented using all-zeroes, this is not obligated by standards and hence is an implementation detail. POSIX mandates just the following: : The header shall define the following symbolic : constant for use as a local address in the structure passed to bind(): : : INADDR_ANY : IPv4 wildcard address. (Note: it is "IPv4 local host address." in previous editions.) : The header shall declare the following external : variable: : : const struct in6_addr in6addr_any : : This variable is initialized by the system to contain the wildcard : IPv6 address. The header also defines the : IN6ADDR_ANY_INIT macro. This macro must be constant at compile time : and can be used to initialize a variable of type struct in6_addr to : the IPv6 wildcard address. : RATIONALE : : The INADDR_ANY and INADDR_BROADCAST values are byte-order-neutral : and thus their byte order is not specified. Many implementations : have additional constants as extensions, such as INADDR_LOOPBACK, : that are not byte-order-neutral. Traditionally, these constants are : in host byte order, requiring the use of htonl() when using them in : a sockaddr_in structure. So it seems quite legitimate to use a different binary representation. To be on the safe side (and improve code readability), it makes sense to consistently set wildcard addresses explicitly, using INADDR_ANY and in6addr_any. It is not so important in this patch though, because addresses do not cross the kernel, used solely for ngx_sock_ntop(). > + sin.sin_family = AF_INET; > + local_sockaddr = (struct sockaddr *) &sin; > + local_socklen = sizeof(struct sockaddr_in); > + break; > + } > + } > + > + buf += ngx_sock_ntop(local_sockaddr, local_socklen, buf, last - buf, 0); > > port = ngx_inet_get_port(c->sockaddr); > - lport = ngx_inet_get_port(c->local_sockaddr); > + lport = ngx_inet_get_port(local_sockaddr); > > return ngx_slprintf(buf, last, " %ui %ui" CRLF, port, lport); > } Picking INADDR_ANY (and IPv6 equivalent) to substitute non-applicable addresses due to mismatching address family may be unnecessarily strict hiding addresses that can be represented in a different family. IPv4 address can be represented in the IPv4-mapped IPv6 address format. IPv6 address can be represented in the IPv4 format if it's IPv4-mapped. Such format is supported in various nginx modules and may be enabled by "listen ... ipv6only=off" to accept IPv4-mapped IPv6 addresses on a wildcard listen socket. For example, if nginx receives "PROXY TCP6" on "listen 127.0.0.1:8081" and is configured to send PROXY protocol further in the stream proxy chain, the server address will be updated to "::ffff:127.0.0.1 8081". And opposite, receiving "PROXY TCP4" on "listen [::]:8082 ipv6only=off" will update the PROXY header server part to "127.0.0.1 8082". That said, something like this should work on top off your change: diff --git a/src/core/ngx_proxy_protocol.c b/src/core/ngx_proxy_protocol.c --- a/src/core/ngx_proxy_protocol.c +++ b/src/core/ngx_proxy_protocol.c @@ -279,12 +279,13 @@ ngx_proxy_protocol_read_port(u_char *p, u_char * ngx_proxy_protocol_write(ngx_connection_t *c, u_char *buf, u_char *last) { - socklen_t local_socklen; - ngx_uint_t port, lport; - struct sockaddr *local_sockaddr; - struct sockaddr_in sin; + socklen_t local_socklen; + ngx_uint_t port, lport; + ngx_sockaddr_t sa; + struct sockaddr *local_sockaddr; #if (NGX_HAVE_INET6) - struct sockaddr_in6 sin6; + in_addr_t addr4; + struct in6_addr *addr6; #endif if (last - buf < NGX_PROXY_PROTOCOL_V1_MAX_HEADER) { @@ -323,22 +324,62 @@ ngx_proxy_protocol_write(ngx_connection_ local_socklen = c->local_socklen; } else { + local_sockaddr = &sa.sockaddr; + + ngx_memzero(&sa, sizeof(ngx_sockaddr_t)); + sa.sockaddr.sa_family = c->sockaddr->sa_family; + ngx_inet_set_port(&sa.sockaddr, ngx_inet_get_port(c->local_sockaddr)); + switch (c->sockaddr->sa_family) { #if (NGX_HAVE_INET6) case AF_INET6: - ngx_memzero(&sin6, sizeof(struct sockaddr_in6)); - sin6.sin6_family = AF_INET6; - local_sockaddr = (struct sockaddr *) &sin6; local_socklen = sizeof(struct sockaddr_in6); + + if (c->local_sockaddr->sa_family == AF_INET) { + addr4 = ntohl(((struct sockaddr_in *) c->local_sockaddr) + ->sin_addr.s_addr); + + addr6 = &sa.sockaddr_in6.sin6_addr; + addr6->s6_addr[10] = 0xff; + addr6->s6_addr[11] = 0xff; + addr6->s6_addr[12] = addr4 >> 24; + addr6->s6_addr[13] = addr4 >> 16; + addr6->s6_addr[14] = addr4 >> 8; + addr6->s6_addr[15] = addr4; + + } else { + sa.sockaddr_in6.sin6_addr = in6addr_any; + } + break; #endif default: /* AF_INET */ - ngx_memzero(&sin, sizeof(struct sockaddr)); - sin.sin_family = AF_INET; - local_sockaddr = (struct sockaddr *) &sin; local_socklen = sizeof(struct sockaddr_in); + +#if (NGX_HAVE_INET6) + if (c->local_sockaddr->sa_family == AF_INET6) { + addr6 = &((struct sockaddr_in6 *) c->local_sockaddr)->sin6_addr; + + if (IN6_IS_ADDR_V4MAPPED(addr6)) { + addr4 = addr6->s6_addr[12] << 24; + addr4 += addr6->s6_addr[13] << 16; + addr4 += addr6->s6_addr[14] << 8; + addr4 += addr6->s6_addr[15]; + + sa.sockaddr_in.sin_addr.s_addr = htonl(addr4); + + } else { + sa.sockaddr_in.sin_addr.s_addr = INADDR_ANY; + } + + } else +#endif + { + sa.sockaddr_in.sin_addr.s_addr = INADDR_ANY; + } + break; } } From pluknet at nginx.com Wed Mar 6 15:36:32 2024 From: pluknet at nginx.com (Sergey Kandaurov) Date: Wed, 6 Mar 2024 19:36:32 +0400 Subject: [PATCH] Configure: add support for Homebrew on Apple Silicon In-Reply-To: References: Message-ID: <2E747794-8E5F-4B41-B0AF-422DC7A10629@nginx.com> > On 28 Feb 2024, at 05:24, Piotr Sikora via nginx-devel wrote: > > # HG changeset patch > # User Piotr Sikora > # Date 1708977643 0 > # Mon Feb 26 20:00:43 2024 +0000 > # Branch patch017 > # Node ID dd95daa55cf6131a7e845edd6ad3b429bcef6f98 > # Parent bb99cbe3a343ae581d2369b990aee66e69679ca2 > Configure: add support for Homebrew on Apple Silicon. > > Signed-off-by: Piotr Sikora Well, this is weird to pick up install prefix depending on the device. Hopefully, maintainers will rethink. Though given the relevant issue #9177 on githab is over 3 years, they would rather not. An obvious question is why do you need this change. Homebrew seems to be quite niche to pay attention. Using appropriate paths in --with-cc-opt / --with-ld-opt should work (not tested). If it really harms though, I think the change should go in. > > diff -r bb99cbe3a343 -r dd95daa55cf6 auto/lib/geoip/conf > --- a/auto/lib/geoip/conf Mon Feb 26 20:00:42 2024 +0000 > +++ b/auto/lib/geoip/conf Mon Feb 26 20:00:43 2024 +0000 A quick grep for MacPorts search paths suggests that some libraries are missing in the change. If this is on purpose, please reflect this in the description. > @@ -64,6 +64,23 @@ > fi > > > +if [ $ngx_found = no ]; then > + > + # Homebrew on Apple Silicon Apple Silicon is something from the marketing language, using Apple ARM instead should be fine. Notably, Homebrew uses Hardware::CPU.arm Ruby language boolean to make the distinction. Further, given the smooth decay on Intel-based hardware, I'd reduce this just to "Homebrew". [..] -- Sergey Kandaurov From jordanc.carter at outlook.com Thu Mar 7 14:22:11 2024 From: jordanc.carter at outlook.com (J Carter) Date: Thu, 7 Mar 2024 14:22:11 +0000 Subject: [PATCH] Configure: add support for Homebrew on Apple Silicon In-Reply-To: <2E747794-8E5F-4B41-B0AF-422DC7A10629@nginx.com> References: <2E747794-8E5F-4B41-B0AF-422DC7A10629@nginx.com> Message-ID: On Wed, 6 Mar 2024 19:36:32 +0400 Sergey Kandaurov wrote: > > On 28 Feb 2024, at 05:24, Piotr Sikora via nginx-devel wrote: > > > > # HG changeset patch > > # User Piotr Sikora > > # Date 1708977643 0 > > # Mon Feb 26 20:00:43 2024 +0000 > > # Branch patch017 > > # Node ID dd95daa55cf6131a7e845edd6ad3b429bcef6f98 > > # Parent bb99cbe3a343ae581d2369b990aee66e69679ca2 > > Configure: add support for Homebrew on Apple Silicon. > > > > > Signed-off-by: Piotr Sikora > > Well, this is weird to pick up install prefix depending on the device. > Hopefully, maintainers will rethink. Though given the relevant > issue #9177 on githab is over 3 years, they would rather not. > > An obvious question is why do you need this change. Homebrew seems > to be quite niche to pay attention. Using appropriate paths in > --with-cc-opt / --with-ld-opt should work (not tested). > If it really harms though, I think the change should go in. > > > > > diff -r bb99cbe3a343 -r dd95daa55cf6 auto/lib/geoip/conf > > --- a/auto/lib/geoip/conf Mon Feb 26 20:00:42 2024 +0000 > > +++ b/auto/lib/geoip/conf Mon Feb 26 20:00:43 2024 +0000 > > A quick grep for MacPorts search paths suggests that some libraries > are missing in the change. If this is on purpose, please reflect > this in the description. > > > @@ -64,6 +64,23 @@ > > fi > > > > > > +if [ $ngx_found = no ]; then > > + > > + # Homebrew on Apple Silicon > > Apple Silicon is something from the marketing language, > using Apple ARM instead should be fine. > > Notably, Homebrew uses Hardware::CPU.arm Ruby language boolean > to make the distinction. > > Further, given the smooth decay on Intel-based hardware, > I'd reduce this just to "Homebrew". > It might be worth it to keep at least 'Apple' distinction in there, if this will be added; Homebrew also supports Linux too, and installs to /home/linuxbrew/.linuxbrew as default prefix path... https://docs.brew.sh/Homebrew-on-Linux > [..] > From pluknet at nginx.com Thu Mar 7 15:46:42 2024 From: pluknet at nginx.com (Sergey Kandaurov) Date: Thu, 7 Mar 2024 19:46:42 +0400 Subject: [PATCH] Core: free connections and read/write events at shutdown In-Reply-To: References: Message-ID: <59780B99-BFAD-4DD6-8887-3C87DDC53AC6@nginx.com> > On 28 Feb 2024, at 05:21, Piotr Sikora via nginx-devel wrote: > > # HG changeset patch > # User Piotr Sikora > # Date 1708977616 0 > # Mon Feb 26 20:00:16 2024 +0000 > # Branch patch002 > # Node ID f8d9fb94eab212f6e640b7a68ed111562e3157d5 > # Parent a8a592b9b62eff7bca03e8b46669f59d2da689ed > Core: free connections and read/write events at shutdown. > > Found with LeakSanitizer. > > Signed-off-by: Piotr Sikora > > diff -r a8a592b9b62e -r f8d9fb94eab2 src/os/unix/ngx_process_cycle.c > --- a/src/os/unix/ngx_process_cycle.c Mon Feb 26 20:00:11 2024 +0000 > +++ b/src/os/unix/ngx_process_cycle.c Mon Feb 26 20:00:16 2024 +0000 > @@ -940,6 +940,7 @@ > ngx_worker_process_exit(ngx_cycle_t *cycle) > { > ngx_uint_t i; > + ngx_event_t *rev, *wev; > ngx_connection_t *c; > > for (i = 0; cycle->modules[i]; i++) { > @@ -989,8 +990,16 @@ > ngx_exit_cycle.files_n = ngx_cycle->files_n; > ngx_cycle = &ngx_exit_cycle; > > + c = cycle->connections; > + rev = cycle->read_events; > + wev = cycle->write_events; > + > ngx_destroy_pool(cycle->pool); > > + ngx_free(c); > + ngx_free(rev); > + ngx_free(wev); > + > ngx_log_error(NGX_LOG_NOTICE, ngx_cycle->log, 0, "exit"); > > exit(0); Hello. Since this is not a real memory leak (the allocations made in the init_process method of the ngx_event_core_module module are used up to this function and already freed on exit automatically, as this function never returns), I don't think there is something to fix. (Further, the patch misses cycle->files for NGX_USE_FD_EVENT event methods. Also, it's probably better to free the memory in the exit_process method to obey the ownership (this would also fix missing ngx_free() calls on win32), but this would require moving code that uses these connections afterwards to catch socket leaks.) -- Sergey Kandaurov From arut at nginx.com Thu Mar 7 17:54:05 2024 From: arut at nginx.com (=?iso-8859-1?q?Roman_Arutyunyan?=) Date: Thu, 07 Mar 2024 21:54:05 +0400 Subject: [PATCH 0 of 2] QUIC packet batching on Linux Message-ID: Hi, The patches add sendmmsg() and recvmmsg() to QUIC on Linux instead of sendmsg() and recvmsg(). This allows to do cross-connection batching of QUIC input and output and significantly reduce the number of syscalls. So far I haven't noticed any noticable performance improvements, but the testing is still ongoing. -- Roman Arutyunyan From arut at nginx.com Thu Mar 7 17:54:06 2024 From: arut at nginx.com (=?iso-8859-1?q?Roman_Arutyunyan?=) Date: Thu, 07 Mar 2024 21:54:06 +0400 Subject: [PATCH 1 of 2] QUIC: output packet batching with sendmmsg() In-Reply-To: References: Message-ID: <5d28510b62bffba3187d.1709834046@arut-laptop> # HG changeset patch # User Roman Arutyunyan # Date 1709833123 -28800 # Fri Mar 08 01:38:43 2024 +0800 # Node ID 5d28510b62bffba3187d7fe69baccd2d2da41a12 # Parent 2ed3f57dca0a664340bca2236c7d614902db4180 QUIC: output packet batching with sendmmsg(). diff --git a/auto/os/linux b/auto/os/linux --- a/auto/os/linux +++ b/auto/os/linux @@ -291,4 +291,16 @@ ngx_feature_test="socklen_t optlen = siz . auto/feature +ngx_feature="sendmmsg()" +ngx_feature_name="NGX_HAVE_SENDMMSG" +ngx_feature_run=no +ngx_feature_incs="#include + #include " +ngx_feature_path= +ngx_feature_libs= +ngx_feature_test="struct mmsghdr msg[UIO_MAXIOV]; + sendmmsg(0, msg, UIO_MAXIOV, 0);" +. auto/feature + + CC_AUX_FLAGS="$cc_aux_flags -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64" diff --git a/auto/sources b/auto/sources --- a/auto/sources +++ b/auto/sources @@ -171,6 +171,7 @@ UNIX_SRCS="$CORE_SRCS $EVENT_SRCS \ src/os/unix/ngx_writev_chain.c \ src/os/unix/ngx_udp_send.c \ src/os/unix/ngx_udp_sendmsg_chain.c \ + src/os/unix/ngx_udp_sendmmsg.c \ src/os/unix/ngx_channel.c \ src/os/unix/ngx_shmem.c \ src/os/unix/ngx_process.c \ diff --git a/src/event/ngx_event.c b/src/event/ngx_event.c --- a/src/event/ngx_event.c +++ b/src/event/ngx_event.c @@ -823,9 +823,11 @@ ngx_event_process_init(ngx_cycle_t *cycl ls[i].connection = c; rev = c->read; + wev = c->write; rev->log = c->log; rev->accept = 1; + wev->log = c->log; #if (NGX_HAVE_DEFERRED_ACCEPT) rev->deferred_accept = ls[i].deferred_accept; @@ -897,6 +899,14 @@ ngx_event_process_init(ngx_cycle_t *cycl #if (NGX_QUIC) } else if (ls[i].quic) { rev->handler = ngx_quic_recvmsg; +#if (NGX_HAVE_SENDMMSG) + wev->handler = ngx_event_sendmmsg; + + c->data = ngx_pcalloc(cycle->pool, sizeof(ngx_sendmmsg_batch_t)); + if (c->data == NULL) { + return NGX_ERROR; + } +#endif #endif } else { rev->handler = ngx_event_recvmsg; diff --git a/src/event/ngx_event_udp.h b/src/event/ngx_event_udp.h --- a/src/event/ngx_event_udp.h +++ b/src/event/ngx_event_udp.h @@ -22,6 +22,8 @@ #endif +#define NGX_SENDMMSG_BUFFER 4194304 /* 4M */ + struct ngx_udp_connection_s { ngx_rbtree_node_t node; @@ -47,6 +49,23 @@ typedef union { #endif } ngx_addrinfo_t; + +#if (NGX_HAVE_SENDMMSG) + +typedef struct { + u_char buffer[NGX_SENDMMSG_BUFFER]; + struct mmsghdr msgvec[UIO_MAXIOV]; + size_t size; + ngx_uint_t vlen; +} ngx_sendmmsg_batch_t; + + +ssize_t ngx_sendmmsg(ngx_connection_t *c, struct msghdr *msg, int flags); +u_char *ngx_sendmmsg_buffer(ngx_connection_t *c, size_t size); +void ngx_event_sendmmsg(ngx_event_t *ev); + +#endif + size_t ngx_set_srcaddr_cmsg(struct cmsghdr *cmsg, struct sockaddr *local_sockaddr); ngx_int_t ngx_get_srcaddr_cmsg(struct cmsghdr *cmsg, diff --git a/src/event/quic/ngx_event_quic_migration.c b/src/event/quic/ngx_event_quic_migration.c --- a/src/event/quic/ngx_event_quic_migration.c +++ b/src/event/quic/ngx_event_quic_migration.c @@ -908,10 +908,12 @@ ngx_quic_expire_path_mtu_discovery(ngx_c static ngx_int_t ngx_quic_send_path_mtu_probe(ngx_connection_t *c, ngx_quic_path_t *path) { + void *bt; size_t mtu; uint64_t pnum; ngx_int_t rc; ngx_uint_t log_error; + ngx_connection_t *lc; ngx_quic_frame_t *frame; ngx_quic_send_ctx_t *ctx; ngx_quic_connection_t *qc; @@ -933,6 +935,10 @@ ngx_quic_send_path_mtu_probe(ngx_connect "mtu:%uz pnum:%uL tries:%ui", path->seqnum, path->mtud, ctx->pnum, path->tries); + lc = c->listening->connection; + bt = lc->data; + lc->data = NULL; + log_error = c->log_error; c->log_error = NGX_ERROR_IGNORE_EMSGSIZE; @@ -943,6 +949,7 @@ ngx_quic_send_path_mtu_probe(ngx_connect path->mtu = mtu; c->log_error = log_error; + lc->data = bt; if (rc == NGX_OK) { path->mtu_pnum[path->tries] = pnum; diff --git a/src/event/quic/ngx_event_quic_output.c b/src/event/quic/ngx_event_quic_output.c --- a/src/event/quic/ngx_event_quic_output.c +++ b/src/event/quic/ngx_event_quic_output.c @@ -61,6 +61,7 @@ static void ngx_quic_init_packet(ngx_con static ngx_uint_t ngx_quic_get_padding_level(ngx_connection_t *c); static ssize_t ngx_quic_send(ngx_connection_t *c, u_char *buf, size_t len, struct sockaddr *sockaddr, socklen_t socklen); +static u_char *ngx_quic_send_buffer(ngx_connection_t *c); static void ngx_quic_set_packet_number(ngx_quic_header_t *pkt, ngx_quic_send_ctx_t *ctx); @@ -114,14 +115,13 @@ ngx_quic_create_datagrams(ngx_connection { size_t len, min; ssize_t n; - u_char *p; + u_char *dst, *p; uint64_t preserved_pnum[NGX_QUIC_SEND_CTX_LAST]; ngx_uint_t i, pad; ngx_quic_path_t *path; ngx_quic_send_ctx_t *ctx; ngx_quic_congestion_t *cg; ngx_quic_connection_t *qc; - static u_char dst[NGX_QUIC_MAX_UDP_PAYLOAD_SIZE]; qc = ngx_quic_get_connection(c); cg = &qc->congestion; @@ -129,6 +129,7 @@ ngx_quic_create_datagrams(ngx_connection while (cg->in_flight < cg->window) { + dst = ngx_quic_send_buffer(c); p = dst; len = ngx_quic_path_limit(c, path, path->mtu); @@ -688,12 +689,12 @@ static ssize_t ngx_quic_send(ngx_connection_t *c, u_char *buf, size_t len, struct sockaddr *sockaddr, socklen_t socklen) { - ssize_t n; - struct iovec iov; - struct msghdr msg; + ssize_t n; + struct iovec iov; + struct msghdr msg; #if (NGX_HAVE_ADDRINFO_CMSG) - struct cmsghdr *cmsg; - char msg_control[CMSG_SPACE(sizeof(ngx_addrinfo_t))]; + struct cmsghdr *cmsg; + char msg_control[CMSG_SPACE(sizeof(ngx_addrinfo_t))]; #endif ngx_memzero(&msg, sizeof(struct msghdr)); @@ -720,7 +721,11 @@ ngx_quic_send(ngx_connection_t *c, u_cha } #endif +#if (NGX_HAVE_SENDMMSG) + n = ngx_sendmmsg(c, &msg, 0); +#else n = ngx_sendmsg(c, &msg, 0); +#endif if (n < 0) { return n; } @@ -731,6 +736,26 @@ ngx_quic_send(ngx_connection_t *c, u_cha } +static u_char * +ngx_quic_send_buffer(ngx_connection_t *c) +{ + static u_char buffer[NGX_QUIC_MAX_UDP_PAYLOAD_SIZE]; + +#if (NGX_HAVE_SENDMMSG) + + u_char *p; + + p = ngx_sendmmsg_buffer(c, NGX_QUIC_MAX_UDP_PAYLOAD_SIZE); + if (p) { + return p; + } + +#endif + + return buffer; +} + + static void ngx_quic_set_packet_number(ngx_quic_header_t *pkt, ngx_quic_send_ctx_t *ctx) { @@ -764,9 +789,9 @@ ngx_quic_set_packet_number(ngx_quic_head ngx_int_t ngx_quic_negotiate_version(ngx_connection_t *c, ngx_quic_header_t *inpkt) { - size_t len; - ngx_quic_header_t pkt; - static u_char buf[NGX_QUIC_MAX_UDP_PAYLOAD_SIZE]; + u_char *buf; + size_t len; + ngx_quic_header_t pkt; ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0, "sending version negotiation packet"); @@ -776,6 +801,8 @@ ngx_quic_negotiate_version(ngx_connectio pkt.dcid = inpkt->scid; pkt.scid = inpkt->dcid; + buf = ngx_quic_send_buffer(c); + len = ngx_quic_create_version_negotiation(&pkt, buf); #ifdef NGX_QUIC_DEBUG_PACKETS @@ -793,10 +820,9 @@ ngx_int_t ngx_quic_send_stateless_reset(ngx_connection_t *c, ngx_quic_conf_t *conf, ngx_quic_header_t *pkt) { - u_char *token; + u_char *token, *buf; size_t len, max; uint16_t rndbytes; - u_char buf[NGX_QUIC_MAX_SR_PACKET]; ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0, "quic handle stateless reset output"); @@ -819,6 +845,8 @@ ngx_quic_send_stateless_reset(ngx_connec + NGX_QUIC_MIN_SR_PACKET; } + buf = ngx_quic_send_buffer(c); + if (RAND_bytes(buf, len - NGX_QUIC_SR_TOKEN_LEN) != 1) { return NGX_ERROR; } @@ -892,9 +920,7 @@ ngx_quic_send_early_cc(ngx_connection_t ngx_quic_keys_t keys; ngx_quic_frame_t frame; ngx_quic_header_t pkt; - - static u_char src[NGX_QUIC_MAX_UDP_PAYLOAD_SIZE]; - static u_char dst[NGX_QUIC_MAX_UDP_PAYLOAD_SIZE]; + static u_char src[NGX_QUIC_MAX_UDP_PAYLOAD_SIZE]; ngx_memzero(&frame, sizeof(ngx_quic_frame_t)); ngx_memzero(&pkt, sizeof(ngx_quic_header_t)); @@ -945,7 +971,7 @@ ngx_quic_send_early_cc(ngx_connection_t pkt.payload.data = src; pkt.payload.len = len; - res.data = dst; + res.data = ngx_quic_send_buffer(c); ngx_quic_log_packet(c->log, &pkt); @@ -974,7 +1000,6 @@ ngx_quic_send_retry(ngx_connection_t *c, ngx_str_t res, token; ngx_quic_header_t pkt; - u_char buf[NGX_QUIC_RETRY_BUFFER_SIZE]; u_char dcid[NGX_QUIC_SERVER_CID_LEN]; u_char tbuf[NGX_QUIC_TOKEN_BUF_SIZE]; @@ -1008,7 +1033,7 @@ ngx_quic_send_retry(ngx_connection_t *c, pkt.token = token; - res.data = buf; + res.data = ngx_quic_send_buffer(c); if (ngx_quic_encrypt(&pkt, &res) != NGX_OK) { return NGX_ERROR; @@ -1193,9 +1218,7 @@ ngx_quic_frame_sendto(ngx_connection_t * ngx_quic_send_ctx_t *ctx; ngx_quic_congestion_t *cg; ngx_quic_connection_t *qc; - static u_char src[NGX_QUIC_MAX_UDP_PAYLOAD_SIZE]; - static u_char dst[NGX_QUIC_MAX_UDP_PAYLOAD_SIZE]; qc = ngx_quic_get_connection(c); cg = &qc->congestion; @@ -1254,7 +1277,7 @@ ngx_quic_frame_sendto(ngx_connection_t * pkt.payload.data = src; pkt.payload.len = len; - res.data = dst; + res.data = ngx_quic_send_buffer(c); ngx_quic_log_packet(c->log, &pkt); diff --git a/src/os/unix/ngx_udp_sendmmsg.c b/src/os/unix/ngx_udp_sendmmsg.c new file mode 100644 --- /dev/null +++ b/src/os/unix/ngx_udp_sendmmsg.c @@ -0,0 +1,185 @@ + +/* + * Copyright (C) Roman Arutyunyan + * Copyright (C) Nginx, Inc. + */ + + +#include +#include +#include + + +#if (NGX_HAVE_SENDMMSG) + +static void ngx_sendmmsg_flush(ngx_connection_t *lc); + + +ssize_t +ngx_sendmmsg(ngx_connection_t *c, struct msghdr *msg, int flags) +{ + u_char *p; + size_t size, aux_size; + ngx_uint_t i; + struct iovec *iov; + struct msghdr *nmsg; + ngx_connection_t *lc; + ngx_sendmmsg_batch_t *sb; + + if (c->listening == NULL || c->listening->connection->data == NULL) { + return ngx_sendmsg(c, msg, flags); + } + + lc = c->listening->connection; + sb = lc->data; + + for (i = 0, size = 0; i < msg->msg_iovlen; i++) { + size += msg->msg_iov[i].iov_len; + } + + c->sent += size; + + nmsg = &sb->msgvec[sb->vlen++].msg_hdr; + + aux_size = NGX_ALIGNMENT + sizeof(struct iovec) + + NGX_ALIGNMENT + msg->msg_namelen + + NGX_ALIGNMENT + msg->msg_controllen; + + if (sb->size + size + aux_size > NGX_SENDMMSG_BUFFER) { + *nmsg = *msg; + goto flush; + } + + ngx_memzero(nmsg, sizeof(struct msghdr)); + + p = sb->buffer + sb->size; + + for (i = 0; i < msg->msg_iovlen; i++) { + if (msg->msg_iov[i].iov_base != p) { + ngx_memcpy(p, msg->msg_iov[i].iov_base, msg->msg_iov[i].iov_len); + } + + p += msg->msg_iov[i].iov_len; + } + + p = ngx_align_ptr(p, NGX_ALIGNMENT); + iov = (struct iovec *) p; + nmsg->msg_iov = iov; + nmsg->msg_iovlen = 1; + iov->iov_base = sb->buffer + sb->size; + iov->iov_len = size; + p += sizeof(struct iovec); + + p = ngx_align_ptr(p, NGX_ALIGNMENT); + nmsg->msg_name = p; + nmsg->msg_namelen = msg->msg_namelen; + p = ngx_cpymem(p, msg->msg_name, msg->msg_namelen); + + if (msg->msg_controllen) { + p = ngx_align_ptr(p, NGX_ALIGNMENT); + nmsg->msg_control = p; + nmsg->msg_controllen = msg->msg_controllen; + p = ngx_cpymem(p, msg->msg_control, msg->msg_controllen); + } + + sb->size = p - sb->buffer; + + ngx_log_debug4(NGX_LOG_DEBUG_EVENT, c->log, 0, + "sendmmsg batch n:%ui s:%uz a:%uz t:%uz", + sb->vlen, size, aux_size, sb->size); + + if (sb->vlen == UIO_MAXIOV) { + goto flush; + } + + ngx_post_event(lc->write, &ngx_posted_events); + + return size; + +flush: + + ngx_sendmmsg_flush(lc); + + return size; +} + + +static void +ngx_sendmmsg_flush(ngx_connection_t *lc) +{ + int n; + ngx_err_t err; + ngx_sendmmsg_batch_t *sb; + + sb = lc->data; + + if (sb == NULL || sb->vlen == 0) { + return; + } + + n = sendmmsg(lc->fd, sb->msgvec, sb->vlen, 0); + + ngx_log_debug3(NGX_LOG_DEBUG_EVENT, lc->log, 0, + "sendmmsg: %d of %ui s:%uz", n, sb->vlen, sb->size); + + if (n == -1) { + err = ngx_errno; + + switch (err) { + case NGX_EAGAIN: + ngx_log_debug0(NGX_LOG_DEBUG_EVENT, lc->log, err, + "sendmmsg() not ready"); + break; + + case NGX_EINTR: + ngx_log_debug0(NGX_LOG_DEBUG_EVENT, lc->log, err, + "sendmmsg() was interrupted"); + break; + + default: + ngx_connection_error(lc, err, "sendmmsg() failed"); + break; + } + } + + sb->size = 0; + sb->vlen = 0; +} + + +void +ngx_event_sendmmsg(ngx_event_t *ev) +{ + ngx_connection_t *lc; + + lc = ev->data; + + ngx_sendmmsg_flush(lc); +} + + +u_char * +ngx_sendmmsg_buffer(ngx_connection_t *c, size_t size) +{ + ngx_connection_t *lc; + ngx_sendmmsg_batch_t *sb; + + if (c->listening == NULL || c->listening->connection->data == NULL) { + return NULL; + } + + if (size > NGX_SENDMMSG_BUFFER) { + return NULL; + } + + lc = c->listening->connection; + sb = lc->data; + + if (sb->size + size > NGX_SENDMMSG_BUFFER) { + ngx_sendmmsg_flush(lc); + } + + return sb->buffer + sb->size; +} + +#endif /* NGX_HAVE_SENDMMSG */ From arut at nginx.com Thu Mar 7 17:54:07 2024 From: arut at nginx.com (=?iso-8859-1?q?Roman_Arutyunyan?=) Date: Thu, 07 Mar 2024 21:54:07 +0400 Subject: [PATCH 2 of 2] QUIC: input packet batching with recvmmsg() In-Reply-To: References: Message-ID: <4584ba4b1d65a90f6920.1709834047@arut-laptop> # HG changeset patch # User Roman Arutyunyan # Date 1707486707 -28800 # Fri Feb 09 21:51:47 2024 +0800 # Node ID 4584ba4b1d65a90f69201cecf1f1e650c1cbd87b # Parent 5d28510b62bffba3187d7fe69baccd2d2da41a12 QUIC: input packet batching with recvmmsg(). diff --git a/auto/os/linux b/auto/os/linux --- a/auto/os/linux +++ b/auto/os/linux @@ -303,4 +303,15 @@ ngx_feature_test="struct mmsghdr msg[UIO . auto/feature +ngx_feature="recvmmsg()" +ngx_feature_name="NGX_HAVE_RECVMMSG" +ngx_feature_run=no +ngx_feature_incs="#include " +ngx_feature_path= +ngx_feature_libs= +ngx_feature_test="struct mmsghdr msg[64]; + recvmmsg(0, msg, 64, 0, NULL);" +. auto/feature + + CC_AUX_FLAGS="$cc_aux_flags -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64" diff --git a/src/event/ngx_event.c b/src/event/ngx_event.c --- a/src/event/ngx_event.c +++ b/src/event/ngx_event.c @@ -898,7 +898,11 @@ ngx_event_process_init(ngx_cycle_t *cycl #if (NGX_QUIC) } else if (ls[i].quic) { +#if (NGX_HAVE_RECVMMSG) + rev->handler = ngx_quic_recvmmsg; +#else rev->handler = ngx_quic_recvmsg; +#endif #if (NGX_HAVE_SENDMMSG) wev->handler = ngx_event_sendmmsg; diff --git a/src/event/quic/ngx_event_quic.h b/src/event/quic/ngx_event_quic.h --- a/src/event/quic/ngx_event_quic.h +++ b/src/event/quic/ngx_event_quic.h @@ -112,6 +112,9 @@ struct ngx_quic_stream_s { void ngx_quic_recvmsg(ngx_event_t *ev); +#if (NGX_HAVE_RECVMMSG) +void ngx_quic_recvmmsg(ngx_event_t *ev); +#endif void ngx_quic_run(ngx_connection_t *c, ngx_quic_conf_t *conf); ngx_connection_t *ngx_quic_open_stream(ngx_connection_t *c, ngx_uint_t bidi); void ngx_quic_finalize_connection(ngx_connection_t *c, ngx_uint_t err, diff --git a/src/event/quic/ngx_event_quic_udp.c b/src/event/quic/ngx_event_quic_udp.c --- a/src/event/quic/ngx_event_quic_udp.c +++ b/src/event/quic/ngx_event_quic_udp.c @@ -11,6 +11,7 @@ #include +static void ngx_quic_handle_msg(ngx_event_t *ev, struct msghdr *msg, size_t n); static void ngx_quic_close_accepted_connection(ngx_connection_t *c); static ngx_connection_t *ngx_quic_lookup_connection(ngx_listening_t *ls, ngx_str_t *key, struct sockaddr *local_sockaddr, socklen_t local_socklen); @@ -20,20 +21,13 @@ void ngx_quic_recvmsg(ngx_event_t *ev) { ssize_t n; - ngx_str_t key; - ngx_buf_t buf; - ngx_log_t *log; ngx_err_t err; - socklen_t socklen, local_socklen; - ngx_event_t *rev, *wev; struct iovec iov[1]; struct msghdr msg; - ngx_sockaddr_t sa, lsa; - struct sockaddr *sockaddr, *local_sockaddr; + ngx_sockaddr_t sa; ngx_listening_t *ls; ngx_event_conf_t *ecf; - ngx_connection_t *c, *lc; - ngx_quic_socket_t *qsock; + ngx_connection_t *lc; static u_char buffer[NGX_QUIC_MAX_UDP_PAYLOAD_SIZE]; #if (NGX_HAVE_ADDRINFO_CMSG) @@ -106,239 +100,10 @@ ngx_quic_recvmsg(ngx_event_t *ev) } #endif - sockaddr = msg.msg_name; - socklen = msg.msg_namelen; - - if (socklen > (socklen_t) sizeof(ngx_sockaddr_t)) { - socklen = sizeof(ngx_sockaddr_t); - } - -#if (NGX_HAVE_UNIX_DOMAIN) - - if (sockaddr->sa_family == AF_UNIX) { - struct sockaddr_un *saun = (struct sockaddr_un *) sockaddr; - - if (socklen <= (socklen_t) offsetof(struct sockaddr_un, sun_path) - || saun->sun_path[0] == '\0') - { - ngx_log_debug0(NGX_LOG_DEBUG_EVENT, ngx_cycle->log, 0, - "unbound unix socket"); - goto next; - } - } - -#endif - - local_sockaddr = ls->sockaddr; - local_socklen = ls->socklen; - -#if (NGX_HAVE_ADDRINFO_CMSG) - - if (ls->wildcard) { - struct cmsghdr *cmsg; - - ngx_memcpy(&lsa, local_sockaddr, local_socklen); - local_sockaddr = &lsa.sockaddr; - - for (cmsg = CMSG_FIRSTHDR(&msg); - cmsg != NULL; - cmsg = CMSG_NXTHDR(&msg, cmsg)) - { - if (ngx_get_srcaddr_cmsg(cmsg, local_sockaddr) == NGX_OK) { - break; - } - } - } - -#endif - - if (ngx_quic_get_packet_dcid(ev->log, buffer, n, &key) != NGX_OK) { - goto next; - } - - c = ngx_quic_lookup_connection(ls, &key, local_sockaddr, local_socklen); - - if (c) { - -#if (NGX_DEBUG) - if (c->log->log_level & NGX_LOG_DEBUG_EVENT) { - ngx_log_handler_pt handler; - - handler = c->log->handler; - c->log->handler = NULL; - - ngx_log_debug2(NGX_LOG_DEBUG_EVENT, c->log, 0, - "quic recvmsg: fd:%d n:%z", c->fd, n); - - c->log->handler = handler; - } -#endif - - ngx_memzero(&buf, sizeof(ngx_buf_t)); - - buf.pos = buffer; - buf.last = buffer + n; - buf.start = buf.pos; - buf.end = buffer + sizeof(buffer); - - qsock = ngx_quic_get_socket(c); - - ngx_memcpy(&qsock->sockaddr, sockaddr, socklen); - qsock->socklen = socklen; - - c->udp->buffer = &buf; - - rev = c->read; - rev->ready = 1; - rev->active = 0; - - rev->handler(rev); - - if (c->udp) { - c->udp->buffer = NULL; - } - - rev->ready = 0; - rev->active = 1; - - goto next; - } - -#if (NGX_STAT_STUB) - (void) ngx_atomic_fetch_add(ngx_stat_accepted, 1); -#endif - - ngx_accept_disabled = ngx_cycle->connection_n / 8 - - ngx_cycle->free_connection_n; - - c = ngx_get_connection(lc->fd, ev->log); - if (c == NULL) { - return; - } - - c->shared = 1; - c->type = SOCK_DGRAM; - c->socklen = socklen; + ngx_log_debug2(NGX_LOG_DEBUG_EVENT, ev->log, 0, + "quic recvmsg: fd:%d n:%z", lc->fd, n); -#if (NGX_STAT_STUB) - (void) ngx_atomic_fetch_add(ngx_stat_active, 1); -#endif - - c->pool = ngx_create_pool(ls->pool_size, ev->log); - if (c->pool == NULL) { - ngx_quic_close_accepted_connection(c); - return; - } - - c->sockaddr = ngx_palloc(c->pool, NGX_SOCKADDRLEN); - if (c->sockaddr == NULL) { - ngx_quic_close_accepted_connection(c); - return; - } - - ngx_memcpy(c->sockaddr, sockaddr, socklen); - - log = ngx_palloc(c->pool, sizeof(ngx_log_t)); - if (log == NULL) { - ngx_quic_close_accepted_connection(c); - return; - } - - *log = ls->log; - - c->log = log; - c->pool->log = log; - c->listening = ls; - - if (local_sockaddr == &lsa.sockaddr) { - local_sockaddr = ngx_palloc(c->pool, local_socklen); - if (local_sockaddr == NULL) { - ngx_quic_close_accepted_connection(c); - return; - } - - ngx_memcpy(local_sockaddr, &lsa, local_socklen); - } - - c->local_sockaddr = local_sockaddr; - c->local_socklen = local_socklen; - - c->buffer = ngx_create_temp_buf(c->pool, n); - if (c->buffer == NULL) { - ngx_quic_close_accepted_connection(c); - return; - } - - c->buffer->last = ngx_cpymem(c->buffer->last, buffer, n); - - rev = c->read; - wev = c->write; - - rev->active = 1; - wev->ready = 1; - - rev->log = log; - wev->log = log; - - /* - * TODO: MT: - ngx_atomic_fetch_add() - * or protection by critical section or light mutex - * - * TODO: MP: - allocated in a shared memory - * - ngx_atomic_fetch_add() - * or protection by critical section or light mutex - */ - - c->number = ngx_atomic_fetch_add(ngx_connection_counter, 1); - - c->start_time = ngx_current_msec; - -#if (NGX_STAT_STUB) - (void) ngx_atomic_fetch_add(ngx_stat_handled, 1); -#endif - - if (ls->addr_ntop) { - c->addr_text.data = ngx_pnalloc(c->pool, ls->addr_text_max_len); - if (c->addr_text.data == NULL) { - ngx_quic_close_accepted_connection(c); - return; - } - - c->addr_text.len = ngx_sock_ntop(c->sockaddr, c->socklen, - c->addr_text.data, - ls->addr_text_max_len, 0); - if (c->addr_text.len == 0) { - ngx_quic_close_accepted_connection(c); - return; - } - } - -#if (NGX_DEBUG) - { - ngx_str_t addr; - u_char text[NGX_SOCKADDR_STRLEN]; - - ngx_debug_accepted_connection(ecf, c); - - if (log->log_level & NGX_LOG_DEBUG_EVENT) { - addr.data = text; - addr.len = ngx_sock_ntop(c->sockaddr, c->socklen, text, - NGX_SOCKADDR_STRLEN, 1); - - ngx_log_debug4(NGX_LOG_DEBUG_EVENT, log, 0, - "*%uA quic recvmsg: %V fd:%d n:%z", - c->number, &addr, c->fd, n); - } - - } -#endif - - log->data = NULL; - log->handler = NULL; - - ls->handler(c); - - next: + ngx_quic_handle_msg(ev, &msg, n); if (ngx_event_flags & NGX_USE_KQUEUE_EVENT) { ev->available -= n; @@ -348,6 +113,386 @@ ngx_quic_recvmsg(ngx_event_t *ev) } +#if (NGX_HAVE_RECVMMSG) + +#define NGX_QUIC_RECVMMSG_NUM 64 + +void +ngx_quic_recvmmsg(ngx_event_t *ev) +{ + int nmsg; + size_t n; + ngx_err_t err; + ngx_int_t i; + struct iovec iov[NGX_QUIC_RECVMMSG_NUM]; + struct msghdr *msg; + ngx_sockaddr_t sa[NGX_QUIC_RECVMMSG_NUM]; + ngx_listening_t *ls; + ngx_event_conf_t *ecf; + ngx_connection_t *lc; + static u_char buffer[NGX_QUIC_RECVMMSG_NUM] + [NGX_QUIC_MAX_UDP_PAYLOAD_SIZE]; + struct mmsghdr msgvec[NGX_QUIC_RECVMMSG_NUM]; + +#if (NGX_HAVE_ADDRINFO_CMSG) + u_char msg_control[NGX_QUIC_RECVMMSG_NUM] + [CMSG_SPACE(sizeof(ngx_addrinfo_t))]; +#endif + + if (ev->timedout) { + if (ngx_enable_accept_events((ngx_cycle_t *) ngx_cycle) != NGX_OK) { + return; + } + + ev->timedout = 0; + } + + ecf = ngx_event_get_conf(ngx_cycle->conf_ctx, ngx_event_core_module); + + if (!(ngx_event_flags & NGX_USE_KQUEUE_EVENT)) { + ev->available = ecf->multi_accept; + } + + lc = ev->data; + ls = lc->listening; + ev->ready = 0; + + ngx_log_debug2(NGX_LOG_DEBUG_EVENT, ev->log, 0, + "quic recvmmsg on %V, ready: %d", + &ls->addr_text, ev->available); + + do { + + ngx_memzero(msgvec, sizeof(msgvec)); + +#if (NGX_HAVE_ADDRINFO_CMSG) + if (ls->wildcard) { + ngx_memzero(msg_control, sizeof(msg_control)); + } +#endif + + for (i = 0; i < NGX_QUIC_RECVMMSG_NUM; i++) { + msg = &msgvec[i].msg_hdr; + + iov[i].iov_base = (void *) buffer[i]; + iov[i].iov_len = NGX_QUIC_MAX_UDP_PAYLOAD_SIZE; + + msg->msg_name = &sa[i]; + msg->msg_namelen = sizeof(ngx_sockaddr_t); + msg->msg_iov = &iov[i]; + msg->msg_iovlen = 1; + +#if (NGX_HAVE_ADDRINFO_CMSG) + if (ls->wildcard) { + msg->msg_control = &msg_control[i]; + msg->msg_controllen = sizeof(msg_control); + } +#endif + } + + nmsg = recvmmsg(lc->fd, msgvec, NGX_QUIC_RECVMMSG_NUM, 0, NULL); + + if (nmsg == -1) { + err = ngx_socket_errno; + + if (err == NGX_EAGAIN) { + ngx_log_debug0(NGX_LOG_DEBUG_EVENT, ev->log, err, + "quic recvmmsg() not ready"); + return; + } + + ngx_log_error(NGX_LOG_ALERT, ev->log, err, + "quic recvmmsg() failed"); + + return; + } + + ngx_log_debug2(NGX_LOG_DEBUG_EVENT, ev->log, 0, + "quic recvmmsg: fd:%d n:%i", lc->fd, nmsg); + + for (i = 0; i < nmsg; i++) { + msg = &msgvec[i].msg_hdr; + n = msgvec[i].msg_len; + +#if (NGX_HAVE_ADDRINFO_CMSG) + if (msg->msg_flags & (MSG_TRUNC|MSG_CTRUNC)) { + ngx_log_error(NGX_LOG_ALERT, ev->log, 0, + "quic recvmmsg() truncated data"); + continue; + } +#endif + + ngx_log_debug1(NGX_LOG_DEBUG_EVENT, ev->log, 0, + "quic recvmmsg: n:%z", n); + + ngx_quic_handle_msg(ev, msg, n); + + if (ngx_event_flags & NGX_USE_KQUEUE_EVENT) { + ev->available -= n; + } + } + + } while (ev->available); +} + +#endif + + +static void +ngx_quic_handle_msg(ngx_event_t *ev, struct msghdr *msg, size_t n) +{ + u_char *buffer; + ngx_str_t key; + ngx_buf_t buf; + ngx_log_t *log; + socklen_t socklen, local_socklen; + ngx_event_t *rev, *wev; + ngx_sockaddr_t lsa; + struct sockaddr *sockaddr, *local_sockaddr; + ngx_listening_t *ls; + ngx_connection_t *c, *lc; + ngx_quic_socket_t *qsock; + + lc = ev->data; + ls = lc->listening; + + buffer = msg->msg_iov[0].iov_base; + + sockaddr = msg->msg_name; + socklen = msg->msg_namelen; + + if (socklen > (socklen_t) sizeof(ngx_sockaddr_t)) { + socklen = sizeof(ngx_sockaddr_t); + } + +#if (NGX_HAVE_UNIX_DOMAIN) + + if (sockaddr->sa_family == AF_UNIX) { + struct sockaddr_un *saun = (struct sockaddr_un *) sockaddr; + + if (socklen <= (socklen_t) offsetof(struct sockaddr_un, sun_path) + || saun->sun_path[0] == '\0') + { + ngx_log_debug0(NGX_LOG_DEBUG_EVENT, ngx_cycle->log, 0, + "unbound unix socket"); + return; + } + } + +#endif + + local_sockaddr = ls->sockaddr; + local_socklen = ls->socklen; + +#if (NGX_HAVE_ADDRINFO_CMSG) + + if (ls->wildcard) { + struct cmsghdr *cmsg; + + ngx_memcpy(&lsa, local_sockaddr, local_socklen); + local_sockaddr = &lsa.sockaddr; + + for (cmsg = CMSG_FIRSTHDR(msg); + cmsg != NULL; + cmsg = CMSG_NXTHDR(msg, cmsg)) + { + if (ngx_get_srcaddr_cmsg(cmsg, local_sockaddr) == NGX_OK) { + break; + } + } + } + +#endif + + if (ngx_quic_get_packet_dcid(ev->log, buffer, n, &key) != NGX_OK) { + return; + } + + c = ngx_quic_lookup_connection(ls, &key, local_sockaddr, local_socklen); + + if (c) { + +#if (NGX_DEBUG) + if (c->log->log_level & NGX_LOG_DEBUG_EVENT) { + ngx_log_handler_pt handler; + + handler = c->log->handler; + c->log->handler = NULL; + + ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0, "quic msg: n:%z", n); + + c->log->handler = handler; + } +#endif + + ngx_memzero(&buf, sizeof(ngx_buf_t)); + + buf.pos = buffer; + buf.last = buffer + n; + buf.start = buf.pos; + buf.end = buffer + sizeof(buffer); + + qsock = ngx_quic_get_socket(c); + + ngx_memcpy(&qsock->sockaddr, sockaddr, socklen); + qsock->socklen = socklen; + + c->udp->buffer = &buf; + + rev = c->read; + rev->ready = 1; + rev->active = 0; + + rev->handler(rev); + + if (c->udp) { + c->udp->buffer = NULL; + } + + rev->ready = 0; + rev->active = 1; + + return; + } + +#if (NGX_STAT_STUB) + (void) ngx_atomic_fetch_add(ngx_stat_accepted, 1); +#endif + + ngx_accept_disabled = ngx_cycle->connection_n / 8 + - ngx_cycle->free_connection_n; + + c = ngx_get_connection(lc->fd, ev->log); + if (c == NULL) { + return; + } + + c->shared = 1; + c->type = SOCK_DGRAM; + c->socklen = socklen; + +#if (NGX_STAT_STUB) + (void) ngx_atomic_fetch_add(ngx_stat_active, 1); +#endif + + c->pool = ngx_create_pool(ls->pool_size, ev->log); + if (c->pool == NULL) { + ngx_quic_close_accepted_connection(c); + return; + } + + c->sockaddr = ngx_palloc(c->pool, NGX_SOCKADDRLEN); + if (c->sockaddr == NULL) { + ngx_quic_close_accepted_connection(c); + return; + } + + ngx_memcpy(c->sockaddr, sockaddr, socklen); + + log = ngx_palloc(c->pool, sizeof(ngx_log_t)); + if (log == NULL) { + ngx_quic_close_accepted_connection(c); + return; + } + + *log = ls->log; + + c->log = log; + c->pool->log = log; + c->listening = ls; + + if (local_sockaddr == &lsa.sockaddr) { + local_sockaddr = ngx_palloc(c->pool, local_socklen); + if (local_sockaddr == NULL) { + ngx_quic_close_accepted_connection(c); + return; + } + + ngx_memcpy(local_sockaddr, &lsa, local_socklen); + } + + c->local_sockaddr = local_sockaddr; + c->local_socklen = local_socklen; + + c->buffer = ngx_create_temp_buf(c->pool, n); + if (c->buffer == NULL) { + ngx_quic_close_accepted_connection(c); + return; + } + + c->buffer->last = ngx_cpymem(c->buffer->last, buffer, n); + + rev = c->read; + wev = c->write; + + rev->active = 1; + wev->ready = 1; + + rev->log = log; + wev->log = log; + + /* + * TODO: MT: - ngx_atomic_fetch_add() + * or protection by critical section or light mutex + * + * TODO: MP: - allocated in a shared memory + * - ngx_atomic_fetch_add() + * or protection by critical section or light mutex + */ + + c->number = ngx_atomic_fetch_add(ngx_connection_counter, 1); + + c->start_time = ngx_current_msec; + +#if (NGX_STAT_STUB) + (void) ngx_atomic_fetch_add(ngx_stat_handled, 1); +#endif + + if (ls->addr_ntop) { + c->addr_text.data = ngx_pnalloc(c->pool, ls->addr_text_max_len); + if (c->addr_text.data == NULL) { + ngx_quic_close_accepted_connection(c); + return; + } + + c->addr_text.len = ngx_sock_ntop(c->sockaddr, c->socklen, + c->addr_text.data, + ls->addr_text_max_len, 0); + if (c->addr_text.len == 0) { + ngx_quic_close_accepted_connection(c); + return; + } + } + +#if (NGX_DEBUG) + { + ngx_str_t addr; + ngx_event_conf_t *ecf; + u_char text[NGX_SOCKADDR_STRLEN]; + + ecf = ngx_event_get_conf(ngx_cycle->conf_ctx, ngx_event_core_module); + + ngx_debug_accepted_connection(ecf, c); + + if (log->log_level & NGX_LOG_DEBUG_EVENT) { + addr.data = text; + addr.len = ngx_sock_ntop(c->sockaddr, c->socklen, text, + NGX_SOCKADDR_STRLEN, 1); + + ngx_log_debug3(NGX_LOG_DEBUG_EVENT, log, 0, + "*%uA quic msg: %V n:%z", c->number, &addr, n); + } + + } +#endif + + log->data = NULL; + log->handler = NULL; + + ls->handler(c); +} + + static void ngx_quic_close_accepted_connection(ngx_connection_t *c) { From piotr at aviatrix.com Fri Mar 8 15:07:22 2024 From: piotr at aviatrix.com (Piotr Sikora) Date: Fri, 8 Mar 2024 15:07:22 +0000 Subject: [PATCH] HTTP: stop emitting server version by default In-Reply-To: References: Message-ID: Hi Sergey, > I don't think this is a good idea to change the default behaviour > for the directive we have for a long-long time. But it's arguably a wrong behavior, and keeping it forever wrong because of a decision made ~20 years ago, doesn't seem like a particularly great idea. Also, while I'm usually all for retaining backward-compatibility, I cannot imagine this breaking anything. > It's always possible > to set `server_tokens off;' in the configuration file. Right, but if you require majority of users to change the defaults, then those defaults are not very good. > Also, this change is required a corresponding change in the > documentation on the nginx.org website. I'm happy to submit the corresponding change if the patch is accepted. Best regards, Piotr Sikora From piotr at aviatrix.com Fri Mar 8 15:31:10 2024 From: piotr at aviatrix.com (Piotr Sikora) Date: Fri, 8 Mar 2024 15:31:10 +0000 Subject: [PATCH] Configure: add support for Homebrew on Apple Silicon In-Reply-To: <2E747794-8E5F-4B41-B0AF-422DC7A10629@nginx.com> References: <2E747794-8E5F-4B41-B0AF-422DC7A10629@nginx.com> Message-ID: Hi Sergey, > An obvious question is why do you need this change. Homebrew seems > to be quite niche to pay attention. Homebrew [1] is orders of magnitude more popular than MacPorts [2], which is already supported by the configure script. > Using appropriate paths in > --with-cc-opt / --with-ld-opt should work (not tested). Everything under auto/lib can be replaced with --with-{cc,ld}-opt, so I don't really understand this reasoning. > A quick grep for MacPorts search paths suggests that some libraries > are missing in the change. If this is on purpose, please reflect > this in the description. libxml2, libxslt, and libexslt are all installed as part of Xcode, which is required to use Homebrew and compile anything on macOS. I'll ship update patch in a moment. > Apple Silicon is something from the marketing language, > using Apple ARM instead should be fine. > > Notably, Homebrew uses Hardware::CPU.arm Ruby language boolean > to make the distinction. There is no such thing as "Apple ARM". The official documentation uses the term "Apple silicon" [3], Homebrew refers to the supported platform as "Apple Silicon" [4], and Wikipedia has an article about "Apple silicon" [5]. > Further, given the smooth decay on Intel-based hardware, > I'd reduce this just to "Homebrew". But that would be misleading, seeing that the new code path doesn't do anything for Homebrew on Intel. And then, there is Homebrew on Linux [6]. [1] https://formulae.brew.sh/analytics/install-on-request/30d/ [2] https://ports.macports.org/statistics/ports/ [3] https://support.apple.com/en-us/116943 [4] https://formulae.brew.sh/formula/nginx#default [5] https://en.wikipedia.org/wiki/Apple_silicon [6] https://docs.brew.sh/Homebrew-on-Linux Best regards, Piotr Sikora From piotr at aviatrix.com Fri Mar 8 15:45:26 2024 From: piotr at aviatrix.com (Piotr Sikora) Date: Fri, 8 Mar 2024 15:45:26 +0000 Subject: [PATCH] Core: free connections and read/write events at shutdown In-Reply-To: <59780B99-BFAD-4DD6-8887-3C87DDC53AC6@nginx.com> References: <59780B99-BFAD-4DD6-8887-3C87DDC53AC6@nginx.com> Message-ID: Hi Sergey, > Since this is not a real memory leak (the allocations made in the > init_process method of the ngx_event_core_module module are used up > to this function and already freed on exit automatically, as this > function never returns), I don't think there is something to fix. Agreed, this is not a "real" leak. But fixing those "false" leaks allows you to run test with sanitizers on the CI, which can protect you and/or external contributors from introducing new bugs. > (Further, the patch misses cycle->files for NGX_USE_FD_EVENT event > methods. Good catch, thanks! > Also, it's probably better to free the memory in the > exit_process method to obey the ownership (this would also fix > missing ngx_free() calls on win32), but this would require moving > code that uses these connections afterwards to catch socket leaks.) Freeing memory in exit_process could result in use-after-free, since cleanups for the cycle->pool might still access those connections. Best regards, Piotr Sikora From teo.en.ming at protonmail.com Sat Mar 9 07:18:04 2024 From: teo.en.ming at protonmail.com (Turritopsis Dohrnii Teo En Ming) Date: Sat, 09 Mar 2024 07:18:04 +0000 Subject: nginx web server configuration file for Suprema BioStar 2 Door Access System Message-ID: Subject: nginx web server configuration file for Suprema BioStar 2 Door Access System Good day from Singapore, On 7 Mar 2024 Thursday, I was installing NEW self-signed SSL certificate for Suprema BioStar 2 door access system version 2.7.12.39 for a law firm in Singapore because the common name (CN) in the existing SSL certificate was pointing to the WRONG private IPv4 address 192.168.0.149. I have referred to the following Suprema technical support guide to install new self-signed SSL certificate for the door access system. Article: [BioStar 2] How to Apply a Private Certificate for HTTPS Link: https://support.supremainc.com/en/support/solutions/articles/24000005211--biostar-2-how-to-apply-a-private-certificate-for-https The server certificate/public key (biostar_cert.crt), private key (biostar_cert.key), PKCS12 file (biostar_cert.p12) and Java Keystore (keystore.jks) are all located inside the folder C:\Program Files\BioStar 2(x64)\nginx\conf Looking at the above directory pathname, it is apparent that the South Korean Suprema BioStar 2 door access system is using the open source nginx web server. But why are ssl_certificate and ssl_certificate_key directives NOT configured for the HTTPS section in the nginx configuration file? The entire HTTPS section was also commented out. I am baffled. Why is there a Java Keystore (keystore.jks)? Is nginx web server being used in conjunction with some type of open source Java web server? Looking forward to your reply. Thank you. I shall reproduce the nginx web server configuration file for the Suprema BioStar 2 door access system below for your reference. nginx.conf is inside C:\Program Files\BioStar 2(x64)\nginx\conf #user nobody; worker_processes 1; #error_log logs/error.log; #error_log logs/error.log notice; #error_log logs/error.log info; #pid logs/nginx.pid; events { worker_connections 1024; } http { include mime.types; default_type application/octet-stream; #log_format main '$remote_addr - $remote_user [$time_local] "$request" ' # '$status $body_bytes_sent "$http_referer" ' # '"$http_user_agent" "$http_x_forwarded_for"'; #access_log logs/access.log main; sendfile on; #tcp_nopush on; #keepalive_timeout 0; keepalive_timeout 65; #gzip on; server { listen 80; server_name localhost; #charset koi8-r; #access_log logs/host.access.log main; location / { root html; index index.html index.htm; } #error_page 404 /404.html; # redirect server error pages to the static page /50x.html # error_page 500 502 503 504 /50x.html; location = /50x.html { root html; } # proxy the PHP scripts to Apache listening on 127.0.0.1:80 # #location ~ \.php$ { # proxy_pass http://127.0.0.1; #} # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000 # #location ~ \.php$ { # root html; # fastcgi_pass 127.0.0.1:9000; # fastcgi_index index.php; # fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name; # include fastcgi_params; #} # Swagger document location location /biostar { root html; } # Report document location location /report { root html; } # FASTCGI location location /api { fastcgi_pass 127.0.0.1:9000; fastcgi_read_timeout 300; include fastcgi_params; } # WEBSOCKET location location /wsapi { proxy_pass http://127.0.0.1:9002; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; } location /webdav { autoindex on; alias html/download; client_body_temp_path html/download; dav_methods PUT DELETE MKCOL COPY MOVE; create_full_put_path on; client_body_in_file_only on; client_body_buffer_size 128K; client_max_body_size 1000M; dav_access user:rw group:rw all:r; } location /resources { root html; autoindex on; } # deny access to .htaccess files, if Apache's document root # concurs with nginx's one # #location ~ /\.ht { # deny all; #} } # another virtual host using mix of IP-, name-, and port-based configuration # #server { # listen 8000; # listen somename:8080; # server_name somename alias another.alias; # location / { # root html; # index index.html index.htm; # } #} # HTTPS server # #server { # listen 443; # server_name localhost; # ssl on; # ssl_certificate cert.pem; # ssl_certificate_key cert.key; # ssl_session_timeout 5m; # ssl_protocols SSLv2 SSLv3 TLSv1; # ssl_ciphers HIGH:!aNULL:!MD5; # ssl_prefer_server_ciphers on; # location / { # root html; # index index.html index.htm; # } #} } Regards, Mr. Turritopsis Dohrnii Teo En Ming Targeted Individual in Singapore Blogs: https://tdtemcerts.blogspot.com https://tdtemcerts.wordpress.com GIMP also stands for Government-Induced Medical Problems. From zaihan at unrealasia.net Sat Mar 9 07:57:05 2024 From: zaihan at unrealasia.net (Muhammad Nuzaihan) Date: Sat, 9 Mar 2024 15:57:05 +0800 Subject: nginx web server configuration file for Suprema BioStar 2 Door Access System In-Reply-To: References: Message-ID: Hello, I don't think nginx uses Java key store and that's specific only for Java applications. you should ask your manufacturer the details on how your door works. Also this (nginx-devel at nginx.org) mailing list is specifically for developers to discuss about code/bugfix/features and not for problems from end-users. Regards, A Singaporean living in Malaysia On Sat, Mar 9, 2024 at 3:18 PM Turritopsis Dohrnii Teo En Ming via nginx-devel wrote: > > Subject: nginx web server configuration file for Suprema BioStar 2 Door Access System > > Good day from Singapore, > > On 7 Mar 2024 Thursday, I was installing NEW self-signed SSL certificate for Suprema BioStar 2 door access system version 2.7.12.39 for a law firm in Singapore because the common name (CN) in the existing SSL certificate was pointing to the WRONG private IPv4 address 192.168.0.149. > > I have referred to the following Suprema technical support guide to install new self-signed SSL certificate for the door access system. > > Article: [BioStar 2] How to Apply a Private Certificate for HTTPS > Link: https://support.supremainc.com/en/support/solutions/articles/24000005211--biostar-2-how-to-apply-a-private-certificate-for-https > > The server certificate/public key (biostar_cert.crt), private key (biostar_cert.key), PKCS12 file (biostar_cert.p12) and Java Keystore (keystore.jks) are all located inside the folder C:\Program Files\BioStar 2(x64)\nginx\conf > > Looking at the above directory pathname, it is apparent that the South Korean Suprema BioStar 2 door access system is using the open source nginx web server. > > But why are ssl_certificate and ssl_certificate_key directives NOT configured for the HTTPS section in the nginx configuration file? The entire HTTPS section was also commented out. > > I am baffled. > > Why is there a Java Keystore (keystore.jks)? Is nginx web server being used in conjunction with some type of open source Java web server? > > Looking forward to your reply. > > Thank you. > > I shall reproduce the nginx web server configuration file for the Suprema BioStar 2 door access system below for your reference. > > nginx.conf is inside C:\Program Files\BioStar 2(x64)\nginx\conf > > > > #user nobody; > worker_processes 1; > > #error_log logs/error.log; > #error_log logs/error.log notice; > #error_log logs/error.log info; > > #pid logs/nginx.pid; > > > events { > worker_connections 1024; > } > > > http { > include mime.types; > default_type application/octet-stream; > > #log_format main '$remote_addr - $remote_user [$time_local] "$request" ' > # '$status $body_bytes_sent "$http_referer" ' > # '"$http_user_agent" "$http_x_forwarded_for"'; > > #access_log logs/access.log main; > > sendfile on; > #tcp_nopush on; > > #keepalive_timeout 0; > keepalive_timeout 65; > > #gzip on; > > server { > listen 80; > server_name localhost; > > #charset koi8-r; > > #access_log logs/host.access.log main; > > location / { > root html; > index index.html index.htm; > } > > #error_page 404 /404.html; > > # redirect server error pages to the static page /50x.html > # > error_page 500 502 503 504 /50x.html; > location = /50x.html { > root html; > } > > # proxy the PHP scripts to Apache listening on 127.0.0.1:80 > # > #location ~ \.php$ { > # proxy_pass http://127.0.0.1; > #} > > # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000 > # > #location ~ \.php$ { > # root html; > # fastcgi_pass 127.0.0.1:9000; > # fastcgi_index index.php; > # fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name; > # include fastcgi_params; > #} > > # Swagger document location > location /biostar { > root html; > } > > # Report document location > location /report { > root html; > } > > # FASTCGI location > location /api { > fastcgi_pass 127.0.0.1:9000; > fastcgi_read_timeout 300; > include fastcgi_params; > } > > # WEBSOCKET location > location /wsapi { > proxy_pass http://127.0.0.1:9002; > proxy_http_version 1.1; > proxy_set_header Upgrade $http_upgrade; > proxy_set_header Connection "upgrade"; > } > > location /webdav { > autoindex on; > alias html/download; > client_body_temp_path html/download; > dav_methods PUT DELETE MKCOL COPY MOVE; > create_full_put_path on; > client_body_in_file_only on; > client_body_buffer_size 128K; > client_max_body_size 1000M; > dav_access user:rw group:rw all:r; > } > > location /resources { > root html; > autoindex on; > } > > # deny access to .htaccess files, if Apache's document root > # concurs with nginx's one > # > #location ~ /\.ht { > # deny all; > #} > } > > > # another virtual host using mix of IP-, name-, and port-based configuration > # > #server { > # listen 8000; > # listen somename:8080; > # server_name somename alias another.alias; > > # location / { > # root html; > # index index.html index.htm; > # } > #} > > > # HTTPS server > # > #server { > # listen 443; > # server_name localhost; > > # ssl on; > # ssl_certificate cert.pem; > # ssl_certificate_key cert.key; > > # ssl_session_timeout 5m; > > # ssl_protocols SSLv2 SSLv3 TLSv1; > # ssl_ciphers HIGH:!aNULL:!MD5; > # ssl_prefer_server_ciphers on; > > # location / { > # root html; > # index index.html index.htm; > # } > #} > > } > > > > Regards, > > Mr. Turritopsis Dohrnii Teo En Ming > Targeted Individual in Singapore > Blogs: > https://tdtemcerts.blogspot.com > https://tdtemcerts.wordpress.com > GIMP also stands for Government-Induced Medical Problems. > > > > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > https://mailman.nginx.org/mailman/listinfo/nginx-devel From arut at nginx.com Mon Mar 11 12:44:15 2024 From: arut at nginx.com (Roman Arutyunyan) Date: Mon, 11 Mar 2024 16:44:15 +0400 Subject: [PATCH] Avoiding mixed socket families in PROXY protocol v1 (ticket #2594) In-Reply-To: References: <2f12c929527b2337c15e.1705920594@arut-laptop> <20240122154801.ycda4ie442ipzw6n@N00W24XTQX> <20240221132920.chmms5v3aekvmc2i@N00W24XTQX> <20240222151726.3dzpvvanswdqhbkh@N00W24XTQX> Message-ID: <20240311124415.67cz4puapxh266is@N00W24XTQX> Hi, On Wed, Mar 06, 2024 at 06:50:26PM +0400, Sergey Kandaurov wrote: > On Thu, Feb 22, 2024 at 07:17:26PM +0400, Roman Arutyunyan wrote: > > Hi, > > > > On Thu, Feb 22, 2024 at 01:59:25AM +0000, J Carter wrote: > > > Hello Roman, > > > > > > On Wed, 21 Feb 2024 17:29:52 +0400 > > > Roman Arutyunyan wrote: > > > > > > > Hi, > > > > > > > > > > > > On Wed, Jan 24, 2024 at 12:03:06AM +0300, Maxim Dounin wrote: > > > > > [..] > > > > > Also, as suggested, using the server address as obtained via PROXY > > > > > protocol from the client might be a better solution as long as the > > > > > client address was set via PROXY protocol (regardless of whether > > > > > address families match or not), and what users expect from the > > > > > "proty_protocol on;" when chaining stream proxies in the first > > > > > place. > > > > > > > Checking whether the address used in PROXY writer is in fact the address > > > > that was passed in the PROXY header, is complicated. This will either require > > > > setting a flag when PROXY address is set by realip, which is ugly. > > > > Another approach is checking if the client address written to a PROXY header > > > > matches the client address in the received PROXY header. However since > > > > currently PROXY protocol addresses are stored as text, and not all addresses > > > > have unique text repersentations, this approach would require refactoring all > > > > PROXY protocol code + realip modules to switch from text to sockaddr. > > Checking for the realip context should do the trick, it is created > once the client address was successfully substituted. We could then > parse the text repr of server address in the PROXY header and use it > in PROXY writer. This will allow not to break API. > Something like this should work called from the stream proxy module, > it allows to preserve an original server address in the PROXY header > keeping both addresses unmutated > (which seems to be the best way to solve this problem). > > : extern ngx_module_t ngx_stream_realip_module; > : > : if (ngx_stream_get_module_ctx(s, ngx_stream_realip_module)) { > : > : local_sockaddr = c->local_sockaddr; > : local_socklen = c->local_socklen; > : > : if (ngx_parse_addr(c->pool, &addr, c->proxy_protocol->dst_addr.data, > : c->proxy_protocol->dst_addr.len) > : != NGX_OK) > : { > : ngx_stream_proxy_finalize(s, NGX_STREAM_INTERNAL_SERVER_ERROR); > : return; > : } > : > : ngx_inet_set_port(addr.sockaddr, c->proxy_protocol->dst_port); > : > : c->local_sockaddr = addr.sockaddr; > : c->local_socklen = addr.socklen; > : } > : > : # call ngx_proxy_protocol_write(), other existing stuff > : > : if (ngx_stream_get_module_ctx(s, ngx_stream_realip_module)) { > : c->local_sockaddr = local_sockaddr; > : c->local_socklen = local_socklen; > : } > > Can't say I am excited from this code, though. Neither am I. > Another way is stick to the INADDR_ANY plan removing the server > address part from the PROXY header if not applicable. > > > > > > > > > I suggest that we follow the first plan (INADDR_ANY etc). > > I agree in general, but see below. > > > > > > > > > > [...] > > > > > > > > Updated patch attached. > > > > > > > # HG changeset patch > > # User Roman Arutyunyan > > # Date 1708522464 -14400 > > # Wed Feb 21 17:34:24 2024 +0400 > > # Node ID 2d9bb7b49d64576fa29a673133129f16de3cfbbe > > # Parent 44da04c2d4db94ad4eefa84b299e07c5fa4a00b9 > > Avoiding mixed socket families in PROXY protocol v1 (ticket #2010). > > > > When using realip module, remote and local addresses of a connection can belong > > to different address families. This previously resulted in generating PROXY > > protocol headers like this: > > > > PROXY TCP4 127.0.0.1 unix:/tmp/nginx1.sock 55544 0 > > Well, UNIX-domain socket addresses are somewhat rather exsotic, > especially in the PROXY protocol. I'd use an example with IPv4/IPv6. > > > > > The PROXY protocol v1 specification does not allow mixed families. The change > > substitutes server address with zero address in this case: > > > > PROXY TCP4 127.0.0.1 0.0.0.0 55544 0 > > > > As an alternative, "PROXY UNKNOWN" header could be used, which unlike this > > header does not contain any useful information about the client. > > > > Also, the above mentioned format for unix socket address is not specified in > > PROXY protocol v1 and is a by-product of internal nginx representation of it. > > The change eliminates such addresses from PROXY protocol headers as well. > > > > diff --git a/src/core/ngx_proxy_protocol.c b/src/core/ngx_proxy_protocol.c > > --- a/src/core/ngx_proxy_protocol.c > > +++ b/src/core/ngx_proxy_protocol.c > > @@ -279,7 +279,13 @@ ngx_proxy_protocol_read_port(u_char *p, > > u_char * > > ngx_proxy_protocol_write(ngx_connection_t *c, u_char *buf, u_char *last) > > { > > - ngx_uint_t port, lport; > > + socklen_t local_socklen; > > + ngx_uint_t port, lport; > > + struct sockaddr *local_sockaddr; > > + struct sockaddr_in sin; > > +#if (NGX_HAVE_INET6) > > + struct sockaddr_in6 sin6; > > +#endif > > > > if (last - buf < NGX_PROXY_PROTOCOL_V1_MAX_HEADER) { > > ngx_log_error(NGX_LOG_ALERT, c->log, 0, > > @@ -312,11 +318,35 @@ ngx_proxy_protocol_write(ngx_connection_ > > > > *buf++ = ' '; > > > > - buf += ngx_sock_ntop(c->local_sockaddr, c->local_socklen, buf, last - buf, > > - 0); > > + if (c->sockaddr->sa_family == c->local_sockaddr->sa_family) { > > + local_sockaddr = c->local_sockaddr; > > + local_socklen = c->local_socklen; > > + > > + } else { > > + switch (c->sockaddr->sa_family) { > > + > > +#if (NGX_HAVE_INET6) > > + case AF_INET6: > > + ngx_memzero(&sin6, sizeof(struct sockaddr_in6)); > > + sin6.sin6_family = AF_INET6; > > + local_sockaddr = (struct sockaddr *) &sin6; > > + local_socklen = sizeof(struct sockaddr_in6); > > + break; > > +#endif > > + > > + default: /* AF_INET */ > > + ngx_memzero(&sin, sizeof(struct sockaddr)); > > Note that although wildcard adresses are typically implemented > using all-zeroes, this is not obligated by standards and hence > is an implementation detail. POSIX mandates just the following: > > : The header shall define the following symbolic > : constant for use as a local address in the structure passed to bind(): > : > : INADDR_ANY > : IPv4 wildcard address. > > (Note: it is "IPv4 local host address." in previous editions.) > > : The header shall declare the following external > : variable: > : > : const struct in6_addr in6addr_any > : > : This variable is initialized by the system to contain the wildcard > : IPv6 address. The header also defines the > : IN6ADDR_ANY_INIT macro. This macro must be constant at compile time > : and can be used to initialize a variable of type struct in6_addr to > : the IPv6 wildcard address. > > : RATIONALE > : > : The INADDR_ANY and INADDR_BROADCAST values are byte-order-neutral > : and thus their byte order is not specified. Many implementations > : have additional constants as extensions, such as INADDR_LOOPBACK, > : that are not byte-order-neutral. Traditionally, these constants are > : in host byte order, requiring the use of htonl() when using them in > : a sockaddr_in structure. > > So it seems quite legitimate to use a different binary representation. > To be on the safe side (and improve code readability), it makes sense > to consistently set wildcard addresses explicitly, using INADDR_ANY > and in6addr_any. It is not so important in this patch though, because > addresses do not cross the kernel, used solely for ngx_sock_ntop(). > > > + sin.sin_family = AF_INET; > > + local_sockaddr = (struct sockaddr *) &sin; > > + local_socklen = sizeof(struct sockaddr_in); > > + break; > > + } > > + } > > + > > + buf += ngx_sock_ntop(local_sockaddr, local_socklen, buf, last - buf, 0); > > > > port = ngx_inet_get_port(c->sockaddr); > > - lport = ngx_inet_get_port(c->local_sockaddr); > > + lport = ngx_inet_get_port(local_sockaddr); > > > > return ngx_slprintf(buf, last, " %ui %ui" CRLF, port, lport); > > } > > Picking INADDR_ANY (and IPv6 equivalent) to substitute non-applicable > addresses due to mismatching address family may be unnecessarily > strict hiding addresses that can be represented in a different family. > > IPv4 address can be represented in the IPv4-mapped IPv6 address format. > IPv6 address can be represented in the IPv4 format if it's IPv4-mapped. What if it's not? > Such format is supported in various nginx modules and may be enabled > by "listen ... ipv6only=off" to accept IPv4-mapped IPv6 addresses on a > wildcard listen socket. > > For example, if nginx receives "PROXY TCP6" on "listen 127.0.0.1:8081" > and is configured to send PROXY protocol further in the stream proxy > chain, the server address will be updated to "::ffff:127.0.0.1 8081". > And opposite, receiving "PROXY TCP4" on "listen [::]:8082 ipv6only=off" > will update the PROXY header server part to "127.0.0.1 8082". > > That said, something like this should work on top off your change: > > diff --git a/src/core/ngx_proxy_protocol.c b/src/core/ngx_proxy_protocol.c > --- a/src/core/ngx_proxy_protocol.c > +++ b/src/core/ngx_proxy_protocol.c > @@ -279,12 +279,13 @@ ngx_proxy_protocol_read_port(u_char *p, > u_char * > ngx_proxy_protocol_write(ngx_connection_t *c, u_char *buf, u_char *last) > { > - socklen_t local_socklen; > - ngx_uint_t port, lport; > - struct sockaddr *local_sockaddr; > - struct sockaddr_in sin; > + socklen_t local_socklen; > + ngx_uint_t port, lport; > + ngx_sockaddr_t sa; > + struct sockaddr *local_sockaddr; > #if (NGX_HAVE_INET6) > - struct sockaddr_in6 sin6; > + in_addr_t addr4; > + struct in6_addr *addr6; > #endif > > if (last - buf < NGX_PROXY_PROTOCOL_V1_MAX_HEADER) { > @@ -323,22 +324,62 @@ ngx_proxy_protocol_write(ngx_connection_ > local_socklen = c->local_socklen; > > } else { > + local_sockaddr = &sa.sockaddr; > + > + ngx_memzero(&sa, sizeof(ngx_sockaddr_t)); > + sa.sockaddr.sa_family = c->sockaddr->sa_family; > + ngx_inet_set_port(&sa.sockaddr, ngx_inet_get_port(c->local_sockaddr)); > + > switch (c->sockaddr->sa_family) { > > #if (NGX_HAVE_INET6) > case AF_INET6: > - ngx_memzero(&sin6, sizeof(struct sockaddr_in6)); > - sin6.sin6_family = AF_INET6; > - local_sockaddr = (struct sockaddr *) &sin6; > local_socklen = sizeof(struct sockaddr_in6); > + > + if (c->local_sockaddr->sa_family == AF_INET) { > + addr4 = ntohl(((struct sockaddr_in *) c->local_sockaddr) > + ->sin_addr.s_addr); > + > + addr6 = &sa.sockaddr_in6.sin6_addr; > + addr6->s6_addr[10] = 0xff; > + addr6->s6_addr[11] = 0xff; > + addr6->s6_addr[12] = addr4 >> 24; > + addr6->s6_addr[13] = addr4 >> 16; > + addr6->s6_addr[14] = addr4 >> 8; > + addr6->s6_addr[15] = addr4; > + > + } else { > + sa.sockaddr_in6.sin6_addr = in6addr_any; > + } > + > break; > #endif > > default: /* AF_INET */ > - ngx_memzero(&sin, sizeof(struct sockaddr)); > - sin.sin_family = AF_INET; > - local_sockaddr = (struct sockaddr *) &sin; > local_socklen = sizeof(struct sockaddr_in); > + > +#if (NGX_HAVE_INET6) > + if (c->local_sockaddr->sa_family == AF_INET6) { > + addr6 = &((struct sockaddr_in6 *) c->local_sockaddr)->sin6_addr; > + > + if (IN6_IS_ADDR_V4MAPPED(addr6)) { > + addr4 = addr6->s6_addr[12] << 24; > + addr4 += addr6->s6_addr[13] << 16; > + addr4 += addr6->s6_addr[14] << 8; > + addr4 += addr6->s6_addr[15]; > + > + sa.sockaddr_in.sin_addr.s_addr = htonl(addr4); > + > + } else { > + sa.sockaddr_in.sin_addr.s_addr = INADDR_ANY; > + } > + > + } else > +#endif > + { > + sa.sockaddr_in.sin_addr.s_addr = INADDR_ANY; > + } > + > break; > } > } I agree we can implement a few more workarounds and make it work. It all looks like we need a normal solution even if the patch will be bigger. Attached is a patch converting PROXY protocol addresses from text to binary representation. IMHO we should've done this long ago. Once it's done, the mixed family patch will be trivial, attached as well. -- Roman Arutyunyan -------------- next part -------------- # HG changeset patch # User Roman Arutyunyan # Date 1709913161 -14400 # Fri Mar 08 19:52:41 2024 +0400 # Node ID 07c8c445746c6c74250bad7b42cd26dbb0ed6dd3 # Parent 2ed3f57dca0a664340bca2236c7d614902db4180 Storing PROXY protocol addresses as sockaddr. Previously they were stored as text. This was beneficial for v1 where the addresses do arrive as text, however counterproductive for v2, where they arrive as binary. The change allows more flexibility when checking whether PROXY protocol address was set as client address. This will be needed in the followup changes. Also, PROXY protocol debugging is improved. Now TLVs are logged after parsing. diff --git a/src/core/ngx_proxy_protocol.c b/src/core/ngx_proxy_protocol.c --- a/src/core/ngx_proxy_protocol.c +++ b/src/core/ngx_proxy_protocol.c @@ -67,9 +67,12 @@ typedef struct { static u_char *ngx_proxy_protocol_read_addr(ngx_connection_t *c, u_char *p, - u_char *last, ngx_str_t *addr); + u_char *last, ngx_addr_t *addr); static u_char *ngx_proxy_protocol_read_port(u_char *p, u_char *last, - in_port_t *port, u_char sep); + struct sockaddr *sa, u_char sep); +#if (NGX_DEBUG) +static void ngx_proxy_protocol_debug(ngx_connection_t *c, ngx_uint_t version); +#endif static u_char *ngx_proxy_protocol_v2_read(ngx_connection_t *c, u_char *buf, u_char *last); static ngx_int_t ngx_proxy_protocol_lookup_tlv(ngx_connection_t *c, @@ -101,6 +104,7 @@ ngx_proxy_protocol_read(ngx_connection_t { size_t len; u_char *p; + ngx_addr_t addr; ngx_proxy_protocol_t *pp; static const u_char signature[] = "\r\n\r\n\0\r\nQUIT\n"; @@ -141,22 +145,28 @@ ngx_proxy_protocol_read(ngx_connection_t return NULL; } - p = ngx_proxy_protocol_read_addr(c, p, last, &pp->src_addr); + p = ngx_proxy_protocol_read_addr(c, p, last, &addr); if (p == NULL) { goto invalid; } - p = ngx_proxy_protocol_read_addr(c, p, last, &pp->dst_addr); + pp->src_sockaddr = addr.sockaddr; + pp->src_socklen = addr.socklen; + + p = ngx_proxy_protocol_read_addr(c, p, last, &addr); if (p == NULL) { goto invalid; } - p = ngx_proxy_protocol_read_port(p, last, &pp->src_port, ' '); + pp->dst_sockaddr = addr.sockaddr; + pp->dst_socklen = addr.socklen; + + p = ngx_proxy_protocol_read_port(p, last, pp->src_sockaddr, ' '); if (p == NULL) { goto invalid; } - p = ngx_proxy_protocol_read_port(p, last, &pp->dst_port, CR); + p = ngx_proxy_protocol_read_port(p, last, pp->dst_sockaddr, CR); if (p == NULL) { goto invalid; } @@ -169,11 +179,11 @@ ngx_proxy_protocol_read(ngx_connection_t goto invalid; } - ngx_log_debug4(NGX_LOG_DEBUG_CORE, c->log, 0, - "PROXY protocol src: %V %d, dst: %V %d", - &pp->src_addr, pp->src_port, &pp->dst_addr, pp->dst_port); + c->proxy_protocol = pp; - c->proxy_protocol = pp; +#if (NGX_DEBUG) + ngx_proxy_protocol_debug(c, 1); +#endif return p; @@ -202,7 +212,7 @@ invalid: static u_char * ngx_proxy_protocol_read_addr(ngx_connection_t *c, u_char *p, u_char *last, - ngx_str_t *addr) + ngx_addr_t *addr) { size_t len; u_char ch, *pos; @@ -231,20 +241,16 @@ ngx_proxy_protocol_read_addr(ngx_connect len = p - pos - 1; - addr->data = ngx_pnalloc(c->pool, len); - if (addr->data == NULL) { + if (ngx_parse_addr(c->pool, addr, pos, len) != NGX_OK) { return NULL; } - ngx_memcpy(addr->data, pos, len); - addr->len = len; - return p; } static u_char * -ngx_proxy_protocol_read_port(u_char *p, u_char *last, in_port_t *port, +ngx_proxy_protocol_read_port(u_char *p, u_char *last, struct sockaddr *sa, u_char sep) { size_t len; @@ -270,11 +276,68 @@ ngx_proxy_protocol_read_port(u_char *p, return NULL; } - *port = (in_port_t) n; + ngx_inet_set_port(sa, n); return p; } +#if (NGX_DEBUG) + +static void +ngx_proxy_protocol_debug(ngx_connection_t *c, ngx_uint_t version) +{ + u_char *p; + size_t n, len; + ngx_str_t src_addr, dst_addr; + ngx_proxy_protocol_tlv_t *tlv; + u_char src_text[NGX_SOCKADDR_STRLEN]; + u_char dst_text[NGX_SOCKADDR_STRLEN]; + + if ((c->log->log_level & NGX_LOG_DEBUG_CORE) == 0) { + return; + } + + src_addr.data = src_text; + src_addr.len = ngx_sock_ntop(c->proxy_protocol->src_sockaddr, + c->proxy_protocol->src_socklen, + src_text, NGX_SOCKADDR_STRLEN, 1); + + dst_addr.data = dst_text; + dst_addr.len = ngx_sock_ntop(c->proxy_protocol->dst_sockaddr, + c->proxy_protocol->dst_socklen, + dst_text, NGX_SOCKADDR_STRLEN, 1); + + ngx_log_debug3(NGX_LOG_DEBUG_CORE, c->log, 0, + "PROXY protocol v%ui %V %V", version, &src_addr, &dst_addr); + + p = c->proxy_protocol->tlvs.data; + n = c->proxy_protocol->tlvs.len; + + for ( ;; ) { + if (n < sizeof(ngx_proxy_protocol_tlv_t)) { + break; + } + + tlv = (ngx_proxy_protocol_tlv_t *) p; + len = ngx_proxy_protocol_parse_uint16(tlv->len); + + p += sizeof(ngx_proxy_protocol_tlv_t); + n -= sizeof(ngx_proxy_protocol_tlv_t); + + if (n < len) { + break; + } + + ngx_log_debug3(NGX_LOG_DEBUG_CORE, c->log, 0, + "PROXY protocol tlv:%02xi %*xs", + (ngx_uint_t) tlv->type, len, p); + + p += len; + n -= len; + } +} + +#endif u_char * ngx_proxy_protocol_write(ngx_connection_t *c, u_char *buf, u_char *last) @@ -327,13 +390,13 @@ ngx_proxy_protocol_v2_read(ngx_connectio { u_char *end; size_t len; - socklen_t socklen; ngx_uint_t version, command, family, transport; - ngx_sockaddr_t src_sockaddr, dst_sockaddr; + struct sockaddr_in *sin; ngx_proxy_protocol_t *pp; ngx_proxy_protocol_header_t *header; ngx_proxy_protocol_inet_addrs_t *in; #if (NGX_HAVE_INET6) + struct sockaddr_in6 *sin6; ngx_proxy_protocol_inet6_addrs_t *in6; #endif @@ -394,18 +457,29 @@ ngx_proxy_protocol_v2_read(ngx_connectio in = (ngx_proxy_protocol_inet_addrs_t *) buf; - src_sockaddr.sockaddr_in.sin_family = AF_INET; - src_sockaddr.sockaddr_in.sin_port = 0; - ngx_memcpy(&src_sockaddr.sockaddr_in.sin_addr, in->src_addr, 4); + sin = ngx_pcalloc(c->pool, sizeof(struct sockaddr_in)); + if (sin == NULL) { + return NULL; + } + + sin->sin_family = AF_INET; + ngx_memcpy(&sin->sin_addr, in->src_addr, 4); + sin->sin_port = htons(ngx_proxy_protocol_parse_uint16(in->src_port)); + + pp->src_sockaddr = (struct sockaddr *) sin; + pp->src_socklen = sizeof(struct sockaddr_in); - dst_sockaddr.sockaddr_in.sin_family = AF_INET; - dst_sockaddr.sockaddr_in.sin_port = 0; - ngx_memcpy(&dst_sockaddr.sockaddr_in.sin_addr, in->dst_addr, 4); + sin = ngx_pcalloc(c->pool, sizeof(struct sockaddr_in)); + if (sin == NULL) { + return NULL; + } - pp->src_port = ngx_proxy_protocol_parse_uint16(in->src_port); - pp->dst_port = ngx_proxy_protocol_parse_uint16(in->dst_port); + sin->sin_family = AF_INET; + ngx_memcpy(&sin->sin_addr, in->dst_addr, 4); + sin->sin_port = htons(ngx_proxy_protocol_parse_uint16(in->dst_port)); - socklen = sizeof(struct sockaddr_in); + pp->dst_sockaddr = (struct sockaddr *) sin; + pp->dst_socklen = sizeof(struct sockaddr_in); buf += sizeof(ngx_proxy_protocol_inet_addrs_t); @@ -421,18 +495,29 @@ ngx_proxy_protocol_v2_read(ngx_connectio in6 = (ngx_proxy_protocol_inet6_addrs_t *) buf; - src_sockaddr.sockaddr_in6.sin6_family = AF_INET6; - src_sockaddr.sockaddr_in6.sin6_port = 0; - ngx_memcpy(&src_sockaddr.sockaddr_in6.sin6_addr, in6->src_addr, 16); + sin6 = ngx_pcalloc(c->pool, sizeof(struct sockaddr_in6)); + if (sin6 == NULL) { + return NULL; + } + + sin6->sin6_family = AF_INET6; + ngx_memcpy(&sin6->sin6_addr, in6->src_addr, 16); + sin6->sin6_port = htons(ngx_proxy_protocol_parse_uint16(in6->src_port)); + + pp->src_sockaddr = (struct sockaddr *) sin6; + pp->src_socklen = sizeof(struct sockaddr_in6); - dst_sockaddr.sockaddr_in6.sin6_family = AF_INET6; - dst_sockaddr.sockaddr_in6.sin6_port = 0; - ngx_memcpy(&dst_sockaddr.sockaddr_in6.sin6_addr, in6->dst_addr, 16); + sin6 = ngx_pcalloc(c->pool, sizeof(struct sockaddr_in6)); + if (sin6 == NULL) { + return NULL; + } - pp->src_port = ngx_proxy_protocol_parse_uint16(in6->src_port); - pp->dst_port = ngx_proxy_protocol_parse_uint16(in6->dst_port); + sin6->sin6_family = AF_INET6; + ngx_memcpy(&sin6->sin6_addr, in6->dst_addr, 16); + sin6->sin6_port = htons(ngx_proxy_protocol_parse_uint16(in6->dst_port)); - socklen = sizeof(struct sockaddr_in6); + pp->dst_sockaddr = (struct sockaddr *) sin6; + pp->dst_socklen = sizeof(struct sockaddr_in6); buf += sizeof(ngx_proxy_protocol_inet6_addrs_t); @@ -447,26 +532,6 @@ ngx_proxy_protocol_v2_read(ngx_connectio return end; } - pp->src_addr.data = ngx_pnalloc(c->pool, NGX_SOCKADDR_STRLEN); - if (pp->src_addr.data == NULL) { - return NULL; - } - - pp->src_addr.len = ngx_sock_ntop(&src_sockaddr.sockaddr, socklen, - pp->src_addr.data, NGX_SOCKADDR_STRLEN, 0); - - pp->dst_addr.data = ngx_pnalloc(c->pool, NGX_SOCKADDR_STRLEN); - if (pp->dst_addr.data == NULL) { - return NULL; - } - - pp->dst_addr.len = ngx_sock_ntop(&dst_sockaddr.sockaddr, socklen, - pp->dst_addr.data, NGX_SOCKADDR_STRLEN, 0); - - ngx_log_debug4(NGX_LOG_DEBUG_CORE, c->log, 0, - "PROXY protocol v2 src: %V %d, dst: %V %d", - &pp->src_addr, pp->src_port, &pp->dst_addr, pp->dst_port); - if (buf < end) { pp->tlvs.data = ngx_pnalloc(c->pool, end - buf); if (pp->tlvs.data == NULL) { @@ -479,6 +544,10 @@ ngx_proxy_protocol_v2_read(ngx_connectio c->proxy_protocol = pp; +#if (NGX_DEBUG) + ngx_proxy_protocol_debug(c, 2); +#endif + return end; } diff --git a/src/core/ngx_proxy_protocol.h b/src/core/ngx_proxy_protocol.h --- a/src/core/ngx_proxy_protocol.h +++ b/src/core/ngx_proxy_protocol.h @@ -18,10 +18,12 @@ struct ngx_proxy_protocol_s { - ngx_str_t src_addr; - ngx_str_t dst_addr; - in_port_t src_port; - in_port_t dst_port; + struct sockaddr *src_sockaddr; + struct sockaddr *dst_sockaddr; + + socklen_t src_socklen; + socklen_t dst_socklen; + ngx_str_t tlvs; }; diff --git a/src/http/modules/ngx_http_realip_module.c b/src/http/modules/ngx_http_realip_module.c --- a/src/http/modules/ngx_http_realip_module.c +++ b/src/http/modules/ngx_http_realip_module.c @@ -152,6 +152,8 @@ ngx_http_realip_handler(ngx_http_request return NGX_DECLINED; } + c = r->connection; + switch (rlcf->type) { case NGX_HTTP_REALIP_XREALIP: @@ -179,14 +181,18 @@ ngx_http_realip_handler(ngx_http_request case NGX_HTTP_REALIP_PROXY: - if (r->connection->proxy_protocol == NULL) { + if (c->proxy_protocol == NULL) { return NGX_DECLINED; } - value = &r->connection->proxy_protocol->src_addr; - xfwd = NULL; + if (ngx_cidr_match(c->sockaddr, rlcf->from) != NGX_OK) { + return NGX_DECLINED; + } - break; + addr.sockaddr = c->proxy_protocol->src_sockaddr; + addr.socklen = c->proxy_protocol->src_socklen; + + return ngx_http_realip_set_addr(r, &addr); default: /* NGX_HTTP_REALIP_HEADER */ @@ -225,8 +231,6 @@ ngx_http_realip_handler(ngx_http_request found: - c = r->connection; - addr.sockaddr = c->sockaddr; addr.socklen = c->socklen; /* addr.name = c->addr_text; */ @@ -235,10 +239,6 @@ found: rlcf->recursive) != NGX_DECLINED) { - if (rlcf->type == NGX_HTTP_REALIP_PROXY) { - ngx_inet_set_port(addr.sockaddr, c->proxy_protocol->src_port); - } - return ngx_http_realip_set_addr(r, &addr); } diff --git a/src/http/ngx_http_variables.c b/src/http/ngx_http_variables.c --- a/src/http/ngx_http_variables.c +++ b/src/http/ngx_http_variables.c @@ -201,20 +201,16 @@ static ngx_http_variable_t ngx_http_cor { ngx_string("remote_port"), NULL, ngx_http_variable_remote_port, 0, 0, 0 }, { ngx_string("proxy_protocol_addr"), NULL, - ngx_http_variable_proxy_protocol_addr, - offsetof(ngx_proxy_protocol_t, src_addr), 0, 0 }, + ngx_http_variable_proxy_protocol_addr, 0, 0, 0 }, { ngx_string("proxy_protocol_port"), NULL, - ngx_http_variable_proxy_protocol_port, - offsetof(ngx_proxy_protocol_t, src_port), 0, 0 }, + ngx_http_variable_proxy_protocol_port, 0, 0, 0 }, { ngx_string("proxy_protocol_server_addr"), NULL, - ngx_http_variable_proxy_protocol_addr, - offsetof(ngx_proxy_protocol_t, dst_addr), 0, 0 }, + ngx_http_variable_proxy_protocol_addr, 1, 0, 0 }, { ngx_string("proxy_protocol_server_port"), NULL, - ngx_http_variable_proxy_protocol_port, - offsetof(ngx_proxy_protocol_t, dst_port), 0, 0 }, + ngx_http_variable_proxy_protocol_port, 1, 0, 0 }, { ngx_string("proxy_protocol_tlv_"), NULL, ngx_http_variable_proxy_protocol_tlv, @@ -1340,7 +1336,8 @@ static ngx_int_t ngx_http_variable_proxy_protocol_addr(ngx_http_request_t *r, ngx_http_variable_value_t *v, uintptr_t data) { - ngx_str_t *addr; + u_char *p; + size_t n; ngx_proxy_protocol_t *pp; pp = r->connection->proxy_protocol; @@ -1349,13 +1346,25 @@ ngx_http_variable_proxy_protocol_addr(ng return NGX_OK; } - addr = (ngx_str_t *) ((char *) pp + data); - - v->len = addr->len; + p = ngx_pnalloc(r->connection->pool, NGX_SOCKADDR_STRLEN); + if (p == NULL) { + return NGX_ERROR; + } + + if (data) { + n = ngx_sock_ntop(pp->dst_sockaddr, pp->dst_socklen, + p, NGX_SOCKADDR_STRLEN, 0); + + } else { + n = ngx_sock_ntop(pp->src_sockaddr, pp->src_socklen, + p, NGX_SOCKADDR_STRLEN, 0); + } + + v->len = n; v->valid = 1; v->no_cacheable = 0; v->not_found = 0; - v->data = addr->data; + v->data = p; return NGX_OK; } @@ -1384,7 +1393,12 @@ ngx_http_variable_proxy_protocol_port(ng return NGX_ERROR; } - port = *(in_port_t *) ((char *) pp + data); + if (data) { + port = ngx_inet_get_port(pp->dst_sockaddr); + + } else { + port = ngx_inet_get_port(pp->src_sockaddr); + } if (port > 0 && port < 65536) { v->len = ngx_sprintf(v->data, "%ui", port) - v->data; diff --git a/src/mail/ngx_mail_auth_http_module.c b/src/mail/ngx_mail_auth_http_module.c --- a/src/mail/ngx_mail_auth_http_module.c +++ b/src/mail/ngx_mail_auth_http_module.c @@ -1249,11 +1249,11 @@ ngx_mail_auth_http_create_request(ngx_ma if (c->proxy_protocol) { len += sizeof("Proxy-Protocol-Addr: ") - 1 - + c->proxy_protocol->src_addr.len + sizeof(CRLF) - 1 + + NGX_SOCKADDR_STRLEN + sizeof(CRLF) - 1 + sizeof("Proxy-Protocol-Port: ") - 1 + sizeof("65535") - 1 + sizeof(CRLF) - 1 + sizeof("Proxy-Protocol-Server-Addr: ") - 1 - + c->proxy_protocol->dst_addr.len + sizeof(CRLF) - 1 + + NGX_SOCKADDR_STRLEN + sizeof(CRLF) - 1 + sizeof("Proxy-Protocol-Server-Port: ") - 1 + sizeof("65535") - 1 + sizeof(CRLF) - 1; } @@ -1352,21 +1352,23 @@ ngx_mail_auth_http_create_request(ngx_ma if (c->proxy_protocol) { b->last = ngx_cpymem(b->last, "Proxy-Protocol-Addr: ", sizeof("Proxy-Protocol-Addr: ") - 1); - b->last = ngx_copy(b->last, c->proxy_protocol->src_addr.data, - c->proxy_protocol->src_addr.len); + b->last += ngx_sock_ntop(c->proxy_protocol->src_sockaddr, + c->proxy_protocol->src_socklen, + b->last, b->end - b->last, 0); *b->last++ = CR; *b->last++ = LF; b->last = ngx_sprintf(b->last, "Proxy-Protocol-Port: %d" CRLF, - c->proxy_protocol->src_port); + (int) ngx_inet_get_port(c->proxy_protocol->src_sockaddr)); b->last = ngx_cpymem(b->last, "Proxy-Protocol-Server-Addr: ", sizeof("Proxy-Protocol-Server-Addr: ") - 1); - b->last = ngx_copy(b->last, c->proxy_protocol->dst_addr.data, - c->proxy_protocol->dst_addr.len); + b->last += ngx_sock_ntop(c->proxy_protocol->dst_sockaddr, + c->proxy_protocol->dst_socklen, + b->last, b->end - b->last, 0); *b->last++ = CR; *b->last++ = LF; b->last = ngx_sprintf(b->last, "Proxy-Protocol-Server-Port: %d" CRLF, - c->proxy_protocol->dst_port); + (int) ngx_inet_get_port(c->proxy_protocol->dst_sockaddr)); } if (s->auth_method == NGX_MAIL_AUTH_NONE) { diff --git a/src/mail/ngx_mail_realip_module.c b/src/mail/ngx_mail_realip_module.c --- a/src/mail/ngx_mail_realip_module.c +++ b/src/mail/ngx_mail_realip_module.c @@ -87,14 +87,8 @@ ngx_mail_realip_handler(ngx_mail_session return NGX_OK; } - if (ngx_parse_addr(c->pool, &addr, c->proxy_protocol->src_addr.data, - c->proxy_protocol->src_addr.len) - != NGX_OK) - { - return NGX_OK; - } - - ngx_inet_set_port(addr.sockaddr, c->proxy_protocol->src_port); + addr.sockaddr = c->proxy_protocol->src_sockaddr; + addr.socklen = c->proxy_protocol->src_socklen; return ngx_mail_realip_set_addr(s, &addr); } diff --git a/src/stream/ngx_stream_realip_module.c b/src/stream/ngx_stream_realip_module.c --- a/src/stream/ngx_stream_realip_module.c +++ b/src/stream/ngx_stream_realip_module.c @@ -116,14 +116,8 @@ ngx_stream_realip_handler(ngx_stream_ses return NGX_DECLINED; } - if (ngx_parse_addr(c->pool, &addr, c->proxy_protocol->src_addr.data, - c->proxy_protocol->src_addr.len) - != NGX_OK) - { - return NGX_DECLINED; - } - - ngx_inet_set_port(addr.sockaddr, c->proxy_protocol->src_port); + addr.sockaddr = c->proxy_protocol->src_sockaddr; + addr.socklen = c->proxy_protocol->src_socklen; return ngx_stream_realip_set_addr(s, &addr); } diff --git a/src/stream/ngx_stream_variables.c b/src/stream/ngx_stream_variables.c --- a/src/stream/ngx_stream_variables.c +++ b/src/stream/ngx_stream_variables.c @@ -66,20 +66,16 @@ static ngx_stream_variable_t ngx_stream ngx_stream_variable_remote_port, 0, 0, 0 }, { ngx_string("proxy_protocol_addr"), NULL, - ngx_stream_variable_proxy_protocol_addr, - offsetof(ngx_proxy_protocol_t, src_addr), 0, 0 }, + ngx_stream_variable_proxy_protocol_addr, 0, 0, 0 }, { ngx_string("proxy_protocol_port"), NULL, - ngx_stream_variable_proxy_protocol_port, - offsetof(ngx_proxy_protocol_t, src_port), 0, 0 }, + ngx_stream_variable_proxy_protocol_port, 0, 0, 0 }, { ngx_string("proxy_protocol_server_addr"), NULL, - ngx_stream_variable_proxy_protocol_addr, - offsetof(ngx_proxy_protocol_t, dst_addr), 0, 0 }, + ngx_stream_variable_proxy_protocol_addr, 1, 0, 0 }, { ngx_string("proxy_protocol_server_port"), NULL, - ngx_stream_variable_proxy_protocol_port, - offsetof(ngx_proxy_protocol_t, dst_port), 0, 0 }, + ngx_stream_variable_proxy_protocol_port, 1, 0, 0 }, { ngx_string("proxy_protocol_tlv_"), NULL, ngx_stream_variable_proxy_protocol_tlv, @@ -573,7 +569,8 @@ static ngx_int_t ngx_stream_variable_proxy_protocol_addr(ngx_stream_session_t *s, ngx_stream_variable_value_t *v, uintptr_t data) { - ngx_str_t *addr; + u_char *p; + size_t n; ngx_proxy_protocol_t *pp; pp = s->connection->proxy_protocol; @@ -582,13 +579,25 @@ ngx_stream_variable_proxy_protocol_addr( return NGX_OK; } - addr = (ngx_str_t *) ((char *) pp + data); + p = ngx_pnalloc(s->connection->pool, NGX_SOCKADDR_STRLEN); + if (p == NULL) { + return NGX_ERROR; + } - v->len = addr->len; + if (data) { + n = ngx_sock_ntop(pp->dst_sockaddr, pp->dst_socklen, + p, NGX_SOCKADDR_STRLEN, 0); + + } else { + n = ngx_sock_ntop(pp->src_sockaddr, pp->src_socklen, + p, NGX_SOCKADDR_STRLEN, 0); + } + + v->len = n; v->valid = 1; v->no_cacheable = 0; v->not_found = 0; - v->data = addr->data; + v->data = p; return NGX_OK; } @@ -617,7 +626,12 @@ ngx_stream_variable_proxy_protocol_port( return NGX_ERROR; } - port = *(in_port_t *) ((char *) pp + data); + if (data) { + port = ngx_inet_get_port(pp->dst_sockaddr); + + } else { + port = ngx_inet_get_port(pp->src_sockaddr); + } if (port > 0 && port < 65536) { v->len = ngx_sprintf(v->data, "%ui", port) - v->data; -------------- next part -------------- # HG changeset patch # User Roman Arutyunyan # Date 1710081227 -14400 # Sun Mar 10 18:33:47 2024 +0400 # Node ID 6957c271a15c789be51d042de6835ce02141fe58 # Parent 07c8c445746c6c74250bad7b42cd26dbb0ed6dd3 Pass PROXY protocol header from client to upstream (ticket #2010). When creating a PROXY protocol header in mail or stream, server address is taken from c->local_sockaddr, which is normally the real server address of the connection. If prior to that client address was substituted by the realip module, then client and server addresses can have different address families, which breaks PROXY protocol syntax. The change fixes this by using server address from client PROXY protocol header as well. diff --git a/src/core/ngx_proxy_protocol.c b/src/core/ngx_proxy_protocol.c --- a/src/core/ngx_proxy_protocol.c +++ b/src/core/ngx_proxy_protocol.c @@ -375,11 +375,24 @@ ngx_proxy_protocol_write(ngx_connection_ *buf++ = ' '; - buf += ngx_sock_ntop(c->local_sockaddr, c->local_socklen, buf, last - buf, - 0); + if (c->proxy_protocol + && ngx_cmp_sockaddr(c->proxy_protocol->src_sockaddr, + c->proxy_protocol->src_socklen, + c->sockaddr, c->socklen, 1) + == NGX_OK) + { + buf += ngx_sock_ntop(c->proxy_protocol->dst_sockaddr, + c->proxy_protocol->dst_socklen, + buf, last - buf, 0); + lport = ngx_inet_get_port(c->proxy_protocol->dst_sockaddr); + + } else { + buf += ngx_sock_ntop(c->local_sockaddr, c->local_socklen, + buf, last - buf, 0); + lport = ngx_inet_get_port(c->local_sockaddr); + } port = ngx_inet_get_port(c->sockaddr); - lport = ngx_inet_get_port(c->local_sockaddr); return ngx_slprintf(buf, last, " %ui %ui" CRLF, port, lport); } From pluknet at nginx.com Mon Mar 11 15:14:40 2024 From: pluknet at nginx.com (Sergey Kandaurov) Date: Mon, 11 Mar 2024 19:14:40 +0400 Subject: [PATCH] Configure: add support for Homebrew on Apple Silicon In-Reply-To: References: <2E747794-8E5F-4B41-B0AF-422DC7A10629@nginx.com> Message-ID: > On 8 Mar 2024, at 19:31, Piotr Sikora via nginx-devel wrote: > > Hi Sergey, > >> An obvious question is why do you need this change. Homebrew seems >> to be quite niche to pay attention. > > Homebrew [1] is orders of magnitude more popular than MacPorts [2], > which is already supported by the configure script. Thanks for clarifying this. > >> Using appropriate paths in >> --with-cc-opt / --with-ld-opt should work (not tested). > > Everything under auto/lib can be replaced with --with-{cc,ld}-opt, > so I don't really understand this reasoning. Given the high popularity, it may be indeed not an option. > >> A quick grep for MacPorts search paths suggests that some libraries >> are missing in the change. If this is on purpose, please reflect >> this in the description. > > libxml2, libxslt, and libexslt are all installed as part of Xcode, > which is required to use Homebrew and compile anything on macOS. This raises a question whether we need to test it for MacPorts as well, which also requires SDK. Obviously, it is out of scope of this patch. > > I'll ship update patch in a moment. > >> Apple Silicon is something from the marketing language, >> using Apple ARM instead should be fine. >> >> Notably, Homebrew uses Hardware::CPU.arm Ruby language boolean >> to make the distinction. > > There is no such thing as "Apple ARM". > > The official documentation uses the term "Apple silicon" [3], > Homebrew refers to the supported platform as "Apple Silicon" [4], > and Wikipedia has an article about "Apple silicon" [5]. Thank you for clarification. > >> Further, given the smooth decay on Intel-based hardware, >> I'd reduce this just to "Homebrew". > > But that would be misleading, seeing that the new code path doesn't do > anything for Homebrew on Intel. > > And then, there is Homebrew on Linux [6]. Given all the above argumentation, it makes sense to push the patch just as is. > > [1] https://formulae.brew.sh/analytics/install-on-request/30d/ > [2] https://ports.macports.org/statistics/ports/ > [3] https://support.apple.com/en-us/116943 > [4] https://formulae.brew.sh/formula/nginx#default > [5] https://en.wikipedia.org/wiki/Apple_silicon > [6] https://docs.brew.sh/Homebrew-on-Linux > > Best regards, > Piotr Sikora > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > https://mailman.nginx.org/mailman/listinfo/nginx-devel -- Sergey Kandaurov From pluknet at nginx.com Wed Mar 13 17:08:43 2024 From: pluknet at nginx.com (Sergey Kandaurov) Date: Wed, 13 Mar 2024 21:08:43 +0400 Subject: [PATCH] Avoiding mixed socket families in PROXY protocol v1 (ticket #2594) In-Reply-To: <20240311124415.67cz4puapxh266is@N00W24XTQX> References: <2f12c929527b2337c15e.1705920594@arut-laptop> <20240122154801.ycda4ie442ipzw6n@N00W24XTQX> <20240221132920.chmms5v3aekvmc2i@N00W24XTQX> <20240222151726.3dzpvvanswdqhbkh@N00W24XTQX> <20240311124415.67cz4puapxh266is@N00W24XTQX> Message-ID: <6leyh7kuhr5uwqf6mm4xvafnfkcunnlbe7klwjsbdif5do5ojh@3evbd5fwq5qx> On Mon, Mar 11, 2024 at 04:44:15PM +0400, Roman Arutyunyan wrote: > Hi, > > On Wed, Mar 06, 2024 at 06:50:26PM +0400, Sergey Kandaurov wrote: > > On Thu, Feb 22, 2024 at 07:17:26PM +0400, Roman Arutyunyan wrote: > > > Hi, > > > > > > On Thu, Feb 22, 2024 at 01:59:25AM +0000, J Carter wrote: > > > > Hello Roman, > > > > > > > > On Wed, 21 Feb 2024 17:29:52 +0400 > > > > Roman Arutyunyan wrote: > > > > > > > > > Hi, > > > > > > > > > > > > > > > > On Wed, Jan 24, 2024 at 12:03:06AM +0300, Maxim Dounin wrote: > > > > > > [..] > > > > > > Also, as suggested, using the server address as obtained via PROXY > > > > > > protocol from the client might be a better solution as long as the > > > > > > client address was set via PROXY protocol (regardless of whether > > > > > > address families match or not), and what users expect from the > > > > > > "proty_protocol on;" when chaining stream proxies in the first > > > > > > place. > > > > > > > > > Checking whether the address used in PROXY writer is in fact the address > > > > > that was passed in the PROXY header, is complicated. This will either require > > > > > setting a flag when PROXY address is set by realip, which is ugly. > > > > > Another approach is checking if the client address written to a PROXY header > > > > > matches the client address in the received PROXY header. However since > > > > > currently PROXY protocol addresses are stored as text, and not all addresses > > > > > have unique text repersentations, this approach would require refactoring all > > > > > PROXY protocol code + realip modules to switch from text to sockaddr. > > > > Checking for the realip context should do the trick, it is created > > once the client address was successfully substituted. We could then > > parse the text repr of server address in the PROXY header and use it > > in PROXY writer. This will allow not to break API. > > Something like this should work called from the stream proxy module, > > it allows to preserve an original server address in the PROXY header > > keeping both addresses unmutated > > (which seems to be the best way to solve this problem). > > > > : extern ngx_module_t ngx_stream_realip_module; > > : > > : if (ngx_stream_get_module_ctx(s, ngx_stream_realip_module)) { > > : > > : local_sockaddr = c->local_sockaddr; > > : local_socklen = c->local_socklen; > > : > > : if (ngx_parse_addr(c->pool, &addr, c->proxy_protocol->dst_addr.data, > > : c->proxy_protocol->dst_addr.len) > > : != NGX_OK) > > : { > > : ngx_stream_proxy_finalize(s, NGX_STREAM_INTERNAL_SERVER_ERROR); > > : return; > > : } > > : > > : ngx_inet_set_port(addr.sockaddr, c->proxy_protocol->dst_port); > > : > > : c->local_sockaddr = addr.sockaddr; > > : c->local_socklen = addr.socklen; > > : } > > : > > : # call ngx_proxy_protocol_write(), other existing stuff > > : > > : if (ngx_stream_get_module_ctx(s, ngx_stream_realip_module)) { > > : c->local_sockaddr = local_sockaddr; > > : c->local_socklen = local_socklen; > > : } > > > > Can't say I am excited from this code, though. > > Neither am I. > > > Another way is stick to the INADDR_ANY plan removing the server > > address part from the PROXY header if not applicable. > > > > > > > > > > > > I suggest that we follow the first plan (INADDR_ANY etc). > > > > I agree in general, but see below. > > > > > > > > > > > > > [...] > > > > > > > > > > Updated patch attached. > > > > > > > > > > # HG changeset patch > > > # User Roman Arutyunyan > > > # Date 1708522464 -14400 > > > # Wed Feb 21 17:34:24 2024 +0400 > > > # Node ID 2d9bb7b49d64576fa29a673133129f16de3cfbbe > > > # Parent 44da04c2d4db94ad4eefa84b299e07c5fa4a00b9 > > > Avoiding mixed socket families in PROXY protocol v1 (ticket #2010). > > > > > > When using realip module, remote and local addresses of a connection can belong > > > to different address families. This previously resulted in generating PROXY > > > protocol headers like this: > > > > > > PROXY TCP4 127.0.0.1 unix:/tmp/nginx1.sock 55544 0 > > > > Well, UNIX-domain socket addresses are somewhat rather exsotic, > > especially in the PROXY protocol. I'd use an example with IPv4/IPv6. > > > > > > > > The PROXY protocol v1 specification does not allow mixed families. The change > > > substitutes server address with zero address in this case: > > > > > > PROXY TCP4 127.0.0.1 0.0.0.0 55544 0 > > > > > > As an alternative, "PROXY UNKNOWN" header could be used, which unlike this > > > header does not contain any useful information about the client. > > > > > > Also, the above mentioned format for unix socket address is not specified in > > > PROXY protocol v1 and is a by-product of internal nginx representation of it. > > > The change eliminates such addresses from PROXY protocol headers as well. > > > > > > diff --git a/src/core/ngx_proxy_protocol.c b/src/core/ngx_proxy_protocol.c > > > --- a/src/core/ngx_proxy_protocol.c > > > +++ b/src/core/ngx_proxy_protocol.c > > > @@ -279,7 +279,13 @@ ngx_proxy_protocol_read_port(u_char *p, > > > u_char * > > > ngx_proxy_protocol_write(ngx_connection_t *c, u_char *buf, u_char *last) > > > { > > > - ngx_uint_t port, lport; > > > + socklen_t local_socklen; > > > + ngx_uint_t port, lport; > > > + struct sockaddr *local_sockaddr; > > > + struct sockaddr_in sin; > > > +#if (NGX_HAVE_INET6) > > > + struct sockaddr_in6 sin6; > > > +#endif > > > > > > if (last - buf < NGX_PROXY_PROTOCOL_V1_MAX_HEADER) { > > > ngx_log_error(NGX_LOG_ALERT, c->log, 0, > > > @@ -312,11 +318,35 @@ ngx_proxy_protocol_write(ngx_connection_ > > > > > > *buf++ = ' '; > > > > > > - buf += ngx_sock_ntop(c->local_sockaddr, c->local_socklen, buf, last - buf, > > > - 0); > > > + if (c->sockaddr->sa_family == c->local_sockaddr->sa_family) { > > > + local_sockaddr = c->local_sockaddr; > > > + local_socklen = c->local_socklen; > > > + > > > + } else { > > > + switch (c->sockaddr->sa_family) { > > > + > > > +#if (NGX_HAVE_INET6) > > > + case AF_INET6: > > > + ngx_memzero(&sin6, sizeof(struct sockaddr_in6)); > > > + sin6.sin6_family = AF_INET6; > > > + local_sockaddr = (struct sockaddr *) &sin6; > > > + local_socklen = sizeof(struct sockaddr_in6); > > > + break; > > > +#endif > > > + > > > + default: /* AF_INET */ > > > + ngx_memzero(&sin, sizeof(struct sockaddr)); > > > > Note that although wildcard adresses are typically implemented > > using all-zeroes, this is not obligated by standards and hence > > is an implementation detail. POSIX mandates just the following: > > > > : The header shall define the following symbolic > > : constant for use as a local address in the structure passed to bind(): > > : > > : INADDR_ANY > > : IPv4 wildcard address. > > > > (Note: it is "IPv4 local host address." in previous editions.) > > > > : The header shall declare the following external > > : variable: > > : > > : const struct in6_addr in6addr_any > > : > > : This variable is initialized by the system to contain the wildcard > > : IPv6 address. The header also defines the > > : IN6ADDR_ANY_INIT macro. This macro must be constant at compile time > > : and can be used to initialize a variable of type struct in6_addr to > > : the IPv6 wildcard address. > > > > : RATIONALE > > : > > : The INADDR_ANY and INADDR_BROADCAST values are byte-order-neutral > > : and thus their byte order is not specified. Many implementations > > : have additional constants as extensions, such as INADDR_LOOPBACK, > > : that are not byte-order-neutral. Traditionally, these constants are > > : in host byte order, requiring the use of htonl() when using them in > > : a sockaddr_in structure. > > > > So it seems quite legitimate to use a different binary representation. > > To be on the safe side (and improve code readability), it makes sense > > to consistently set wildcard addresses explicitly, using INADDR_ANY > > and in6addr_any. It is not so important in this patch though, because > > addresses do not cross the kernel, used solely for ngx_sock_ntop(). > > > > > + sin.sin_family = AF_INET; > > > + local_sockaddr = (struct sockaddr *) &sin; > > > + local_socklen = sizeof(struct sockaddr_in); > > > + break; > > > + } > > > + } > > > + > > > + buf += ngx_sock_ntop(local_sockaddr, local_socklen, buf, last - buf, 0); > > > > > > port = ngx_inet_get_port(c->sockaddr); > > > - lport = ngx_inet_get_port(c->local_sockaddr); > > > + lport = ngx_inet_get_port(local_sockaddr); > > > > > > return ngx_slprintf(buf, last, " %ui %ui" CRLF, port, lport); > > > } > > > > Picking INADDR_ANY (and IPv6 equivalent) to substitute non-applicable > > addresses due to mismatching address family may be unnecessarily > > strict hiding addresses that can be represented in a different family. > > > > IPv4 address can be represented in the IPv4-mapped IPv6 address format. > > IPv6 address can be represented in the IPv4 format if it's IPv4-mapped. > > What if it's not? This is not a silver bullet. If not, such address is replaced with INADDR_ANY (or IPv6 equivalent). [..] > I agree we can implement a few more workarounds and make it work. > It all looks like we need a normal solution even if the patch will be bigger. > Attached is a patch converting PROXY protocol addresses from text to binary > representation. IMHO we should've done this long ago. Once it's done, > the mixed family patch will be trivial, attached as well. > > -- > Roman Arutyunyan > # HG changeset patch > # User Roman Arutyunyan > # Date 1709913161 -14400 > # Fri Mar 08 19:52:41 2024 +0400 > # Node ID 07c8c445746c6c74250bad7b42cd26dbb0ed6dd3 > # Parent 2ed3f57dca0a664340bca2236c7d614902db4180 > Storing PROXY protocol addresses as sockaddr. > > Previously they were stored as text. This was beneficial for v1 where the Nitpicking on log description: s/was/is/ ? Using text format is still beneficial for v1. > addresses do arrive as text, however counterproductive for v2, where they - using emphasising "do" verb here looks unreasonable - inconsistent comma before "where" > arrive as binary. > > The change allows more flexibility when checking whether PROXY protocol > address was set as client address. This will be needed in the followup changes. > > Also, PROXY protocol debugging is improved. Now TLVs are logged after parsing. While logging certainly looks more convenient now, because all TLVs are logged at once, this somewhat duplicates logging of $proxy_protocol_tlv_ variable evaluation, see ngx_proxy_protocol_get_tlv(). IMHO, it makes sense to remove it from there. > > diff --git a/src/core/ngx_proxy_protocol.c b/src/core/ngx_proxy_protocol.c > --- a/src/core/ngx_proxy_protocol.c > +++ b/src/core/ngx_proxy_protocol.c > @@ -67,9 +67,12 @@ typedef struct { > > > static u_char *ngx_proxy_protocol_read_addr(ngx_connection_t *c, u_char *p, > - u_char *last, ngx_str_t *addr); > + u_char *last, ngx_addr_t *addr); > static u_char *ngx_proxy_protocol_read_port(u_char *p, u_char *last, > - in_port_t *port, u_char sep); > + struct sockaddr *sa, u_char sep); > +#if (NGX_DEBUG) > +static void ngx_proxy_protocol_debug(ngx_connection_t *c, ngx_uint_t version); I suggest to rename it to ngx_proxy_protocol_log(), for consistency with ngx_ssl_handshake_log(), and place below all callers including ngx_proxy_protocol_v2_read(). > +#endif > static u_char *ngx_proxy_protocol_v2_read(ngx_connection_t *c, u_char *buf, > u_char *last); > static ngx_int_t ngx_proxy_protocol_lookup_tlv(ngx_connection_t *c, > @@ -101,6 +104,7 @@ ngx_proxy_protocol_read(ngx_connection_t > { > size_t len; > u_char *p; > + ngx_addr_t addr; > ngx_proxy_protocol_t *pp; > > static const u_char signature[] = "\r\n\r\n\0\r\nQUIT\n"; > @@ -141,22 +145,28 @@ ngx_proxy_protocol_read(ngx_connection_t > return NULL; > } > > - p = ngx_proxy_protocol_read_addr(c, p, last, &pp->src_addr); > + p = ngx_proxy_protocol_read_addr(c, p, last, &addr); > if (p == NULL) { > goto invalid; > } > > - p = ngx_proxy_protocol_read_addr(c, p, last, &pp->dst_addr); > + pp->src_sockaddr = addr.sockaddr; > + pp->src_socklen = addr.socklen; > + > + p = ngx_proxy_protocol_read_addr(c, p, last, &addr); > if (p == NULL) { > goto invalid; > } > > - p = ngx_proxy_protocol_read_port(p, last, &pp->src_port, ' '); > + pp->dst_sockaddr = addr.sockaddr; > + pp->dst_socklen = addr.socklen; > + > + p = ngx_proxy_protocol_read_port(p, last, pp->src_sockaddr, ' '); > if (p == NULL) { > goto invalid; > } > > - p = ngx_proxy_protocol_read_port(p, last, &pp->dst_port, CR); > + p = ngx_proxy_protocol_read_port(p, last, pp->dst_sockaddr, CR); > if (p == NULL) { > goto invalid; > } > @@ -169,11 +179,11 @@ ngx_proxy_protocol_read(ngx_connection_t > goto invalid; > } > > - ngx_log_debug4(NGX_LOG_DEBUG_CORE, c->log, 0, > - "PROXY protocol src: %V %d, dst: %V %d", > - &pp->src_addr, pp->src_port, &pp->dst_addr, pp->dst_port); > + c->proxy_protocol = pp; > > - c->proxy_protocol = pp; > +#if (NGX_DEBUG) > + ngx_proxy_protocol_debug(c, 1); > +#endif > > return p; > > @@ -202,7 +212,7 @@ invalid: > > static u_char * > ngx_proxy_protocol_read_addr(ngx_connection_t *c, u_char *p, u_char *last, > - ngx_str_t *addr) > + ngx_addr_t *addr) > { > size_t len; > u_char ch, *pos; > @@ -231,20 +241,16 @@ ngx_proxy_protocol_read_addr(ngx_connect > > len = p - pos - 1; > > - addr->data = ngx_pnalloc(c->pool, len); > - if (addr->data == NULL) { > + if (ngx_parse_addr(c->pool, addr, pos, len) != NGX_OK) { > return NULL; > } > > - ngx_memcpy(addr->data, pos, len); > - addr->len = len; > - > return p; > } > > > static u_char * > -ngx_proxy_protocol_read_port(u_char *p, u_char *last, in_port_t *port, > +ngx_proxy_protocol_read_port(u_char *p, u_char *last, struct sockaddr *sa, > u_char sep) > { > size_t len; > @@ -270,11 +276,68 @@ ngx_proxy_protocol_read_port(u_char *p, > return NULL; > } > > - *port = (in_port_t) n; > + ngx_inet_set_port(sa, n); "n" still needs to be casted to "in_port_t". This used to fix build on MSVC in the past. See ngx_parse_addr_port(), which is very similar. > > return p; > } style: two blank lines here > > +#if (NGX_DEBUG) > + > +static void > +ngx_proxy_protocol_debug(ngx_connection_t *c, ngx_uint_t version) > +{ > + u_char *p; > + size_t n, len; > + ngx_str_t src_addr, dst_addr; > + ngx_proxy_protocol_tlv_t *tlv; > + u_char src_text[NGX_SOCKADDR_STRLEN]; > + u_char dst_text[NGX_SOCKADDR_STRLEN]; > + > + if ((c->log->log_level & NGX_LOG_DEBUG_CORE) == 0) { > + return; > + } > + > + src_addr.data = src_text; > + src_addr.len = ngx_sock_ntop(c->proxy_protocol->src_sockaddr, > + c->proxy_protocol->src_socklen, > + src_text, NGX_SOCKADDR_STRLEN, 1); > + > + dst_addr.data = dst_text; > + dst_addr.len = ngx_sock_ntop(c->proxy_protocol->dst_sockaddr, > + c->proxy_protocol->dst_socklen, > + dst_text, NGX_SOCKADDR_STRLEN, 1); > + > + ngx_log_debug3(NGX_LOG_DEBUG_CORE, c->log, 0, > + "PROXY protocol v%ui %V %V", version, &src_addr, &dst_addr); > + > + p = c->proxy_protocol->tlvs.data; > + n = c->proxy_protocol->tlvs.len; > + > + for ( ;; ) { > + if (n < sizeof(ngx_proxy_protocol_tlv_t)) { > + break; > + } > + > + tlv = (ngx_proxy_protocol_tlv_t *) p; > + len = ngx_proxy_protocol_parse_uint16(tlv->len); > + > + p += sizeof(ngx_proxy_protocol_tlv_t); > + n -= sizeof(ngx_proxy_protocol_tlv_t); > + > + if (n < len) { > + break; > + } > + > + ngx_log_debug3(NGX_LOG_DEBUG_CORE, c->log, 0, > + "PROXY protocol tlv:%02xi %*xs", > + (ngx_uint_t) tlv->type, len, p); > + > + p += len; > + n -= len; > + } > +} > + > +#endif and there > > u_char * > ngx_proxy_protocol_write(ngx_connection_t *c, u_char *buf, u_char *last) > @@ -327,13 +390,13 @@ ngx_proxy_protocol_v2_read(ngx_connectio > { > u_char *end; > size_t len; > - socklen_t socklen; > ngx_uint_t version, command, family, transport; > - ngx_sockaddr_t src_sockaddr, dst_sockaddr; > + struct sockaddr_in *sin; > ngx_proxy_protocol_t *pp; > ngx_proxy_protocol_header_t *header; > ngx_proxy_protocol_inet_addrs_t *in; > #if (NGX_HAVE_INET6) > + struct sockaddr_in6 *sin6; > ngx_proxy_protocol_inet6_addrs_t *in6; > #endif > > @@ -394,18 +457,29 @@ ngx_proxy_protocol_v2_read(ngx_connectio > > in = (ngx_proxy_protocol_inet_addrs_t *) buf; > > - src_sockaddr.sockaddr_in.sin_family = AF_INET; > - src_sockaddr.sockaddr_in.sin_port = 0; > - ngx_memcpy(&src_sockaddr.sockaddr_in.sin_addr, in->src_addr, 4); > + sin = ngx_pcalloc(c->pool, sizeof(struct sockaddr_in)); > + if (sin == NULL) { > + return NULL; > + } > + > + sin->sin_family = AF_INET; > + ngx_memcpy(&sin->sin_addr, in->src_addr, 4); > + sin->sin_port = htons(ngx_proxy_protocol_parse_uint16(in->src_port)); > + > + pp->src_sockaddr = (struct sockaddr *) sin; > + pp->src_socklen = sizeof(struct sockaddr_in); > > - dst_sockaddr.sockaddr_in.sin_family = AF_INET; > - dst_sockaddr.sockaddr_in.sin_port = 0; > - ngx_memcpy(&dst_sockaddr.sockaddr_in.sin_addr, in->dst_addr, 4); > + sin = ngx_pcalloc(c->pool, sizeof(struct sockaddr_in)); > + if (sin == NULL) { > + return NULL; > + } > > - pp->src_port = ngx_proxy_protocol_parse_uint16(in->src_port); > - pp->dst_port = ngx_proxy_protocol_parse_uint16(in->dst_port); > + sin->sin_family = AF_INET; > + ngx_memcpy(&sin->sin_addr, in->dst_addr, 4); > + sin->sin_port = htons(ngx_proxy_protocol_parse_uint16(in->dst_port)); > > - socklen = sizeof(struct sockaddr_in); > + pp->dst_sockaddr = (struct sockaddr *) sin; > + pp->dst_socklen = sizeof(struct sockaddr_in); > > buf += sizeof(ngx_proxy_protocol_inet_addrs_t); > > @@ -421,18 +495,29 @@ ngx_proxy_protocol_v2_read(ngx_connectio > > in6 = (ngx_proxy_protocol_inet6_addrs_t *) buf; > > - src_sockaddr.sockaddr_in6.sin6_family = AF_INET6; > - src_sockaddr.sockaddr_in6.sin6_port = 0; > - ngx_memcpy(&src_sockaddr.sockaddr_in6.sin6_addr, in6->src_addr, 16); > + sin6 = ngx_pcalloc(c->pool, sizeof(struct sockaddr_in6)); > + if (sin6 == NULL) { > + return NULL; > + } > + > + sin6->sin6_family = AF_INET6; > + ngx_memcpy(&sin6->sin6_addr, in6->src_addr, 16); > + sin6->sin6_port = htons(ngx_proxy_protocol_parse_uint16(in6->src_port)); > + > + pp->src_sockaddr = (struct sockaddr *) sin6; > + pp->src_socklen = sizeof(struct sockaddr_in6); > > - dst_sockaddr.sockaddr_in6.sin6_family = AF_INET6; > - dst_sockaddr.sockaddr_in6.sin6_port = 0; > - ngx_memcpy(&dst_sockaddr.sockaddr_in6.sin6_addr, in6->dst_addr, 16); > + sin6 = ngx_pcalloc(c->pool, sizeof(struct sockaddr_in6)); > + if (sin6 == NULL) { > + return NULL; > + } > > - pp->src_port = ngx_proxy_protocol_parse_uint16(in6->src_port); > - pp->dst_port = ngx_proxy_protocol_parse_uint16(in6->dst_port); > + sin6->sin6_family = AF_INET6; > + ngx_memcpy(&sin6->sin6_addr, in6->dst_addr, 16); > + sin6->sin6_port = htons(ngx_proxy_protocol_parse_uint16(in6->dst_port)); > > - socklen = sizeof(struct sockaddr_in6); > + pp->dst_sockaddr = (struct sockaddr *) sin6; > + pp->dst_socklen = sizeof(struct sockaddr_in6); > > buf += sizeof(ngx_proxy_protocol_inet6_addrs_t); > > @@ -447,26 +532,6 @@ ngx_proxy_protocol_v2_read(ngx_connectio > return end; > } > > - pp->src_addr.data = ngx_pnalloc(c->pool, NGX_SOCKADDR_STRLEN); > - if (pp->src_addr.data == NULL) { > - return NULL; > - } > - > - pp->src_addr.len = ngx_sock_ntop(&src_sockaddr.sockaddr, socklen, > - pp->src_addr.data, NGX_SOCKADDR_STRLEN, 0); > - > - pp->dst_addr.data = ngx_pnalloc(c->pool, NGX_SOCKADDR_STRLEN); > - if (pp->dst_addr.data == NULL) { > - return NULL; > - } > - > - pp->dst_addr.len = ngx_sock_ntop(&dst_sockaddr.sockaddr, socklen, > - pp->dst_addr.data, NGX_SOCKADDR_STRLEN, 0); > - > - ngx_log_debug4(NGX_LOG_DEBUG_CORE, c->log, 0, > - "PROXY protocol v2 src: %V %d, dst: %V %d", > - &pp->src_addr, pp->src_port, &pp->dst_addr, pp->dst_port); > - > if (buf < end) { > pp->tlvs.data = ngx_pnalloc(c->pool, end - buf); > if (pp->tlvs.data == NULL) { > @@ -479,6 +544,10 @@ ngx_proxy_protocol_v2_read(ngx_connectio > > c->proxy_protocol = pp; > > +#if (NGX_DEBUG) > + ngx_proxy_protocol_debug(c, 2); > +#endif > + > return end; > } > > diff --git a/src/core/ngx_proxy_protocol.h b/src/core/ngx_proxy_protocol.h > --- a/src/core/ngx_proxy_protocol.h > +++ b/src/core/ngx_proxy_protocol.h > @@ -18,10 +18,12 @@ > > > struct ngx_proxy_protocol_s { > - ngx_str_t src_addr; > - ngx_str_t dst_addr; > - in_port_t src_port; > - in_port_t dst_port; > + struct sockaddr *src_sockaddr; > + struct sockaddr *dst_sockaddr; > + > + socklen_t src_socklen; > + socklen_t dst_socklen; > + > ngx_str_t tlvs; > }; > > diff --git a/src/http/modules/ngx_http_realip_module.c b/src/http/modules/ngx_http_realip_module.c > --- a/src/http/modules/ngx_http_realip_module.c > +++ b/src/http/modules/ngx_http_realip_module.c > @@ -152,6 +152,8 @@ ngx_http_realip_handler(ngx_http_request > return NGX_DECLINED; > } > > + c = r->connection; > + > switch (rlcf->type) { > > case NGX_HTTP_REALIP_XREALIP: > @@ -179,14 +181,18 @@ ngx_http_realip_handler(ngx_http_request > > case NGX_HTTP_REALIP_PROXY: > > - if (r->connection->proxy_protocol == NULL) { > + if (c->proxy_protocol == NULL) { > return NGX_DECLINED; > } > > - value = &r->connection->proxy_protocol->src_addr; > - xfwd = NULL; > + if (ngx_cidr_match(c->sockaddr, rlcf->from) != NGX_OK) { > + return NGX_DECLINED; > + } > > - break; > + addr.sockaddr = c->proxy_protocol->src_sockaddr; > + addr.socklen = c->proxy_protocol->src_socklen; > + > + return ngx_http_realip_set_addr(r, &addr); > > default: /* NGX_HTTP_REALIP_HEADER */ > > @@ -225,8 +231,6 @@ ngx_http_realip_handler(ngx_http_request > > found: > > - c = r->connection; > - > addr.sockaddr = c->sockaddr; > addr.socklen = c->socklen; > /* addr.name = c->addr_text; */ > @@ -235,10 +239,6 @@ found: > rlcf->recursive) > != NGX_DECLINED) > { > - if (rlcf->type == NGX_HTTP_REALIP_PROXY) { > - ngx_inet_set_port(addr.sockaddr, c->proxy_protocol->src_port); > - } > - > return ngx_http_realip_set_addr(r, &addr); > } > > diff --git a/src/http/ngx_http_variables.c b/src/http/ngx_http_variables.c > --- a/src/http/ngx_http_variables.c > +++ b/src/http/ngx_http_variables.c > @@ -201,20 +201,16 @@ static ngx_http_variable_t ngx_http_cor > { ngx_string("remote_port"), NULL, ngx_http_variable_remote_port, 0, 0, 0 }, > > { ngx_string("proxy_protocol_addr"), NULL, > - ngx_http_variable_proxy_protocol_addr, > - offsetof(ngx_proxy_protocol_t, src_addr), 0, 0 }, > + ngx_http_variable_proxy_protocol_addr, 0, 0, 0 }, > > { ngx_string("proxy_protocol_port"), NULL, > - ngx_http_variable_proxy_protocol_port, > - offsetof(ngx_proxy_protocol_t, src_port), 0, 0 }, > + ngx_http_variable_proxy_protocol_port, 0, 0, 0 }, > > { ngx_string("proxy_protocol_server_addr"), NULL, > - ngx_http_variable_proxy_protocol_addr, > - offsetof(ngx_proxy_protocol_t, dst_addr), 0, 0 }, > + ngx_http_variable_proxy_protocol_addr, 1, 0, 0 }, > > { ngx_string("proxy_protocol_server_port"), NULL, > - ngx_http_variable_proxy_protocol_port, > - offsetof(ngx_proxy_protocol_t, dst_port), 0, 0 }, > + ngx_http_variable_proxy_protocol_port, 1, 0, 0 }, > > { ngx_string("proxy_protocol_tlv_"), NULL, > ngx_http_variable_proxy_protocol_tlv, > @@ -1340,7 +1336,8 @@ static ngx_int_t > ngx_http_variable_proxy_protocol_addr(ngx_http_request_t *r, > ngx_http_variable_value_t *v, uintptr_t data) > { > - ngx_str_t *addr; > + u_char *p; > + size_t n; > ngx_proxy_protocol_t *pp; > > pp = r->connection->proxy_protocol; > @@ -1349,13 +1346,25 @@ ngx_http_variable_proxy_protocol_addr(ng > return NGX_OK; > } > > - addr = (ngx_str_t *) ((char *) pp + data); > - > - v->len = addr->len; > + p = ngx_pnalloc(r->connection->pool, NGX_SOCKADDR_STRLEN); NGX_SOCKADDR_STRLEN includes UNIX-domain sockets as well, which means ~100 bytes memory over-allocation, with such addresses prohibited in the PROXY protocol. It can be saved if allocate the actual size returned from ngx_sock_ntop(). > + if (p == NULL) { > + return NGX_ERROR; > + } > + > + if (data) { > + n = ngx_sock_ntop(pp->dst_sockaddr, pp->dst_socklen, > + p, NGX_SOCKADDR_STRLEN, 0); > + > + } else { > + n = ngx_sock_ntop(pp->src_sockaddr, pp->src_socklen, > + p, NGX_SOCKADDR_STRLEN, 0); > + } > + > + v->len = n; > v->valid = 1; > v->no_cacheable = 0; > v->not_found = 0; > - v->data = addr->data; > + v->data = p; > > return NGX_OK; > } > @@ -1384,7 +1393,12 @@ ngx_http_variable_proxy_protocol_port(ng > return NGX_ERROR; > } > > - port = *(in_port_t *) ((char *) pp + data); > + if (data) { > + port = ngx_inet_get_port(pp->dst_sockaddr); > + > + } else { > + port = ngx_inet_get_port(pp->src_sockaddr); > + } > > if (port > 0 && port < 65536) { > v->len = ngx_sprintf(v->data, "%ui", port) - v->data; What I don't get is why we parse zero port as valid (see ngx_atoi() call in ngx_proxy_protocol_read_port()) but reject evaluating it as such in variable handlers. IMHO, it makes sense to remove this restriction and write the port value unconditionally, including "0", especially that after storing port as binary it cannot be invalid. > diff --git a/src/mail/ngx_mail_auth_http_module.c b/src/mail/ngx_mail_auth_http_module.c > --- a/src/mail/ngx_mail_auth_http_module.c > +++ b/src/mail/ngx_mail_auth_http_module.c > @@ -1249,11 +1249,11 @@ ngx_mail_auth_http_create_request(ngx_ma > > if (c->proxy_protocol) { > len += sizeof("Proxy-Protocol-Addr: ") - 1 > - + c->proxy_protocol->src_addr.len + sizeof(CRLF) - 1 > + + NGX_SOCKADDR_STRLEN + sizeof(CRLF) - 1 See above about wasted memory. > + sizeof("Proxy-Protocol-Port: ") - 1 > + sizeof("65535") - 1 + sizeof(CRLF) - 1 > + sizeof("Proxy-Protocol-Server-Addr: ") - 1 > - + c->proxy_protocol->dst_addr.len + sizeof(CRLF) - 1 > + + NGX_SOCKADDR_STRLEN + sizeof(CRLF) - 1 > + sizeof("Proxy-Protocol-Server-Port: ") - 1 > + sizeof("65535") - 1 + sizeof(CRLF) - 1; > } > @@ -1352,21 +1352,23 @@ ngx_mail_auth_http_create_request(ngx_ma > if (c->proxy_protocol) { > b->last = ngx_cpymem(b->last, "Proxy-Protocol-Addr: ", > sizeof("Proxy-Protocol-Addr: ") - 1); > - b->last = ngx_copy(b->last, c->proxy_protocol->src_addr.data, > - c->proxy_protocol->src_addr.len); > + b->last += ngx_sock_ntop(c->proxy_protocol->src_sockaddr, > + c->proxy_protocol->src_socklen, > + b->last, b->end - b->last, 0); > *b->last++ = CR; *b->last++ = LF; > > b->last = ngx_sprintf(b->last, "Proxy-Protocol-Port: %d" CRLF, > - c->proxy_protocol->src_port); > + (int) ngx_inet_get_port(c->proxy_protocol->src_sockaddr)); > > b->last = ngx_cpymem(b->last, "Proxy-Protocol-Server-Addr: ", > sizeof("Proxy-Protocol-Server-Addr: ") - 1); > - b->last = ngx_copy(b->last, c->proxy_protocol->dst_addr.data, > - c->proxy_protocol->dst_addr.len); > + b->last += ngx_sock_ntop(c->proxy_protocol->dst_sockaddr, > + c->proxy_protocol->dst_socklen, > + b->last, b->end - b->last, 0); > *b->last++ = CR; *b->last++ = LF; > > b->last = ngx_sprintf(b->last, "Proxy-Protocol-Server-Port: %d" CRLF, > - c->proxy_protocol->dst_port); > + (int) ngx_inet_get_port(c->proxy_protocol->dst_sockaddr)); > } > > if (s->auth_method == NGX_MAIL_AUTH_NONE) { > diff --git a/src/mail/ngx_mail_realip_module.c b/src/mail/ngx_mail_realip_module.c > --- a/src/mail/ngx_mail_realip_module.c > +++ b/src/mail/ngx_mail_realip_module.c > @@ -87,14 +87,8 @@ ngx_mail_realip_handler(ngx_mail_session > return NGX_OK; > } > > - if (ngx_parse_addr(c->pool, &addr, c->proxy_protocol->src_addr.data, > - c->proxy_protocol->src_addr.len) > - != NGX_OK) > - { > - return NGX_OK; > - } > - > - ngx_inet_set_port(addr.sockaddr, c->proxy_protocol->src_port); > + addr.sockaddr = c->proxy_protocol->src_sockaddr; > + addr.socklen = c->proxy_protocol->src_socklen; > > return ngx_mail_realip_set_addr(s, &addr); > } > diff --git a/src/stream/ngx_stream_realip_module.c b/src/stream/ngx_stream_realip_module.c > --- a/src/stream/ngx_stream_realip_module.c > +++ b/src/stream/ngx_stream_realip_module.c > @@ -116,14 +116,8 @@ ngx_stream_realip_handler(ngx_stream_ses > return NGX_DECLINED; > } > > - if (ngx_parse_addr(c->pool, &addr, c->proxy_protocol->src_addr.data, > - c->proxy_protocol->src_addr.len) > - != NGX_OK) > - { > - return NGX_DECLINED; > - } > - > - ngx_inet_set_port(addr.sockaddr, c->proxy_protocol->src_port); > + addr.sockaddr = c->proxy_protocol->src_sockaddr; > + addr.socklen = c->proxy_protocol->src_socklen; > > return ngx_stream_realip_set_addr(s, &addr); > } > diff --git a/src/stream/ngx_stream_variables.c b/src/stream/ngx_stream_variables.c > --- a/src/stream/ngx_stream_variables.c > +++ b/src/stream/ngx_stream_variables.c > @@ -66,20 +66,16 @@ static ngx_stream_variable_t ngx_stream > ngx_stream_variable_remote_port, 0, 0, 0 }, > > { ngx_string("proxy_protocol_addr"), NULL, > - ngx_stream_variable_proxy_protocol_addr, > - offsetof(ngx_proxy_protocol_t, src_addr), 0, 0 }, > + ngx_stream_variable_proxy_protocol_addr, 0, 0, 0 }, > > { ngx_string("proxy_protocol_port"), NULL, > - ngx_stream_variable_proxy_protocol_port, > - offsetof(ngx_proxy_protocol_t, src_port), 0, 0 }, > + ngx_stream_variable_proxy_protocol_port, 0, 0, 0 }, > > { ngx_string("proxy_protocol_server_addr"), NULL, > - ngx_stream_variable_proxy_protocol_addr, > - offsetof(ngx_proxy_protocol_t, dst_addr), 0, 0 }, > + ngx_stream_variable_proxy_protocol_addr, 1, 0, 0 }, > > { ngx_string("proxy_protocol_server_port"), NULL, > - ngx_stream_variable_proxy_protocol_port, > - offsetof(ngx_proxy_protocol_t, dst_port), 0, 0 }, > + ngx_stream_variable_proxy_protocol_port, 1, 0, 0 }, > > { ngx_string("proxy_protocol_tlv_"), NULL, > ngx_stream_variable_proxy_protocol_tlv, > @@ -573,7 +569,8 @@ static ngx_int_t > ngx_stream_variable_proxy_protocol_addr(ngx_stream_session_t *s, > ngx_stream_variable_value_t *v, uintptr_t data) > { > - ngx_str_t *addr; > + u_char *p; > + size_t n; > ngx_proxy_protocol_t *pp; > > pp = s->connection->proxy_protocol; > @@ -582,13 +579,25 @@ ngx_stream_variable_proxy_protocol_addr( > return NGX_OK; > } > > - addr = (ngx_str_t *) ((char *) pp + data); > + p = ngx_pnalloc(s->connection->pool, NGX_SOCKADDR_STRLEN); > + if (p == NULL) { > + return NGX_ERROR; > + } > > - v->len = addr->len; > + if (data) { > + n = ngx_sock_ntop(pp->dst_sockaddr, pp->dst_socklen, > + p, NGX_SOCKADDR_STRLEN, 0); > + > + } else { > + n = ngx_sock_ntop(pp->src_sockaddr, pp->src_socklen, > + p, NGX_SOCKADDR_STRLEN, 0); > + } > + > + v->len = n; > v->valid = 1; > v->no_cacheable = 0; > v->not_found = 0; > - v->data = addr->data; > + v->data = p; > > return NGX_OK; > } > @@ -617,7 +626,12 @@ ngx_stream_variable_proxy_protocol_port( > return NGX_ERROR; > } > > - port = *(in_port_t *) ((char *) pp + data); > + if (data) { > + port = ngx_inet_get_port(pp->dst_sockaddr); > + > + } else { > + port = ngx_inet_get_port(pp->src_sockaddr); > + } > > if (port > 0 && port < 65536) { > v->len = ngx_sprintf(v->data, "%ui", port) - v->data; > # HG changeset patch > # User Roman Arutyunyan > # Date 1710081227 -14400 > # Sun Mar 10 18:33:47 2024 +0400 > # Node ID 6957c271a15c789be51d042de6835ce02141fe58 > # Parent 07c8c445746c6c74250bad7b42cd26dbb0ed6dd3 > Pass PROXY protocol header from client to upstream (ticket #2010). > > When creating a PROXY protocol header in mail or stream, server address is > taken from c->local_sockaddr, which is normally the real server address of the > connection. If prior to that client address was substituted by the realip > module, then client and server addresses can have different address families, > which breaks PROXY protocol syntax. The change fixes this by using server > address from client PROXY protocol header as well. > > diff --git a/src/core/ngx_proxy_protocol.c b/src/core/ngx_proxy_protocol.c > --- a/src/core/ngx_proxy_protocol.c > +++ b/src/core/ngx_proxy_protocol.c > @@ -375,11 +375,24 @@ ngx_proxy_protocol_write(ngx_connection_ > > *buf++ = ' '; > The above ngx_connection_local_sockaddr() call is not needed now if server address is obtained from the saved PROXY header, it can be put under conditional. > - buf += ngx_sock_ntop(c->local_sockaddr, c->local_socklen, buf, last - buf, > - 0); > + if (c->proxy_protocol > + && ngx_cmp_sockaddr(c->proxy_protocol->src_sockaddr, > + c->proxy_protocol->src_socklen, > + c->sockaddr, c->socklen, 1) > + == NGX_OK) This relies on a weak assumption that matching client addresses is the result of client address substitution by the realip module. This allows to substitute server address in PROXY writer to a client chosen value regardless of address replacement being allowed with set_real_ip_from. (That's why an explicit hint from realip might be beneficial.) > + { > + buf += ngx_sock_ntop(c->proxy_protocol->dst_sockaddr, > + c->proxy_protocol->dst_socklen, > + buf, last - buf, 0); I'd add a blank line > + lport = ngx_inet_get_port(c->proxy_protocol->dst_sockaddr); > + > + } else { > + buf += ngx_sock_ntop(c->local_sockaddr, c->local_socklen, > + buf, last - buf, 0); As ngx_connection_local_sockaddr() calls ngx_sock_ntop() internally, the result could be re-used here, to get rid of the 2nd explicit ngx_sock_ntop() call (with all its implied overhead). > + lport = ngx_inet_get_port(c->local_sockaddr); > + } > > port = ngx_inet_get_port(c->sockaddr); > - lport = ngx_inet_get_port(c->local_sockaddr); > > return ngx_slprintf(buf, last, " %ui %ui" CRLF, port, lport); > } From arut at nginx.com Thu Mar 14 15:55:15 2024 From: arut at nginx.com (Roman Arutyunyan) Date: Thu, 14 Mar 2024 19:55:15 +0400 Subject: [PATCH] Geo: fix uninitialized memory access In-Reply-To: References: Message-ID: <20240314155515.f3c6bttkxzoo2unz@N00W24XTQX> Hi Piotr, On Wed, Feb 28, 2024 at 01:21:40AM +0000, Piotr Sikora via nginx-devel wrote: > # HG changeset patch > # User Piotr Sikora > # Date 1708977621 0 > # Mon Feb 26 20:00:21 2024 +0000 > # Branch patch005 > # Node ID fe6f8a72d42970df176ea53f4f0aea16947ba5b8 > # Parent 52936793ac076072c3544aa4e27f973d2f8fecda > Geo: fix uninitialized memory access. > > Found with MemorySanitizer. > > Signed-off-by: Piotr Sikora > > diff -r 52936793ac07 -r fe6f8a72d429 src/http/modules/ngx_http_geo_module.c > --- a/src/http/modules/ngx_http_geo_module.c Mon Feb 26 20:00:19 2024 +0000 > +++ b/src/http/modules/ngx_http_geo_module.c Mon Feb 26 20:00:21 2024 +0000 > @@ -1259,7 +1259,7 @@ > return gvvn->value; > } > > - val = ngx_palloc(ctx->pool, sizeof(ngx_http_variable_value_t)); > + val = ngx_pcalloc(ctx->pool, sizeof(ngx_http_variable_value_t)); > if (val == NULL) { > return NULL; > } > diff -r 52936793ac07 -r fe6f8a72d429 src/stream/ngx_stream_geo_module.c > --- a/src/stream/ngx_stream_geo_module.c Mon Feb 26 20:00:19 2024 +0000 > +++ b/src/stream/ngx_stream_geo_module.c Mon Feb 26 20:00:21 2024 +0000 > @@ -1209,7 +1209,7 @@ > return gvvn->value; > } > > - val = ngx_palloc(ctx->pool, sizeof(ngx_stream_variable_value_t)); > + val = ngx_pcalloc(ctx->pool, sizeof(ngx_stream_variable_value_t)); > if (val == NULL) { > return NULL; > } > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > https://mailman.nginx.org/mailman/listinfo/nginx-devel Thanks for the patch, looks valid, except we no longer need to explicitly initialize fields to zero. Also, I think we need more details about the uninitialized memory access. See updated patch. -- Roman Arutyunyan -------------- next part -------------- # HG changeset patch # User Piotr Sikora # Date 1710427040 -14400 # Thu Mar 14 18:37:20 2024 +0400 # Node ID bd1a4807521bd830ab73c11d6ff3c9b75f5c45f0 # Parent 2ed3f57dca0a664340bca2236c7d614902db4180 Geo: fix uninitialized memory access. While copying ngx_http_variable_value_t structures to geo binary base in ngx_http_geo_copy_values(), uninitialized parts of these structures are copied as well. These include the "escape" field and possible holes. Calculating crc32 of this data triggers uninitialized memory access. Found with MemorySanitizer. Signed-off-by: Piotr Sikora diff --git a/src/http/modules/ngx_http_geo_module.c b/src/http/modules/ngx_http_geo_module.c --- a/src/http/modules/ngx_http_geo_module.c +++ b/src/http/modules/ngx_http_geo_module.c @@ -1259,7 +1259,7 @@ ngx_http_geo_value(ngx_conf_t *cf, ngx_h return gvvn->value; } - val = ngx_palloc(ctx->pool, sizeof(ngx_http_variable_value_t)); + val = ngx_pcalloc(ctx->pool, sizeof(ngx_http_variable_value_t)); if (val == NULL) { return NULL; } @@ -1271,8 +1271,6 @@ ngx_http_geo_value(ngx_conf_t *cf, ngx_h } val->valid = 1; - val->no_cacheable = 0; - val->not_found = 0; gvvn = ngx_palloc(ctx->temp_pool, sizeof(ngx_http_geo_variable_value_node_t)); diff --git a/src/stream/ngx_stream_geo_module.c b/src/stream/ngx_stream_geo_module.c --- a/src/stream/ngx_stream_geo_module.c +++ b/src/stream/ngx_stream_geo_module.c @@ -1209,7 +1209,7 @@ ngx_stream_geo_value(ngx_conf_t *cf, ngx return gvvn->value; } - val = ngx_palloc(ctx->pool, sizeof(ngx_stream_variable_value_t)); + val = ngx_pcalloc(ctx->pool, sizeof(ngx_stream_variable_value_t)); if (val == NULL) { return NULL; } @@ -1221,8 +1221,6 @@ ngx_stream_geo_value(ngx_conf_t *cf, ngx } val->valid = 1; - val->no_cacheable = 0; - val->not_found = 0; gvvn = ngx_palloc(ctx->temp_pool, sizeof(ngx_stream_geo_variable_value_node_t)); From pluknet at nginx.com Fri Mar 15 16:12:50 2024 From: pluknet at nginx.com (Sergey Kandaurov) Date: Fri, 15 Mar 2024 20:12:50 +0400 Subject: [PATCH] Core: fix conversion of IPv4-mapped IPv6 addresses In-Reply-To: <5584232259d28489efba.1709083309@ip-172-31-36-66.ec2.internal> References: <5584232259d28489efba.1709083309@ip-172-31-36-66.ec2.internal> Message-ID: > On 28 Feb 2024, at 05:21, Piotr Sikora via nginx-devel wrote: > > # HG changeset patch > # User Piotr Sikora > # Date 1708977626 0 > # Mon Feb 26 20:00:26 2024 +0000 > # Branch patch007 > # Node ID 5584232259d28489efba149f2f5ae730691ff0d4 > # Parent 03e5549976765912818120e11f6b08410a2af6a9 > Core: fix conversion of IPv4-mapped IPv6 addresses. > > Found with UndefinedBehaviorSanitizer (shift). Hello, thanks for the patch. The "shift" remark doesn't describe a problem in details. What actually happens is: - loading from "u_char", with integer promotion - left shift by 24 to most significant bit 31 The latter cannot be represented in (signed) integer in popular data type models: bit 31 gets to the integer type sign bit, and is caught by UndefinedBehaviorSanitizer, e.g.: | runtime error: left shift of 255 by 24 places cannot be represented in type 'int' The lvalue typically has an unsigned integer type of the same size (POSIX states in_addr_t is equivalent to uint32_t, which is of the same type as "unsigned int" in ILP32 and LP64). Known compilers, such as GCC and Clang, represent intermediate result used as a source for left shift in a wide register such that it doesn't get truncated when storing it in the left value. This makes objects don't change after adding an explicit cast, comparing md5sum over them matches. Notably, this differs from left-shifting signed integer into the sign bit fixed in ad736705a744, where, as described in the change, the left value is of the larger type width. Still, I won't object to apply something similar to ad736705a744 in order to suppress such UndefinedBehaviorSanitizer reports. They look valid, although seemingly innocent. > > Signed-off-by: Piotr Sikora > > diff -r 03e554997676 -r 5584232259d2 src/core/ngx_inet.c > --- a/src/core/ngx_inet.c Mon Feb 26 20:00:23 2024 +0000 > +++ b/src/core/ngx_inet.c Mon Feb 26 20:00:26 2024 +0000 > @@ -507,10 +507,10 @@ > > p = inaddr6->s6_addr; > > - inaddr = p[12] << 24; > - inaddr += p[13] << 16; > - inaddr += p[14] << 8; > - inaddr += p[15]; > + inaddr = (in_addr_t) p[12] << 24; > + inaddr += (in_addr_t) p[13] << 16; > + inaddr += (in_addr_t) p[14] << 8; > + inaddr += (in_addr_t) p[15]; > > inaddr = htonl(inaddr); > } > diff -r 03e554997676 -r 5584232259d2 src/http/modules/ngx_http_access_module.c > --- a/src/http/modules/ngx_http_access_module.c Mon Feb 26 20:00:23 2024 +0000 > +++ b/src/http/modules/ngx_http_access_module.c Mon Feb 26 20:00:26 2024 +0000 > @@ -148,10 +148,10 @@ > p = sin6->sin6_addr.s6_addr; > > if (alcf->rules && IN6_IS_ADDR_V4MAPPED(&sin6->sin6_addr)) { > - addr = p[12] << 24; > - addr += p[13] << 16; > - addr += p[14] << 8; > - addr += p[15]; > + addr = (in_addr_t) p[12] << 24; > + addr += (in_addr_t) p[13] << 16; > + addr += (in_addr_t) p[14] << 8; > + addr += (in_addr_t) p[15]; > return ngx_http_access_inet(r, alcf, htonl(addr)); > } > > diff -r 03e554997676 -r 5584232259d2 src/http/modules/ngx_http_geo_module.c > --- a/src/http/modules/ngx_http_geo_module.c Mon Feb 26 20:00:23 2024 +0000 > +++ b/src/http/modules/ngx_http_geo_module.c Mon Feb 26 20:00:26 2024 +0000 > @@ -199,10 +199,10 @@ > p = inaddr6->s6_addr; > > if (IN6_IS_ADDR_V4MAPPED(inaddr6)) { > - inaddr = p[12] << 24; > - inaddr += p[13] << 16; > - inaddr += p[14] << 8; > - inaddr += p[15]; > + inaddr = (in_addr_t) p[12] << 24; > + inaddr += (in_addr_t) p[13] << 16; > + inaddr += (in_addr_t) p[14] << 8; > + inaddr += (in_addr_t) p[15]; > > vv = (ngx_http_variable_value_t *) > ngx_radix32tree_find(ctx->u.trees.tree, inaddr); > @@ -272,10 +272,10 @@ > if (IN6_IS_ADDR_V4MAPPED(inaddr6)) { > p = inaddr6->s6_addr; > > - inaddr = p[12] << 24; > - inaddr += p[13] << 16; > - inaddr += p[14] << 8; > - inaddr += p[15]; > + inaddr = (in_addr_t) p[12] << 24; > + inaddr += (in_addr_t) p[13] << 16; > + inaddr += (in_addr_t) p[14] << 8; > + inaddr += (in_addr_t) p[15]; > > } else { > inaddr = INADDR_NONE; > diff -r 03e554997676 -r 5584232259d2 src/http/modules/ngx_http_geoip_module.c > --- a/src/http/modules/ngx_http_geoip_module.c Mon Feb 26 20:00:23 2024 +0000 > +++ b/src/http/modules/ngx_http_geoip_module.c Mon Feb 26 20:00:26 2024 +0000 > @@ -266,10 +266,10 @@ > if (IN6_IS_ADDR_V4MAPPED(inaddr6)) { > p = inaddr6->s6_addr; > > - inaddr = p[12] << 24; > - inaddr += p[13] << 16; > - inaddr += p[14] << 8; > - inaddr += p[15]; > + inaddr = (in_addr_t) p[12] << 24; > + inaddr += (in_addr_t) p[13] << 16; > + inaddr += (in_addr_t) p[14] << 8; > + inaddr += (in_addr_t) p[15]; > > return inaddr; > } > diff -r 03e554997676 -r 5584232259d2 src/stream/ngx_stream_access_module.c > --- a/src/stream/ngx_stream_access_module.c Mon Feb 26 20:00:23 2024 +0000 > +++ b/src/stream/ngx_stream_access_module.c Mon Feb 26 20:00:26 2024 +0000 > @@ -144,10 +144,10 @@ > p = sin6->sin6_addr.s6_addr; > > if (ascf->rules && IN6_IS_ADDR_V4MAPPED(&sin6->sin6_addr)) { > - addr = p[12] << 24; > - addr += p[13] << 16; > - addr += p[14] << 8; > - addr += p[15]; > + addr = (in_addr_t) p[12] << 24; > + addr += (in_addr_t) p[13] << 16; > + addr += (in_addr_t) p[14] << 8; > + addr += (in_addr_t) p[15]; > return ngx_stream_access_inet(s, ascf, htonl(addr)); > } > > diff -r 03e554997676 -r 5584232259d2 src/stream/ngx_stream_geo_module.c > --- a/src/stream/ngx_stream_geo_module.c Mon Feb 26 20:00:23 2024 +0000 > +++ b/src/stream/ngx_stream_geo_module.c Mon Feb 26 20:00:26 2024 +0000 > @@ -190,10 +190,10 @@ > p = inaddr6->s6_addr; > > if (IN6_IS_ADDR_V4MAPPED(inaddr6)) { > - inaddr = p[12] << 24; > - inaddr += p[13] << 16; > - inaddr += p[14] << 8; > - inaddr += p[15]; > + inaddr = (in_addr_t) p[12] << 24; > + inaddr += (in_addr_t) p[13] << 16; > + inaddr += (in_addr_t) p[14] << 8; > + inaddr += (in_addr_t) p[15]; > > vv = (ngx_stream_variable_value_t *) > ngx_radix32tree_find(ctx->u.trees.tree, inaddr); > @@ -263,10 +263,10 @@ > if (IN6_IS_ADDR_V4MAPPED(inaddr6)) { > p = inaddr6->s6_addr; > > - inaddr = p[12] << 24; > - inaddr += p[13] << 16; > - inaddr += p[14] << 8; > - inaddr += p[15]; > + inaddr = (in_addr_t) p[12] << 24; > + inaddr += (in_addr_t) p[13] << 16; > + inaddr += (in_addr_t) p[14] << 8; > + inaddr += (in_addr_t) p[15]; > > } else { > inaddr = INADDR_NONE; > diff -r 03e554997676 -r 5584232259d2 src/stream/ngx_stream_geoip_module.c > --- a/src/stream/ngx_stream_geoip_module.c Mon Feb 26 20:00:23 2024 +0000 > +++ b/src/stream/ngx_stream_geoip_module.c Mon Feb 26 20:00:26 2024 +0000 > @@ -236,10 +236,10 @@ > if (IN6_IS_ADDR_V4MAPPED(inaddr6)) { > p = inaddr6->s6_addr; > > - inaddr = p[12] << 24; > - inaddr += p[13] << 16; > - inaddr += p[14] << 8; > - inaddr += p[15]; > + inaddr = (in_addr_t) p[12] << 24; > + inaddr += (in_addr_t) p[13] << 16; > + inaddr += (in_addr_t) p[14] << 8; > + inaddr += (in_addr_t) p[15]; > > return inaddr; > } > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > https://mailman.nginx.org/mailman/listinfo/nginx-devel -- Sergey Kandaurov From pluknet at nginx.com Fri Mar 15 16:18:56 2024 From: pluknet at nginx.com (Sergey Kandaurov) Date: Fri, 15 Mar 2024 20:18:56 +0400 Subject: [PATCH] Core: free connections and read/write events at shutdown In-Reply-To: References: <59780B99-BFAD-4DD6-8887-3C87DDC53AC6@nginx.com> Message-ID: > On 8 Mar 2024, at 19:45, Piotr Sikora via nginx-devel wrote: > > Hi Sergey, > >> Since this is not a real memory leak (the allocations made in the >> init_process method of the ngx_event_core_module module are used up >> to this function and already freed on exit automatically, as this >> function never returns), I don't think there is something to fix. > > Agreed, this is not a "real" leak. > > But fixing those "false" leaks allows you to run test with sanitizers > on the CI, which can protect you and/or external contributors from > introducing new bugs. While I agree that false positives do not allow to run LeakSanitizer in a clean fashion, I don't think it is nginx which should be fixed. Rather, sanitizer analysis could be improved instead to prevent FPs. Meanwhile, leak sanitizer can be used with suppressions as appropriate to run tests cleanly. For example, this allows to suppress memory leak reports for allocations made during worker process init, such as cycle connections and read/write events: $ cat suppr.txt leak:ngx_worker_process_init $ LSAN_OPTIONS=suppressions=suppr.txt prove -r > >> (Further, the patch misses cycle->files for NGX_USE_FD_EVENT event >> methods. > > Good catch, thanks! > >> Also, it's probably better to free the memory in the >> exit_process method to obey the ownership (this would also fix >> missing ngx_free() calls on win32), but this would require moving >> code that uses these connections afterwards to catch socket leaks.) > > Freeing memory in exit_process could result in use-after-free, since > cleanups for the cycle->pool might still access those connections. > Yep, that's why it is freed afterwards. -- Sergey Kandaurov From arut at nginx.com Mon Mar 18 12:13:00 2024 From: arut at nginx.com (Roman Arutyunyan) Date: Mon, 18 Mar 2024 16:13:00 +0400 Subject: [PATCH] Core: fix build without libcrypt In-Reply-To: References: Message-ID: <20240318121300.522yn5bbwnd2ext4@N00W24XTQX> Hi Piotr, On Wed, Feb 28, 2024 at 01:23:08AM +0000, Piotr Sikora via nginx-devel wrote: > # HG changeset patch > # User Piotr Sikora > # Date 1708977637 0 > # Mon Feb 26 20:00:37 2024 +0000 > # Branch patch013 > # Node ID cdc173477ea99fd6c952a85e5cd11db66452076a > # Parent 04e3155b3b9651fee708898aaf82ac35532806ee > Core: fix build without libcrypt. > > libcrypt is no longer part of glibc, so it might not be available. Can you provide an example of a system where this fix is needed? > Signed-off-by: Piotr Sikora > > diff -r 04e3155b3b96 -r cdc173477ea9 auto/unix > --- a/auto/unix Mon Feb 26 20:00:35 2024 +0000 > +++ b/auto/unix Mon Feb 26 20:00:37 2024 +0000 > @@ -150,7 +150,7 @@ > > > ngx_feature="crypt()" > -ngx_feature_name= > +ngx_feature_name="NGX_HAVE_CRYPT" > ngx_feature_run=no > ngx_feature_incs= > ngx_feature_path= > @@ -162,7 +162,7 @@ > if [ $ngx_found = no ]; then > > ngx_feature="crypt() in libcrypt" > - ngx_feature_name= > + ngx_feature_name="NGX_HAVE_CRYPT" > ngx_feature_run=no > ngx_feature_incs= > ngx_feature_path= > diff -r 04e3155b3b96 -r cdc173477ea9 src/os/unix/ngx_linux_config.h > --- a/src/os/unix/ngx_linux_config.h Mon Feb 26 20:00:35 2024 +0000 > +++ b/src/os/unix/ngx_linux_config.h Mon Feb 26 20:00:37 2024 +0000 > @@ -52,7 +52,6 @@ > #include /* memalign() */ > #include /* IOV_MAX */ > #include > -#include > #include /* uname() */ > > #include > @@ -61,6 +60,11 @@ > #include > > > +#if (NGX_HAVE_CRYPT_H) > +#include > +#endif > + > + > #if (NGX_HAVE_POSIX_SEM) > #include > #endif > diff -r 04e3155b3b96 -r cdc173477ea9 src/os/unix/ngx_user.c > --- a/src/os/unix/ngx_user.c Mon Feb 26 20:00:35 2024 +0000 > +++ b/src/os/unix/ngx_user.c Mon Feb 26 20:00:37 2024 +0000 > @@ -44,7 +44,7 @@ > return NGX_ERROR; > } > > -#else > +#elif (NGX_HAVE_CRYPT) > > ngx_int_t > ngx_libc_crypt(ngx_pool_t *pool, u_char *key, u_char *salt, u_char **encrypted) > @@ -76,6 +76,14 @@ > return NGX_ERROR; > } > > +#else > + > +ngx_int_t > +ngx_libc_crypt(ngx_pool_t *pool, u_char *key, u_char *salt, u_char **encrypted) > +{ > + return NGX_ERROR; > +} > + > #endif > > #endif /* NGX_CRYPT */ > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > https://mailman.nginx.org/mailman/listinfo/nginx-devel -- Roman Arutyunyan From pluknet at nginx.com Mon Mar 18 13:16:56 2024 From: pluknet at nginx.com (Sergey Kandaurov) Date: Mon, 18 Mar 2024 17:16:56 +0400 Subject: [PATCH] Core: fix conversion of IPv4-mapped IPv6 addresses In-Reply-To: References: <5584232259d28489efba.1709083309@ip-172-31-36-66.ec2.internal> Message-ID: On Fri, Mar 15, 2024 at 08:12:50PM +0400, Sergey Kandaurov wrote: > > > On 28 Feb 2024, at 05:21, Piotr Sikora via nginx-devel wrote: > > > > # HG changeset patch > > # User Piotr Sikora > > # Date 1708977626 0 > > # Mon Feb 26 20:00:26 2024 +0000 > > # Branch patch007 > > # Node ID 5584232259d28489efba149f2f5ae730691ff0d4 > > # Parent 03e5549976765912818120e11f6b08410a2af6a9 > > Core: fix conversion of IPv4-mapped IPv6 addresses. > > > > Found with UndefinedBehaviorSanitizer (shift). > > Hello, thanks for the patch. > > The "shift" remark doesn't describe a problem in details. > > What actually happens is: > - loading from "u_char", with integer promotion > - left shift by 24 to most significant bit 31 > > The latter cannot be represented in (signed) integer in popular > data type models: bit 31 gets to the integer type sign bit, > and is caught by UndefinedBehaviorSanitizer, e.g.: > | runtime error: left shift of 255 by 24 places cannot be represented in type 'int' > > The lvalue typically has an unsigned integer type of the same > size (POSIX states in_addr_t is equivalent to uint32_t, which > is of the same type as "unsigned int" in ILP32 and LP64). > Known compilers, such as GCC and Clang, represent intermediate > result used as a source for left shift in a wide register such that > it doesn't get truncated when storing it in the left value. > This makes objects don't change after adding an explicit cast, > comparing md5sum over them matches. > > Notably, this differs from left-shifting signed integer > into the sign bit fixed in ad736705a744, where, as described > in the change, the left value is of the larger type width. > > Still, I won't object to apply something similar to ad736705a744 > in order to suppress such UndefinedBehaviorSanitizer reports. > They look valid, although seemingly innocent. > > > > > Signed-off-by: Piotr Sikora > > > > diff -r 03e554997676 -r 5584232259d2 src/core/ngx_inet.c > > --- a/src/core/ngx_inet.c Mon Feb 26 20:00:23 2024 +0000 > > +++ b/src/core/ngx_inet.c Mon Feb 26 20:00:26 2024 +0000 > > @@ -507,10 +507,10 @@ > > > > p = inaddr6->s6_addr; > > > > - inaddr = p[12] << 24; > > - inaddr += p[13] << 16; > > - inaddr += p[14] << 8; > > - inaddr += p[15]; > > + inaddr = (in_addr_t) p[12] << 24; > > + inaddr += (in_addr_t) p[13] << 16; > > + inaddr += (in_addr_t) p[14] << 8; > > + inaddr += (in_addr_t) p[15]; > > > > [...] The patch. # HG changeset patch # User Sergey Kandaurov # Date 1710767670 -14400 # Mon Mar 18 17:14:30 2024 +0400 # Node ID 543ca08e56fbd28f82552564d70edb74d0e628a7 # Parent 89bff782528a91ad123b63b624f798e6fd9c8e68 Core: fixed undefined behaviour with IPv4-mapped IPv6 addresses. When converting to IPv4 addresses, with implicit integer promotion to hold the intermediate value, the most significant bit appeared on the sign bit after left shifting by 24 places, which is undefined behaviour. In practice, a subsequent store into the left value resulted in the same behaviour as if there would be a proper cast, at least on known compilers, such as GCC and Clang, because in_addr_t has the same type width as the intermediate after integer promotion (POSIX states in_addr_t is equivalent to uint32_t, which is the same type as "unsigned int" in ILP32 and LP64 data type models). The reason is that they keep intermediate result in a wide enough register such that its value is not truncated before storing it in the left value. This makes to produce object file that don't change after adding an explicit cast; comparing md5sum over them matches. Still, it is useful to avoid undefined behaviour and to suppress such reports. Found with UndefinedBehaviorSanitizer. Spotted by: Piotr Sikora diff --git a/src/core/ngx_inet.c b/src/core/ngx_inet.c --- a/src/core/ngx_inet.c +++ b/src/core/ngx_inet.c @@ -507,7 +507,7 @@ ngx_cidr_match(struct sockaddr *sa, ngx_ p = inaddr6->s6_addr; - inaddr = p[12] << 24; + inaddr = (in_addr_t) p[12] << 24; inaddr += p[13] << 16; inaddr += p[14] << 8; inaddr += p[15]; diff --git a/src/http/modules/ngx_http_access_module.c b/src/http/modules/ngx_http_access_module.c --- a/src/http/modules/ngx_http_access_module.c +++ b/src/http/modules/ngx_http_access_module.c @@ -148,7 +148,7 @@ ngx_http_access_handler(ngx_http_request p = sin6->sin6_addr.s6_addr; if (alcf->rules && IN6_IS_ADDR_V4MAPPED(&sin6->sin6_addr)) { - addr = p[12] << 24; + addr = (in_addr_t) p[12] << 24; addr += p[13] << 16; addr += p[14] << 8; addr += p[15]; diff --git a/src/http/modules/ngx_http_geo_module.c b/src/http/modules/ngx_http_geo_module.c --- a/src/http/modules/ngx_http_geo_module.c +++ b/src/http/modules/ngx_http_geo_module.c @@ -199,7 +199,7 @@ ngx_http_geo_cidr_variable(ngx_http_requ p = inaddr6->s6_addr; if (IN6_IS_ADDR_V4MAPPED(inaddr6)) { - inaddr = p[12] << 24; + inaddr = (in_addr_t) p[12] << 24; inaddr += p[13] << 16; inaddr += p[14] << 8; inaddr += p[15]; @@ -272,7 +272,7 @@ ngx_http_geo_range_variable(ngx_http_req if (IN6_IS_ADDR_V4MAPPED(inaddr6)) { p = inaddr6->s6_addr; - inaddr = p[12] << 24; + inaddr = (in_addr_t) p[12] << 24; inaddr += p[13] << 16; inaddr += p[14] << 8; inaddr += p[15]; diff --git a/src/http/modules/ngx_http_geoip_module.c b/src/http/modules/ngx_http_geoip_module.c --- a/src/http/modules/ngx_http_geoip_module.c +++ b/src/http/modules/ngx_http_geoip_module.c @@ -266,7 +266,7 @@ ngx_http_geoip_addr(ngx_http_request_t * if (IN6_IS_ADDR_V4MAPPED(inaddr6)) { p = inaddr6->s6_addr; - inaddr = p[12] << 24; + inaddr = (in_addr_t) p[12] << 24; inaddr += p[13] << 16; inaddr += p[14] << 8; inaddr += p[15]; diff --git a/src/stream/ngx_stream_access_module.c b/src/stream/ngx_stream_access_module.c --- a/src/stream/ngx_stream_access_module.c +++ b/src/stream/ngx_stream_access_module.c @@ -144,7 +144,7 @@ ngx_stream_access_handler(ngx_stream_ses p = sin6->sin6_addr.s6_addr; if (ascf->rules && IN6_IS_ADDR_V4MAPPED(&sin6->sin6_addr)) { - addr = p[12] << 24; + addr = (in_addr_t) p[12] << 24; addr += p[13] << 16; addr += p[14] << 8; addr += p[15]; diff --git a/src/stream/ngx_stream_geo_module.c b/src/stream/ngx_stream_geo_module.c --- a/src/stream/ngx_stream_geo_module.c +++ b/src/stream/ngx_stream_geo_module.c @@ -190,7 +190,7 @@ ngx_stream_geo_cidr_variable(ngx_stream_ p = inaddr6->s6_addr; if (IN6_IS_ADDR_V4MAPPED(inaddr6)) { - inaddr = p[12] << 24; + inaddr = (in_addr_t) p[12] << 24; inaddr += p[13] << 16; inaddr += p[14] << 8; inaddr += p[15]; @@ -263,7 +263,7 @@ ngx_stream_geo_range_variable(ngx_stream if (IN6_IS_ADDR_V4MAPPED(inaddr6)) { p = inaddr6->s6_addr; - inaddr = p[12] << 24; + inaddr = (in_addr_t) p[12] << 24; inaddr += p[13] << 16; inaddr += p[14] << 8; inaddr += p[15]; diff --git a/src/stream/ngx_stream_geoip_module.c b/src/stream/ngx_stream_geoip_module.c --- a/src/stream/ngx_stream_geoip_module.c +++ b/src/stream/ngx_stream_geoip_module.c @@ -236,7 +236,7 @@ ngx_stream_geoip_addr(ngx_stream_session if (IN6_IS_ADDR_V4MAPPED(inaddr6)) { p = inaddr6->s6_addr; - inaddr = p[12] << 24; + inaddr = (in_addr_t) p[12] << 24; inaddr += p[13] << 16; inaddr += p[14] << 8; inaddr += p[15]; From arut at nginx.com Mon Mar 18 13:55:36 2024 From: arut at nginx.com (Roman Arutyunyan) Date: Mon, 18 Mar 2024 17:55:36 +0400 Subject: [PATCH 1 of 2] SSL: add support for AWS-LC In-Reply-To: <5e923992006199748e79.1709083334@ip-172-31-36-66.ec2.internal> References: <5e923992006199748e79.1709083334@ip-172-31-36-66.ec2.internal> Message-ID: <20240318135536.bnqzgmqemz67pxgo@N00W24XTQX> Hi Piotr, On Wed, Feb 28, 2024 at 01:22:14AM +0000, Piotr Sikora via nginx-devel wrote: > # HG changeset patch > # User Piotr Sikora > # Date 1708977630 0 > # Mon Feb 26 20:00:30 2024 +0000 > # Branch patch009 > # Node ID 5e923992006199748e79b08b1e65c4ef41f07495 > # Parent 3cde11b747c08c69889edc014a700317fe4d1d88 > SSL: add support for AWS-LC. > > AWS-LC is a fork of BoringSSL with some performance improvements, > useful features (OCSP and multiple certificates), and support for > more platforms. > > Signed-off-by: Piotr Sikora > > diff -r 3cde11b747c0 -r 5e9239920061 src/event/ngx_event_openssl.h > --- a/src/event/ngx_event_openssl.h Mon Feb 26 20:00:28 2024 +0000 > +++ b/src/event/ngx_event_openssl.h Mon Feb 26 20:00:30 2024 +0000 > @@ -25,7 +25,7 @@ > #endif > #include > #if (NGX_QUIC) > -#ifdef OPENSSL_IS_BORINGSSL > +#if defined(OPENSSL_IS_BORINGSSL) || defined(OPENSSL_IS_AWSLC) > #include > #include > #else > diff -r 3cde11b747c0 -r 5e9239920061 src/event/quic/ngx_event_quic.c > --- a/src/event/quic/ngx_event_quic.c Mon Feb 26 20:00:28 2024 +0000 > +++ b/src/event/quic/ngx_event_quic.c Mon Feb 26 20:00:30 2024 +0000 > @@ -962,7 +962,7 @@ > return NGX_DECLINED; > } > > -#if !defined (OPENSSL_IS_BORINGSSL) > +#if !defined(OPENSSL_IS_BORINGSSL) && !defined(OPENSSL_IS_AWSLC) > /* OpenSSL provides read keys for an application level before it's ready */ > > if (pkt->level == ssl_encryption_application && !c->ssl->handshaked) { > diff -r 3cde11b747c0 -r 5e9239920061 src/event/quic/ngx_event_quic_protection.c > --- a/src/event/quic/ngx_event_quic_protection.c Mon Feb 26 20:00:28 2024 +0000 > +++ b/src/event/quic/ngx_event_quic_protection.c Mon Feb 26 20:00:30 2024 +0000 > @@ -30,7 +30,7 @@ > > static ngx_int_t ngx_quic_crypto_open(ngx_quic_secret_t *s, ngx_str_t *out, > u_char *nonce, ngx_str_t *in, ngx_str_t *ad, ngx_log_t *log); > -#ifndef OPENSSL_IS_BORINGSSL > +#if !defined(OPENSSL_IS_BORINGSSL) && !defined(OPENSSL_IS_AWSLC) > static ngx_int_t ngx_quic_crypto_common(ngx_quic_secret_t *s, ngx_str_t *out, > u_char *nonce, ngx_str_t *in, ngx_str_t *ad, ngx_log_t *log); > #endif > @@ -55,7 +55,7 @@ > switch (id) { > > case TLS1_3_CK_AES_128_GCM_SHA256: > -#ifdef OPENSSL_IS_BORINGSSL > +#if defined(OPENSSL_IS_BORINGSSL) || defined(OPENSSL_IS_AWSLC) > ciphers->c = EVP_aead_aes_128_gcm(); > #else > ciphers->c = EVP_aes_128_gcm(); > @@ -66,7 +66,7 @@ > break; > > case TLS1_3_CK_AES_256_GCM_SHA384: > -#ifdef OPENSSL_IS_BORINGSSL > +#if defined(OPENSSL_IS_BORINGSSL) || defined(OPENSSL_IS_AWSLC) > ciphers->c = EVP_aead_aes_256_gcm(); > #else > ciphers->c = EVP_aes_256_gcm(); > @@ -77,12 +77,12 @@ > break; > > case TLS1_3_CK_CHACHA20_POLY1305_SHA256: > -#ifdef OPENSSL_IS_BORINGSSL > +#if defined(OPENSSL_IS_BORINGSSL) || defined(OPENSSL_IS_AWSLC) > ciphers->c = EVP_aead_chacha20_poly1305(); > #else > ciphers->c = EVP_chacha20_poly1305(); > #endif > -#ifdef OPENSSL_IS_BORINGSSL > +#if defined(OPENSSL_IS_BORINGSSL) || defined(OPENSSL_IS_AWSLC) > ciphers->hp = (const EVP_CIPHER *) EVP_aead_chacha20_poly1305(); > #else > ciphers->hp = EVP_chacha20(); > @@ -91,7 +91,7 @@ > len = 32; > break; > > -#ifndef OPENSSL_IS_BORINGSSL > +#if !defined(OPENSSL_IS_BORINGSSL) && !defined(OPENSSL_IS_AWSLC) > case TLS1_3_CK_AES_128_CCM_SHA256: > ciphers->c = EVP_aes_128_ccm(); > ciphers->hp = EVP_aes_128_ctr(); > @@ -259,7 +259,7 @@ > ngx_hkdf_expand(u_char *out_key, size_t out_len, const EVP_MD *digest, > const uint8_t *prk, size_t prk_len, const u_char *info, size_t info_len) > { > -#ifdef OPENSSL_IS_BORINGSSL > +#if defined(OPENSSL_IS_BORINGSSL) || defined(OPENSSL_IS_AWSLC) > > if (HKDF_expand(out_key, out_len, digest, prk, prk_len, info, info_len) > == 0) > @@ -321,7 +321,7 @@ > const u_char *secret, size_t secret_len, const u_char *salt, > size_t salt_len) > { > -#ifdef OPENSSL_IS_BORINGSSL > +#if defined(OPENSSL_IS_BORINGSSL) || defined(OPENSSL_IS_AWSLC) > > if (HKDF_extract(out_key, out_len, digest, secret, secret_len, salt, > salt_len) > @@ -384,7 +384,7 @@ > ngx_quic_md_t *key, ngx_int_t enc, ngx_log_t *log) > { > > -#ifdef OPENSSL_IS_BORINGSSL > +#if defined(OPENSSL_IS_BORINGSSL) || defined(OPENSSL_IS_AWSLC) > EVP_AEAD_CTX *ctx; > > ctx = EVP_AEAD_CTX_new(cipher, key->data, key->len, > @@ -444,7 +444,7 @@ > ngx_quic_crypto_open(ngx_quic_secret_t *s, ngx_str_t *out, u_char *nonce, > ngx_str_t *in, ngx_str_t *ad, ngx_log_t *log) > { > -#ifdef OPENSSL_IS_BORINGSSL > +#if defined(OPENSSL_IS_BORINGSSL) || defined(OPENSSL_IS_AWSLC) > if (EVP_AEAD_CTX_open(s->ctx, out->data, &out->len, out->len, nonce, > s->iv.len, in->data, in->len, ad->data, ad->len) > != 1) > @@ -464,7 +464,7 @@ > ngx_quic_crypto_seal(ngx_quic_secret_t *s, ngx_str_t *out, u_char *nonce, > ngx_str_t *in, ngx_str_t *ad, ngx_log_t *log) > { > -#ifdef OPENSSL_IS_BORINGSSL > +#if defined(OPENSSL_IS_BORINGSSL) || defined(OPENSSL_IS_AWSLC) > if (EVP_AEAD_CTX_seal(s->ctx, out->data, &out->len, out->len, nonce, > s->iv.len, in->data, in->len, ad->data, ad->len) > != 1) > @@ -480,7 +480,7 @@ > } > > > -#ifndef OPENSSL_IS_BORINGSSL > +#if !defined(OPENSSL_IS_BORINGSSL) && !defined(OPENSSL_IS_AWSLC) > > static ngx_int_t > ngx_quic_crypto_common(ngx_quic_secret_t *s, ngx_str_t *out, u_char *nonce, > @@ -559,7 +559,7 @@ > ngx_quic_crypto_cleanup(ngx_quic_secret_t *s) > { > if (s->ctx) { > -#ifdef OPENSSL_IS_BORINGSSL > +#if defined(OPENSSL_IS_BORINGSSL) || defined(OPENSSL_IS_AWSLC) > EVP_AEAD_CTX_free(s->ctx); > #else > EVP_CIPHER_CTX_free(s->ctx); > @@ -575,7 +575,7 @@ > { > EVP_CIPHER_CTX *ctx; > > -#ifdef OPENSSL_IS_BORINGSSL > +#if defined(OPENSSL_IS_BORINGSSL) || defined(OPENSSL_IS_AWSLC) > if (cipher == (EVP_CIPHER *) EVP_aead_chacha20_poly1305()) { > /* no EVP interface */ > s->hp_ctx = NULL; > @@ -610,7 +610,7 @@ > > ctx = s->hp_ctx; > > -#ifdef OPENSSL_IS_BORINGSSL > +#if defined(OPENSSL_IS_BORINGSSL) || defined(OPENSSL_IS_AWSLC) > uint32_t cnt; > > if (ctx == NULL) { > diff -r 3cde11b747c0 -r 5e9239920061 src/event/quic/ngx_event_quic_protection.h > --- a/src/event/quic/ngx_event_quic_protection.h Mon Feb 26 20:00:28 2024 +0000 > +++ b/src/event/quic/ngx_event_quic_protection.h Mon Feb 26 20:00:30 2024 +0000 > @@ -24,7 +24,7 @@ > #define NGX_QUIC_MAX_MD_SIZE 48 > > > -#ifdef OPENSSL_IS_BORINGSSL > +#if defined(OPENSSL_IS_BORINGSSL) || defined(OPENSSL_IS_AWSLC) > #define ngx_quic_cipher_t EVP_AEAD > #define ngx_quic_crypto_ctx_t EVP_AEAD_CTX > #else > diff -r 3cde11b747c0 -r 5e9239920061 src/event/quic/ngx_event_quic_ssl.c > --- a/src/event/quic/ngx_event_quic_ssl.c Mon Feb 26 20:00:28 2024 +0000 > +++ b/src/event/quic/ngx_event_quic_ssl.c Mon Feb 26 20:00:30 2024 +0000 > @@ -11,6 +11,7 @@ > > > #if defined OPENSSL_IS_BORINGSSL \ > + || defined OPENSSL_IS_AWSLC \ > || defined LIBRESSL_VERSION_NUMBER \ > || NGX_QUIC_OPENSSL_COMPAT > #define NGX_QUIC_BORINGSSL_API 1 > @@ -578,7 +579,7 @@ > return NGX_ERROR; > } > > -#ifdef OPENSSL_IS_BORINGSSL > +#if defined(OPENSSL_IS_BORINGSSL) || defined(OPENSSL_IS_AWSLC) > if (SSL_set_quic_early_data_context(ssl_conn, p, clen) == 0) { > ngx_log_error(NGX_LOG_INFO, c->log, 0, > "quic SSL_set_quic_early_data_context() failed"); It looks like this library is not super popular, but the patch is relatively large. Also, compiling nginx with -DOPENSSL_IS_BORINGSSL should probably solve the issue. -- Roman Arutyunyan From pluknet at nginx.com Mon Mar 18 14:55:50 2024 From: pluknet at nginx.com (Sergey Kandaurov) Date: Mon, 18 Mar 2024 18:55:50 +0400 Subject: [PATCH] Core: fix build without libcrypt In-Reply-To: References: Message-ID: <44248533-2EB4-4E1D-84A4-BCFF8C05CE75@nginx.com> > On 28 Feb 2024, at 05:23, Piotr Sikora via nginx-devel wrote: > > # HG changeset patch > # User Piotr Sikora > # Date 1708977637 0 > # Mon Feb 26 20:00:37 2024 +0000 > # Branch patch013 > # Node ID cdc173477ea99fd6c952a85e5cd11db66452076a > # Parent 04e3155b3b9651fee708898aaf82ac35532806ee > Core: fix build without libcrypt. > > libcrypt is no longer part of glibc, so it might not be available. > > Signed-off-by: Piotr Sikora In popular RPM- and Debian-based Linux distributions with glibc 2.38, libcrypt.so is now indeed installed in a separate package. However, it appears that this package is critical and provides dependency either for systemd or libc, which cannot be removed. So, I don't see so far, how nginx is affected in practice. > > diff -r 04e3155b3b96 -r cdc173477ea9 auto/unix > --- a/auto/unix Mon Feb 26 20:00:35 2024 +0000 > +++ b/auto/unix Mon Feb 26 20:00:37 2024 +0000 > @@ -150,7 +150,7 @@ > > > ngx_feature="crypt()" > -ngx_feature_name= > +ngx_feature_name="NGX_HAVE_CRYPT" > ngx_feature_run=no > ngx_feature_incs= > ngx_feature_path= > @@ -162,7 +162,7 @@ > if [ $ngx_found = no ]; then > > ngx_feature="crypt() in libcrypt" > - ngx_feature_name= > + ngx_feature_name="NGX_HAVE_CRYPT" > ngx_feature_run=no > ngx_feature_incs= > ngx_feature_path= > diff -r 04e3155b3b96 -r cdc173477ea9 src/os/unix/ngx_linux_config.h > --- a/src/os/unix/ngx_linux_config.h Mon Feb 26 20:00:35 2024 +0000 > +++ b/src/os/unix/ngx_linux_config.h Mon Feb 26 20:00:37 2024 +0000 > @@ -52,7 +52,6 @@ > #include /* memalign() */ > #include /* IOV_MAX */ > #include > -#include > #include /* uname() */ > > #include > @@ -61,6 +60,11 @@ > #include > > > +#if (NGX_HAVE_CRYPT_H) > +#include > +#endif > + > + > #if (NGX_HAVE_POSIX_SEM) > #include > #endif > diff -r 04e3155b3b96 -r cdc173477ea9 src/os/unix/ngx_user.c > --- a/src/os/unix/ngx_user.c Mon Feb 26 20:00:35 2024 +0000 > +++ b/src/os/unix/ngx_user.c Mon Feb 26 20:00:37 2024 +0000 > @@ -44,7 +44,7 @@ > return NGX_ERROR; > } > > -#else > +#elif (NGX_HAVE_CRYPT) > > ngx_int_t > ngx_libc_crypt(ngx_pool_t *pool, u_char *key, u_char *salt, u_char **encrypted) > @@ -76,6 +76,14 @@ > return NGX_ERROR; > } > > +#else > + > +ngx_int_t > +ngx_libc_crypt(ngx_pool_t *pool, u_char *key, u_char *salt, u_char **encrypted) > +{ > + return NGX_ERROR; > +} > + > #endif > > #endif /* NGX_CRYPT */ > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > https://mailman.nginx.org/mailman/listinfo/nginx-devel -- Sergey Kandaurov From chipitsine at gmail.com Mon Mar 18 15:42:18 2024 From: chipitsine at gmail.com (=?UTF-8?B?0JjQu9GM0Y8g0KjQuNC/0LjRhtC40L0=?=) Date: Mon, 18 Mar 2024 16:42:18 +0100 Subject: [PATCH] Core: fix build without libcrypt In-Reply-To: <44248533-2EB4-4E1D-84A4-BCFF8C05CE75@nginx.com> References: <44248533-2EB4-4E1D-84A4-BCFF8C05CE75@nginx.com> Message-ID: пн, 18 мар. 2024 г. в 15:56, Sergey Kandaurov : > > > On 28 Feb 2024, at 05:23, Piotr Sikora via nginx-devel < > nginx-devel at nginx.org> wrote: > > > > # HG changeset patch > > # User Piotr Sikora > > # Date 1708977637 0 > > # Mon Feb 26 20:00:37 2024 +0000 > > # Branch patch013 > > # Node ID cdc173477ea99fd6c952a85e5cd11db66452076a > > # Parent 04e3155b3b9651fee708898aaf82ac35532806ee > > Core: fix build without libcrypt. > > > > libcrypt is no longer part of glibc, so it might not be available. > > > > Signed-off-by: Piotr Sikora > > In popular RPM- and Debian-based Linux distributions with glibc 2.38, > libcrypt.so is now indeed installed in a separate package. > However, it appears that this package is critical and provides > dependency either for systemd or libc, which cannot be removed. > So, I don't see so far, how nginx is affected in practice. > I guess docker images are meant. usually you do not have systemd inside docker > > > > > diff -r 04e3155b3b96 -r cdc173477ea9 auto/unix > > --- a/auto/unix Mon Feb 26 20:00:35 2024 +0000 > > +++ b/auto/unix Mon Feb 26 20:00:37 2024 +0000 > > @@ -150,7 +150,7 @@ > > > > > > ngx_feature="crypt()" > > -ngx_feature_name= > > +ngx_feature_name="NGX_HAVE_CRYPT" > > ngx_feature_run=no > > ngx_feature_incs= > > ngx_feature_path= > > @@ -162,7 +162,7 @@ > > if [ $ngx_found = no ]; then > > > > ngx_feature="crypt() in libcrypt" > > - ngx_feature_name= > > + ngx_feature_name="NGX_HAVE_CRYPT" > > ngx_feature_run=no > > ngx_feature_incs= > > ngx_feature_path= > > diff -r 04e3155b3b96 -r cdc173477ea9 src/os/unix/ngx_linux_config.h > > --- a/src/os/unix/ngx_linux_config.h Mon Feb 26 20:00:35 2024 +0000 > > +++ b/src/os/unix/ngx_linux_config.h Mon Feb 26 20:00:37 2024 +0000 > > @@ -52,7 +52,6 @@ > > #include /* memalign() */ > > #include /* IOV_MAX */ > > #include > > -#include > > #include /* uname() */ > > > > #include > > @@ -61,6 +60,11 @@ > > #include > > > > > > +#if (NGX_HAVE_CRYPT_H) > > +#include > > +#endif > > + > > + > > #if (NGX_HAVE_POSIX_SEM) > > #include > > #endif > > diff -r 04e3155b3b96 -r cdc173477ea9 src/os/unix/ngx_user.c > > --- a/src/os/unix/ngx_user.c Mon Feb 26 20:00:35 2024 +0000 > > +++ b/src/os/unix/ngx_user.c Mon Feb 26 20:00:37 2024 +0000 > > @@ -44,7 +44,7 @@ > > return NGX_ERROR; > > } > > > > -#else > > +#elif (NGX_HAVE_CRYPT) > > > > ngx_int_t > > ngx_libc_crypt(ngx_pool_t *pool, u_char *key, u_char *salt, u_char > **encrypted) > > @@ -76,6 +76,14 @@ > > return NGX_ERROR; > > } > > > > +#else > > + > > +ngx_int_t > > +ngx_libc_crypt(ngx_pool_t *pool, u_char *key, u_char *salt, u_char > **encrypted) > > +{ > > + return NGX_ERROR; > > +} > > + > > #endif > > > > #endif /* NGX_CRYPT */ > > _______________________________________________ > > nginx-devel mailing list > > nginx-devel at nginx.org > > https://mailman.nginx.org/mailman/listinfo/nginx-devel > > -- > Sergey Kandaurov > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > https://mailman.nginx.org/mailman/listinfo/nginx-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xeioex at nginx.com Tue Mar 19 05:56:50 2024 From: xeioex at nginx.com (=?utf-8?q?Dmitry_Volyntsev?=) Date: Tue, 19 Mar 2024 05:56:50 +0000 Subject: [njs] Introduced njs_vm_global() to get global object. Message-ID: details: https://hg.nginx.org/njs/rev/00b89201fb71 branches: changeset: 2297:00b89201fb71 user: Dmitry Volyntsev date: Thu Mar 14 23:28:03 2024 -0700 description: Introduced njs_vm_global() to get global object. diffstat: src/njs.h | 1 + src/njs_vm.c | 8 ++++++++ 2 files changed, 9 insertions(+), 0 deletions(-) diffs (29 lines): diff -r 8309b884e265 -r 00b89201fb71 src/njs.h --- a/src/njs.h Tue Mar 05 11:43:49 2024 -0800 +++ b/src/njs.h Thu Mar 14 23:28:03 2024 -0700 @@ -368,6 +368,7 @@ NJS_EXPORT njs_int_t njs_vm_bind(njs_vm_ njs_int_t njs_vm_bind_handler(njs_vm_t *vm, const njs_str_t *var_name, njs_prop_handler_t handler, uint16_t magic16, uint32_t magic32, njs_bool_t shared); +NJS_EXPORT njs_int_t njs_vm_global(njs_vm_t *vm, njs_value_t *retval); NJS_EXPORT njs_int_t njs_vm_value(njs_vm_t *vm, const njs_str_t *path, njs_value_t *retval); NJS_EXPORT njs_function_t *njs_vm_function(njs_vm_t *vm, const njs_str_t *name); diff -r 8309b884e265 -r 00b89201fb71 src/njs_vm.c --- a/src/njs_vm.c Tue Mar 05 11:43:49 2024 -0800 +++ b/src/njs_vm.c Thu Mar 14 23:28:03 2024 -0700 @@ -814,6 +814,14 @@ njs_vm_error3(njs_vm_t *vm, unsigned typ njs_int_t +njs_vm_global(njs_vm_t *vm, njs_value_t *retval) +{ + njs_value_assign(retval, &vm->global_value); + return NJS_OK; +} + + +njs_int_t njs_vm_value(njs_vm_t *vm, const njs_str_t *path, njs_value_t *retval) { u_char *start, *p, *end; From xeioex at nginx.com Tue Mar 19 05:56:52 2024 From: xeioex at nginx.com (=?utf-8?q?Dmitry_Volyntsev?=) Date: Tue, 19 Mar 2024 05:56:52 +0000 Subject: [njs] Introduced njs_vm_prototype() to get a value's prototype. Message-ID: details: https://hg.nginx.org/njs/rev/6808a27d254a branches: changeset: 2298:6808a27d254a user: Dmitry Volyntsev date: Thu Mar 14 23:28:03 2024 -0700 description: Introduced njs_vm_prototype() to get a value's prototype. diffstat: src/njs.h | 2 ++ src/njs_object.c | 2 +- src/njs_object.h | 2 ++ src/njs_vm.c | 13 +++++++++++++ 4 files changed, 18 insertions(+), 1 deletions(-) diffs (59 lines): diff -r 00b89201fb71 -r 6808a27d254a src/njs.h --- a/src/njs.h Thu Mar 14 23:28:03 2024 -0700 +++ b/src/njs.h Thu Mar 14 23:28:03 2024 -0700 @@ -373,6 +373,8 @@ NJS_EXPORT njs_int_t njs_vm_value(njs_vm njs_value_t *retval); NJS_EXPORT njs_function_t *njs_vm_function(njs_vm_t *vm, const njs_str_t *name); NJS_EXPORT njs_bool_t njs_vm_constructor(njs_vm_t *vm); +NJS_EXPORT njs_int_t njs_vm_prototype(njs_vm_t *vm, njs_value_t *value, + njs_value_t *retval); NJS_EXPORT void njs_vm_throw(njs_vm_t *vm, const njs_value_t *value); NJS_EXPORT void njs_vm_error2(njs_vm_t *vm, unsigned error_type, diff -r 00b89201fb71 -r 6808a27d254a src/njs_object.c --- a/src/njs_object.c Thu Mar 14 23:28:03 2024 -0700 +++ b/src/njs_object.c Thu Mar 14 23:28:03 2024 -0700 @@ -1598,7 +1598,7 @@ njs_object_get_own_property(njs_vm_t *vm } -static njs_int_t +njs_int_t njs_object_get_prototype_of(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, njs_index_t unused, njs_value_t *retval) { diff -r 00b89201fb71 -r 6808a27d254a src/njs_object.h --- a/src/njs_object.h Thu Mar 14 23:28:03 2024 -0700 +++ b/src/njs_object.h Thu Mar 14 23:28:03 2024 -0700 @@ -106,6 +106,8 @@ njs_int_t njs_object_prop_define(njs_vm_ njs_value_t *name, njs_value_t *value, unsigned flags, uint32_t hash); njs_int_t njs_object_prop_descriptor(njs_vm_t *vm, njs_value_t *dest, njs_value_t *value, njs_value_t *setval); +njs_int_t njs_object_get_prototype_of(njs_vm_t *vm, njs_value_t *args, + njs_uint_t nargs, njs_index_t unused, njs_value_t *retval); const char *njs_prop_type_string(njs_object_prop_type_t type); njs_int_t njs_object_prop_init(njs_vm_t *vm, const njs_object_init_t* init, const njs_object_prop_t *base, njs_value_t *value, njs_value_t *retval); diff -r 00b89201fb71 -r 6808a27d254a src/njs_vm.c --- a/src/njs_vm.c Thu Mar 14 23:28:03 2024 -0700 +++ b/src/njs_vm.c Thu Mar 14 23:28:03 2024 -0700 @@ -1201,6 +1201,19 @@ njs_vm_object_keys(njs_vm_t *vm, njs_val njs_int_t +njs_vm_prototype(njs_vm_t *vm, njs_value_t *value, njs_value_t *retval) +{ + njs_value_t arguments[2]; + + njs_set_undefined(&arguments[0]); + njs_value_assign(&arguments[1], value); + + return njs_object_get_prototype_of(vm, njs_value_arg(&arguments), 2, 0, + retval); +} + + +njs_int_t njs_vm_array_alloc(njs_vm_t *vm, njs_value_t *retval, uint32_t spare) { njs_array_t *array; From xeioex at nginx.com Tue Mar 19 05:56:54 2024 From: xeioex at nginx.com (=?utf-8?q?Dmitry_Volyntsev?=) Date: Tue, 19 Mar 2024 05:56:54 +0000 Subject: [njs] Refactoring njs_value_own_enumerate() and friends. Message-ID: details: https://hg.nginx.org/njs/rev/b0e42db5ca84 branches: changeset: 2299:b0e42db5ca84 user: Dmitry Volyntsev date: Fri Mar 15 22:47:50 2024 -0700 description: Refactoring njs_value_own_enumerate() and friends. All three flag-like arguments are merged into a single flag argument. diffstat: src/njs_array.c | 4 +- src/njs_builtin.c | 2 +- src/njs_json.c | 8 +- src/njs_object.c | 154 +++++++++++++++++++++++++---------------------------- src/njs_object.h | 4 +- src/njs_value.c | 20 ++---- src/njs_value.h | 21 +++--- src/njs_vm.c | 4 +- src/njs_vmcode.c | 6 +- 9 files changed, 106 insertions(+), 117 deletions(-) diffs (727 lines): diff -r 6808a27d254a -r b0e42db5ca84 src/njs_array.c --- a/src/njs_array.c Thu Mar 14 23:28:03 2024 -0700 +++ b/src/njs_array.c Fri Mar 15 22:47:50 2024 -0700 @@ -1868,8 +1868,8 @@ njs_array_keys(njs_vm_t *vm, njs_value_t { njs_array_t *keys; - keys = njs_value_own_enumerate(vm, object, NJS_ENUM_KEYS, NJS_ENUM_STRING, - all); + keys = njs_value_own_enumerate(vm, object, NJS_ENUM_KEYS | NJS_ENUM_STRING + | (!all ? NJS_ENUM_ENUMERABLE_ONLY : 0)); if (njs_slow_path(keys == NULL)) { return NULL; } diff -r 6808a27d254a -r b0e42db5ca84 src/njs_builtin.c --- a/src/njs_builtin.c Thu Mar 14 23:28:03 2024 -0700 +++ b/src/njs_builtin.c Fri Mar 15 22:47:50 2024 -0700 @@ -648,7 +648,7 @@ njs_object_completions(njs_vm_t *vm, njs object->type = NJS_OBJECT; } - keys = njs_value_enumerate(vm, object, NJS_ENUM_KEYS, NJS_ENUM_STRING, 1); + keys = njs_value_enumerate(vm, object, NJS_ENUM_KEYS | NJS_ENUM_STRING); if (njs_slow_path(keys == NULL)) { goto done; } diff -r 6808a27d254a -r b0e42db5ca84 src/njs_json.c --- a/src/njs_json.c Thu Mar 14 23:28:03 2024 -0700 +++ b/src/njs_json.c Fri Mar 15 22:47:50 2024 -0700 @@ -44,7 +44,7 @@ typedef struct { njs_value_t replacer; njs_str_t space; u_char space_buf[16]; - njs_object_enum_type_t keys_type; + uint32_t keys_type; } njs_json_stringify_t; @@ -998,8 +998,10 @@ njs_json_push_stringify_state(njs_json_s } } else { - state->keys = njs_value_own_enumerate(stringify->vm, value, NJS_ENUM_KEYS, - stringify->keys_type, 0); + state->keys = njs_value_own_enumerate(stringify->vm, value, + NJS_ENUM_KEYS + | stringify->keys_type + | NJS_ENUM_ENUMERABLE_ONLY); if (njs_slow_path(state->keys == NULL)) { return NULL; diff -r 6808a27d254a -r b0e42db5ca84 src/njs_object.c --- a/src/njs_object.c Thu Mar 14 23:28:03 2024 -0700 +++ b/src/njs_object.c Fri Mar 15 22:47:50 2024 -0700 @@ -18,17 +18,16 @@ static njs_int_t njs_object_hash_test(nj static njs_object_prop_t *njs_object_exist_in_proto(const njs_object_t *begin, const njs_object_t *end, njs_lvlhsh_query_t *lhq); static njs_int_t njs_object_enumerate_array(njs_vm_t *vm, - const njs_array_t *array, njs_array_t *items, njs_object_enum_t kind); + const njs_array_t *array, njs_array_t *items, uint32_t flags); static njs_int_t njs_object_enumerate_typed_array(njs_vm_t *vm, - const njs_typed_array_t *array, njs_array_t *items, njs_object_enum_t kind); + const njs_typed_array_t *array, njs_array_t *items, uint32_t flags); static njs_int_t njs_object_enumerate_string(njs_vm_t *vm, - const njs_value_t *value, njs_array_t *items, njs_object_enum_t kind); + const njs_value_t *value, njs_array_t *items, uint32_t flags); static njs_int_t njs_object_enumerate_object(njs_vm_t *vm, - const njs_object_t *object, njs_array_t *items, njs_object_enum_t kind, - njs_object_enum_type_t type, njs_bool_t all); + const njs_object_t *object, njs_array_t *items, uint32_t flags); static njs_int_t njs_object_own_enumerate_object(njs_vm_t *vm, const njs_object_t *object, const njs_object_t *parent, njs_array_t *items, - njs_object_enum_t kind, njs_object_enum_type_t type, njs_bool_t all); + uint32_t flags); static njs_int_t njs_object_define_properties(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, njs_index_t unused, njs_value_t *retval); static njs_int_t njs_object_set_prototype(njs_vm_t *vm, njs_object_t *object, @@ -324,8 +323,8 @@ njs_object_keys(njs_vm_t *vm, njs_value_ return NJS_ERROR; } - keys = njs_value_own_enumerate(vm, value, NJS_ENUM_KEYS, - NJS_ENUM_STRING, 0); + keys = njs_value_own_enumerate(vm, value, NJS_ENUM_KEYS | NJS_ENUM_STRING + | NJS_ENUM_ENUMERABLE_ONLY); if (keys == NULL) { return NJS_ERROR; } @@ -352,8 +351,8 @@ njs_object_values(njs_vm_t *vm, njs_valu return NJS_ERROR; } - array = njs_value_own_enumerate(vm, value, NJS_ENUM_VALUES, - NJS_ENUM_STRING, 0); + array = njs_value_own_enumerate(vm, value, NJS_ENUM_VALUES | NJS_ENUM_STRING + | NJS_ENUM_ENUMERABLE_ONLY); if (array == NULL) { return NJS_ERROR; } @@ -380,8 +379,8 @@ njs_object_entries(njs_vm_t *vm, njs_val return NJS_ERROR; } - array = njs_value_own_enumerate(vm, value, NJS_ENUM_BOTH, - NJS_ENUM_STRING, 0); + array = njs_value_own_enumerate(vm, value, NJS_ENUM_BOTH | NJS_ENUM_STRING + | NJS_ENUM_ENUMERABLE_ONLY); if (array == NULL) { return NJS_ERROR; } @@ -429,23 +428,22 @@ next: njs_inline njs_int_t njs_object_enumerate_value(njs_vm_t *vm, const njs_object_t *object, - njs_array_t *items, njs_object_enum_t kind, njs_object_enum_type_t type, - njs_bool_t all) + njs_array_t *items, uint32_t flags) { njs_int_t ret; njs_object_value_t *obj_val; - if (type & NJS_ENUM_STRING) { + if (flags & NJS_ENUM_STRING) { switch (object->type) { case NJS_ARRAY: ret = njs_object_enumerate_array(vm, (njs_array_t *) object, items, - kind); + flags); break; case NJS_TYPED_ARRAY: ret = njs_object_enumerate_typed_array(vm, (njs_typed_array_t *) object, - items, kind); + items, flags); break; case NJS_OBJECT_VALUE: @@ -453,7 +451,7 @@ njs_object_enumerate_value(njs_vm_t *vm, if (njs_is_string(&obj_val->value)) { ret = njs_object_enumerate_string(vm, &obj_val->value, items, - kind); + flags); break; } @@ -470,7 +468,7 @@ njs_object_enumerate_value(njs_vm_t *vm, object: - ret = njs_object_enumerate_object(vm, object, items, kind, type, all); + ret = njs_object_enumerate_object(vm, object, items, flags); if (njs_slow_path(ret != NJS_OK)) { return NJS_ERROR; } @@ -481,23 +479,22 @@ object: njs_inline njs_int_t njs_object_own_enumerate_value(njs_vm_t *vm, const njs_object_t *object, - const njs_object_t *parent, njs_array_t *items, njs_object_enum_t kind, - njs_object_enum_type_t type, njs_bool_t all) + const njs_object_t *parent, njs_array_t *items, uint32_t flags) { njs_int_t ret; njs_object_value_t *obj_val; - if (type & NJS_ENUM_STRING) { + if (flags & NJS_ENUM_STRING) { switch (object->type) { case NJS_ARRAY: ret = njs_object_enumerate_array(vm, (njs_array_t *) object, items, - kind); + flags); break; case NJS_TYPED_ARRAY: ret = njs_object_enumerate_typed_array(vm, (njs_typed_array_t *) object, - items, kind); + items, flags); break; case NJS_OBJECT_VALUE: @@ -505,7 +502,7 @@ njs_object_own_enumerate_value(njs_vm_t if (njs_is_string(&obj_val->value)) { ret = njs_object_enumerate_string(vm, &obj_val->value, items, - kind); + flags); break; } @@ -522,8 +519,7 @@ njs_object_own_enumerate_value(njs_vm_t object: - ret = njs_object_own_enumerate_object(vm, object, parent, items, kind, - type, all); + ret = njs_object_own_enumerate_object(vm, object, parent, items, flags); if (njs_slow_path(ret != NJS_OK)) { return NJS_ERROR; } @@ -534,7 +530,7 @@ object: njs_array_t * njs_object_enumerate(njs_vm_t *vm, const njs_object_t *object, - njs_object_enum_t kind, njs_object_enum_type_t type, njs_bool_t all) + uint32_t flags) { njs_int_t ret; njs_array_t *items; @@ -544,7 +540,7 @@ njs_object_enumerate(njs_vm_t *vm, const return NULL; } - ret = njs_object_enumerate_value(vm, object, items, kind, type, all); + ret = njs_object_enumerate_value(vm, object, items, flags); if (njs_slow_path(ret != NJS_OK)) { return NULL; } @@ -555,7 +551,7 @@ njs_object_enumerate(njs_vm_t *vm, const njs_array_t * njs_object_own_enumerate(njs_vm_t *vm, const njs_object_t *object, - njs_object_enum_t kind, njs_object_enum_type_t type, njs_bool_t all) + uint32_t flags) { njs_int_t ret; njs_array_t *items; @@ -565,8 +561,7 @@ njs_object_own_enumerate(njs_vm_t *vm, c return NULL; } - ret = njs_object_own_enumerate_value(vm, object, object, items, kind, type, - all); + ret = njs_object_own_enumerate_value(vm, object, object, items, flags); if (njs_slow_path(ret != NJS_OK)) { return NULL; } @@ -576,16 +571,16 @@ njs_object_own_enumerate(njs_vm_t *vm, c njs_inline njs_bool_t -njs_is_enumerable(const njs_value_t *value, njs_object_enum_type_t type) +njs_is_enumerable(const njs_value_t *value, uint32_t flags) { - return (njs_is_string(value) && (type & NJS_ENUM_STRING)) - || (njs_is_symbol(value) && (type & NJS_ENUM_SYMBOL)); + return (njs_is_string(value) && (flags & NJS_ENUM_STRING)) + || (njs_is_symbol(value) && (flags & NJS_ENUM_SYMBOL)); } static njs_int_t njs_object_enumerate_array(njs_vm_t *vm, const njs_array_t *array, - njs_array_t *items, njs_object_enum_t kind) + njs_array_t *items, uint32_t flags) { njs_int_t ret; njs_value_t *p, *start, *end; @@ -600,7 +595,7 @@ njs_object_enumerate_array(njs_vm_t *vm, p = start; end = p + array->length; - switch (kind) { + switch (njs_object_enum_kind(flags)) { case NJS_ENUM_KEYS: while (p < end) { if (njs_is_valid(p)) { @@ -662,7 +657,7 @@ njs_object_enumerate_array(njs_vm_t *vm, static njs_int_t njs_object_enumerate_typed_array(njs_vm_t *vm, const njs_typed_array_t *array, - njs_array_t *items, njs_object_enum_t kind) + njs_array_t *items, uint32_t flags) { uint32_t i, length; njs_int_t ret; @@ -678,7 +673,7 @@ njs_object_enumerate_typed_array(njs_vm_ item = &items->start[items->length]; - switch (kind) { + switch (njs_object_enum_kind(flags)) { case NJS_ENUM_KEYS: for (i = 0; i < length; i++) { njs_uint32_to_string(item++, i); @@ -717,7 +712,7 @@ njs_object_enumerate_typed_array(njs_vm_ static njs_int_t njs_object_enumerate_string(njs_vm_t *vm, const njs_value_t *value, - njs_array_t *items, njs_object_enum_t kind) + njs_array_t *items, uint32_t flags) { u_char *begin; uint32_t i, len, size; @@ -736,7 +731,7 @@ njs_object_enumerate_string(njs_vm_t *vm item = &items->start[items->length]; - switch (kind) { + switch (njs_object_enum_kind(flags)) { case NJS_ENUM_KEYS: for (i = 0; i < len; i++) { njs_uint32_to_string(item++, i); @@ -843,14 +838,12 @@ njs_object_enumerate_string(njs_vm_t *vm static njs_int_t njs_object_enumerate_object(njs_vm_t *vm, const njs_object_t *object, - njs_array_t *items, njs_object_enum_t kind, njs_object_enum_type_t type, - njs_bool_t all) + njs_array_t *items, uint32_t flags) { njs_int_t ret; const njs_object_t *proto; - ret = njs_object_own_enumerate_object(vm, object, object, items, kind, - type, all); + ret = njs_object_own_enumerate_object(vm, object, object, items, flags); if (njs_slow_path(ret != NJS_OK)) { return NJS_ERROR; } @@ -858,8 +851,7 @@ njs_object_enumerate_object(njs_vm_t *vm proto = object->__proto__; while (proto != NULL) { - ret = njs_object_own_enumerate_value(vm, proto, object, items, kind, - type, all); + ret = njs_object_own_enumerate_value(vm, proto, object, items, flags); if (njs_slow_path(ret != NJS_OK)) { return NJS_ERROR; } @@ -871,8 +863,8 @@ njs_object_enumerate_object(njs_vm_t *vm } -#define njs_process_prop(vm, prop, type, items, items_symbol) \ - if (!(type & NJS_ENUM_SYMBOL && njs_is_symbol(&prop->name))) { \ +#define njs_process_prop(vm, prop, flags, items, items_symbol) \ + if (!(flags & NJS_ENUM_SYMBOL && njs_is_symbol(&prop->name))) { \ /* \ * prop from shared_hash is not symbol: \ * add to items before props from hash \ @@ -897,8 +889,7 @@ njs_object_enumerate_object(njs_vm_t *vm static njs_int_t njs_get_own_ordered_keys(njs_vm_t *vm, const njs_object_t *object, - const njs_object_t *parent, njs_array_t *items, - njs_object_enum_type_t type, njs_bool_t all) + const njs_object_t *parent, njs_array_t *items, uint32_t flags) { double num; uint32_t items_length; @@ -933,7 +924,7 @@ njs_get_own_ordered_keys(njs_vm_t *vm, c break; } - if (!njs_is_enumerable(&prop->name, type)) { + if (!njs_is_enumerable(&prop->name, flags)) { continue; } @@ -947,7 +938,7 @@ njs_get_own_ordered_keys(njs_vm_t *vm, c ret = njs_lvlhsh_find(&object->hash, &lhq); if (ret != NJS_OK) { - if (!(prop->enumerable || all)) { + if (!(prop->enumerable || !(flags & NJS_ENUM_ENUMERABLE_ONLY))) { continue; } @@ -955,7 +946,7 @@ njs_get_own_ordered_keys(njs_vm_t *vm, c num = njs_string_to_index(&prop->name); if (!njs_number_is_integer_index(num)) { - njs_process_prop(vm, prop, type, items_string, items_symbol); + njs_process_prop(vm, prop, flags, items_string, items_symbol); } else { ret = njs_array_add(vm, items, &prop->name); @@ -966,7 +957,9 @@ njs_get_own_ordered_keys(njs_vm_t *vm, c } else { - if (!(((njs_object_prop_t *)(lhq.value))->enumerable || all)) { + if (!(((njs_object_prop_t *)(lhq.value))->enumerable + || !(flags & NJS_ENUM_ENUMERABLE_ONLY))) + { continue; } @@ -984,7 +977,7 @@ njs_get_own_ordered_keys(njs_vm_t *vm, c if (hash_prop->type != NJS_WHITEOUT && !(hash_prop->enum_in_object_hash)) { - njs_process_prop(vm, prop, type, items_string, + njs_process_prop(vm, prop, flags, items_string, items_symbol); } } @@ -1001,8 +994,8 @@ njs_get_own_ordered_keys(njs_vm_t *vm, c break; } - if (!njs_is_enumerable(&prop->name, type) || - !(prop->enumerable || all) || + if (!njs_is_enumerable(&prop->name, flags) || + !(prop->enumerable || !(flags & NJS_ENUM_ENUMERABLE_ONLY)) || prop->type == NJS_WHITEOUT) { continue; @@ -1031,7 +1024,7 @@ njs_get_own_ordered_keys(njs_vm_t *vm, c /* select names of not deleted props */ - njs_process_prop(vm, prop, type, items_string, + njs_process_prop(vm, prop, flags, items_string, items_symbol); } else { @@ -1040,7 +1033,7 @@ njs_get_own_ordered_keys(njs_vm_t *vm, c /* select names of not deleted and created again */ if (prop->enum_in_object_hash) { - njs_process_prop(vm, prop, type, items_string, + njs_process_prop(vm, prop, flags, items_string, items_symbol); } } @@ -1086,7 +1079,7 @@ njs_get_own_ordered_keys(njs_vm_t *vm, c static njs_int_t njs_add_obj_prop_kind(njs_vm_t *vm, const njs_object_t *object, const njs_lvlhsh_t *hash, njs_lvlhsh_query_t *lhq, - njs_object_enum_t kind, njs_array_t *items) + uint32_t flags, njs_array_t *items) { njs_int_t ret; njs_value_t value, *v, value1; @@ -1124,7 +1117,7 @@ njs_add_obj_prop_kind(njs_vm_t *vm, cons } add: - if (kind != NJS_ENUM_VALUES) { + if (njs_object_enum_kind(flags) != NJS_ENUM_VALUES) { entry = njs_array_alloc(vm, 0, 2, 0); if (njs_slow_path(entry == NULL)) { return NJS_ERROR; @@ -1148,8 +1141,7 @@ add: static njs_int_t njs_object_own_enumerate_object(njs_vm_t *vm, const njs_object_t *object, - const njs_object_t *parent, njs_array_t *items, njs_object_enum_t kind, - njs_object_enum_type_t type, njs_bool_t all) + const njs_object_t *parent, njs_array_t *items, uint32_t flags) { njs_int_t ret; uint32_t i; @@ -1161,9 +1153,9 @@ njs_object_own_enumerate_object(njs_vm_t njs_lvlhsh_each_init(&lhe, &njs_object_hash_proto); - switch (kind) { + switch (njs_object_enum_kind(flags)) { case NJS_ENUM_KEYS: - ret = njs_get_own_ordered_keys(vm, object, parent, items, type, all); + ret = njs_get_own_ordered_keys(vm, object, parent, items, flags); if (ret != NJS_OK) { return NJS_ERROR; } @@ -1177,8 +1169,7 @@ njs_object_own_enumerate_object(njs_vm_t return NJS_ERROR; } - ret = njs_get_own_ordered_keys(vm, object, parent, items_sorted, type, - all); + ret = njs_get_own_ordered_keys(vm, object, parent, items_sorted, flags); if (ret != NJS_OK) { return NJS_ERROR; } @@ -1189,7 +1180,7 @@ njs_object_own_enumerate_object(njs_vm_t njs_object_property_key_set(&lhq, &items_sorted->start[i], lhe.key_hash); - ret = njs_add_obj_prop_kind(vm, object, &object->hash, &lhq, kind, + ret = njs_add_obj_prop_kind(vm, object, &object->hash, &lhq, flags, items); if (ret != NJS_DECLINED) { if (ret != NJS_OK) { @@ -1198,7 +1189,7 @@ njs_object_own_enumerate_object(njs_vm_t } else { ret = njs_add_obj_prop_kind(vm, object, &object->shared_hash, - &lhq, kind, items); + &lhq, flags, items); njs_assert(ret != NJS_DECLINED); if (ret != NJS_OK) { return NJS_ERROR; @@ -1277,8 +1268,8 @@ njs_object_traverse(njs_vm_t *vm, njs_ob s->parent = NULL; s->index = 0; njs_set_object(&s->value, object); - s->keys = njs_value_own_enumerate(vm, &s->value, NJS_ENUM_KEYS, - NJS_ENUM_STRING | NJS_ENUM_SYMBOL, 1); + s->keys = njs_value_own_enumerate(vm, &s->value, NJS_ENUM_KEYS + | NJS_ENUM_STRING | NJS_ENUM_SYMBOL); if (njs_slow_path(s->keys == NULL)) { return NJS_ERROR; } @@ -1357,8 +1348,8 @@ njs_object_traverse(njs_vm_t *vm, njs_ob s->parent = &s[-1]; s->index = 0; njs_value_assign(&s->value, &value); - s->keys = njs_value_own_enumerate(vm, &s->value, NJS_ENUM_KEYS, - NJS_ENUM_STRING | NJS_ENUM_SYMBOL, 1); + s->keys = njs_value_own_enumerate(vm, &s->value, NJS_ENUM_KEYS + | NJS_ENUM_STRING | NJS_ENUM_SYMBOL); if (njs_slow_path(s->keys == NULL)) { return NJS_ERROR; } @@ -1429,8 +1420,8 @@ njs_object_define_properties(njs_vm_t *v return ret; } - keys = njs_value_own_enumerate(vm, descs, NJS_ENUM_KEYS, - NJS_ENUM_STRING | NJS_ENUM_SYMBOL, 0); + keys = njs_value_own_enumerate(vm, descs, NJS_ENUM_KEYS | NJS_ENUM_STRING + | NJS_ENUM_SYMBOL); if (njs_slow_path(keys == NULL)) { return NJS_ERROR; } @@ -1517,8 +1508,8 @@ njs_object_get_own_property_descriptors( return NJS_ERROR; } - names = njs_value_own_enumerate(vm, value, NJS_ENUM_KEYS, - NJS_ENUM_STRING | NJS_ENUM_SYMBOL, 1); + names = njs_value_own_enumerate(vm, value, NJS_ENUM_KEYS | NJS_ENUM_STRING + | NJS_ENUM_SYMBOL); if (njs_slow_path(names == NULL)) { return NJS_ERROR; } @@ -1586,8 +1577,7 @@ njs_object_get_own_property(njs_vm_t *vm return NJS_ERROR; } - names = njs_value_own_enumerate(vm, value, NJS_ENUM_KEYS, - type, 1); + names = njs_value_own_enumerate(vm, value, NJS_ENUM_KEYS | type); if (names == NULL) { return NJS_ERROR; } @@ -1876,8 +1866,8 @@ njs_object_assign(njs_vm_t *vm, njs_valu for (i = 2; i < nargs; i++) { source = &args[i]; - names = njs_value_own_enumerate(vm, source, NJS_ENUM_KEYS, - NJS_ENUM_STRING | NJS_ENUM_SYMBOL, 1); + names = njs_value_own_enumerate(vm, source, NJS_ENUM_KEYS + | NJS_ENUM_STRING | NJS_ENUM_SYMBOL); if (njs_slow_path(names == NULL)) { return NJS_ERROR; } diff -r 6808a27d254a -r b0e42db5ca84 src/njs_object.h --- a/src/njs_object.h Thu Mar 14 23:28:03 2024 -0700 +++ b/src/njs_object.h Fri Mar 15 22:47:50 2024 -0700 @@ -67,9 +67,9 @@ njs_object_t *njs_object_value_copy(njs_ njs_object_value_t *njs_object_value_alloc(njs_vm_t *vm, njs_uint_t index, size_t extra,const njs_value_t *value); njs_array_t *njs_object_enumerate(njs_vm_t *vm, const njs_object_t *object, - njs_object_enum_t kind, njs_object_enum_type_t type, njs_bool_t all); + uint32_t flags); njs_array_t *njs_object_own_enumerate(njs_vm_t *vm, const njs_object_t *object, - njs_object_enum_t kind, njs_object_enum_type_t type, njs_bool_t all); + uint32_t flags); njs_int_t njs_object_traverse(njs_vm_t *vm, njs_object_t *object, void *ctx, njs_object_traverse_cb_t cb); njs_int_t njs_object_hash_create(njs_vm_t *vm, njs_lvlhsh_t *hash, diff -r 6808a27d254a -r b0e42db5ca84 src/njs_value.c --- a/src/njs_value.c Thu Mar 14 23:28:03 2024 -0700 +++ b/src/njs_value.c Fri Mar 15 22:47:50 2024 -0700 @@ -136,8 +136,7 @@ njs_value_to_primitive(njs_vm_t *vm, njs njs_array_t * -njs_value_enumerate(njs_vm_t *vm, njs_value_t *value, - njs_object_enum_t kind, njs_object_enum_type_t type, njs_bool_t all) +njs_value_enumerate(njs_vm_t *vm, njs_value_t *value, uint32_t flags) { njs_int_t ret; njs_value_t keys; @@ -145,7 +144,7 @@ njs_value_enumerate(njs_vm_t *vm, njs_va njs_exotic_slots_t *slots; if (njs_is_object(value)) { - if (kind == NJS_ENUM_KEYS && (type & NJS_ENUM_STRING)) { + if ((flags & NJS_ENUM_KEYS) && (flags & NJS_ENUM_STRING)) { slots = njs_object_slots(value); if (slots != NULL && slots->keys != NULL) { ret = slots->keys(vm, value, &keys); @@ -157,7 +156,7 @@ njs_value_enumerate(njs_vm_t *vm, njs_va } } - return njs_object_enumerate(vm, njs_object(value), kind, type, all); + return njs_object_enumerate(vm, njs_object(value), flags); } if (value->type != NJS_STRING) { @@ -167,13 +166,12 @@ njs_value_enumerate(njs_vm_t *vm, njs_va obj_val.object = vm->string_object; obj_val.value = *value; - return njs_object_enumerate(vm, (njs_object_t *) &obj_val, kind, type, all); + return njs_object_enumerate(vm, (njs_object_t *) &obj_val, flags); } njs_array_t * -njs_value_own_enumerate(njs_vm_t *vm, njs_value_t *value, - njs_object_enum_t kind, njs_object_enum_type_t type, njs_bool_t all) +njs_value_own_enumerate(njs_vm_t *vm, njs_value_t *value, uint32_t flags) { njs_int_t ret, len; njs_array_t *values, *entry; @@ -184,8 +182,7 @@ njs_value_own_enumerate(njs_vm_t *vm, nj if (njs_is_object(value)) { slots = njs_object_slots(value); if (slots == NULL || slots->keys == NULL) { - return njs_object_own_enumerate(vm, njs_object(value), kind, type, - all); + return njs_object_own_enumerate(vm, njs_object(value), flags); } ret = slots->keys(vm, value, &keys); @@ -193,7 +190,7 @@ njs_value_own_enumerate(njs_vm_t *vm, nj return NULL; } - switch (kind) { + switch (njs_object_enum_kind(flags)) { case NJS_ENUM_KEYS: return njs_array(&keys); @@ -258,8 +255,7 @@ njs_value_own_enumerate(njs_vm_t *vm, nj obj_val.object = vm->string_object; obj_val.value = *value; - return njs_object_own_enumerate(vm, (njs_object_t *) &obj_val, kind, - type, all); + return njs_object_own_enumerate(vm, (njs_object_t *) &obj_val, flags); } diff -r 6808a27d254a -r b0e42db5ca84 src/njs_value.h --- a/src/njs_value.h Thu Mar 14 23:28:03 2024 -0700 +++ b/src/njs_value.h Fri Mar 15 22:47:50 2024 -0700 @@ -313,19 +313,18 @@ struct njs_object_type_init_s { typedef enum { - NJS_ENUM_KEYS, - NJS_ENUM_VALUES, - NJS_ENUM_BOTH, +#define njs_object_enum_kind(flags) (flags & 7) + NJS_ENUM_KEYS = 1, + NJS_ENUM_VALUES = 2, + NJS_ENUM_BOTH = 4, +#define njs_object_enum(flags) (flags & (NJS_ENUM_STRING | NJS_ENUM_SYMBOL)) + NJS_ENUM_STRING = 8, + NJS_ENUM_SYMBOL = 16, + NJS_ENUM_ENUMERABLE_ONLY = 32, } njs_object_enum_t; typedef enum { - NJS_ENUM_STRING = 1, - NJS_ENUM_SYMBOL = 2, -} njs_object_enum_type_t; - - -typedef enum { NJS_PROPERTY = 0, NJS_ACCESSOR, NJS_PROPERTY_REF, @@ -1032,9 +1031,9 @@ njs_set_object_value(njs_value_t *value, njs_int_t njs_value_to_primitive(njs_vm_t *vm, njs_value_t *dst, njs_value_t *value, njs_uint_t hint); njs_array_t *njs_value_enumerate(njs_vm_t *vm, njs_value_t *value, - njs_object_enum_t kind, njs_object_enum_type_t type, njs_bool_t all); + uint32_t flags); njs_array_t *njs_value_own_enumerate(njs_vm_t *vm, njs_value_t *value, - njs_object_enum_t kind, njs_object_enum_type_t type, njs_bool_t all); + uint32_t flags); njs_int_t njs_value_of(njs_vm_t *vm, njs_value_t *value, njs_value_t *retval); njs_int_t njs_value_length(njs_vm_t *vm, njs_value_t *value, int64_t *dst); const char *njs_type_string(njs_value_type_t type); diff -r 6808a27d254a -r b0e42db5ca84 src/njs_vm.c --- a/src/njs_vm.c Thu Mar 14 23:28:03 2024 -0700 +++ b/src/njs_vm.c Fri Mar 15 22:47:50 2024 -0700 @@ -1188,8 +1188,8 @@ njs_vm_object_keys(njs_vm_t *vm, njs_val { njs_array_t *keys; - keys = njs_value_own_enumerate(vm, value, NJS_ENUM_KEYS, - NJS_ENUM_STRING, 0); + keys = njs_value_own_enumerate(vm, value, NJS_ENUM_KEYS | NJS_ENUM_STRING + | NJS_ENUM_ENUMERABLE_ONLY); if (njs_slow_path(keys == NULL)) { return NULL; } diff -r 6808a27d254a -r b0e42db5ca84 src/njs_vmcode.c --- a/src/njs_vmcode.c Thu Mar 14 23:28:03 2024 -0700 +++ b/src/njs_vmcode.c Fri Mar 15 22:47:50 2024 -0700 @@ -2164,8 +2164,10 @@ njs_vmcode_property_foreach(njs_vm_t *vm } next->index = 0; - next->array = njs_value_enumerate(vm, object, NJS_ENUM_KEYS, - NJS_ENUM_STRING, 0); + next->array = njs_value_enumerate(vm, object, + NJS_ENUM_KEYS + | NJS_ENUM_STRING + | NJS_ENUM_ENUMERABLE_ONLY); if (njs_slow_path(next->array == NULL)) { njs_memory_error(vm); return NJS_ERROR; From xeioex at nginx.com Tue Mar 19 05:56:56 2024 From: xeioex at nginx.com (=?utf-8?q?Dmitry_Volyntsev?=) Date: Tue, 19 Mar 2024 05:56:56 +0000 Subject: [njs] Introduced njs_vm_value_enumerate() and njs_vm_value_own_enumerate(). Message-ID: details: https://hg.nginx.org/njs/rev/1aa2bb15c966 branches: changeset: 2300:1aa2bb15c966 user: Dmitry Volyntsev date: Fri Mar 15 23:14:39 2024 -0700 description: Introduced njs_vm_value_enumerate() and njs_vm_value_own_enumerate(). diffstat: src/njs.h | 16 ++++++++++ src/njs_value.h | 12 ------- src/njs_vm.c | 88 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 104 insertions(+), 12 deletions(-) diffs (153 lines): diff -r b0e42db5ca84 -r 1aa2bb15c966 src/njs.h --- a/src/njs.h Fri Mar 15 22:47:50 2024 -0700 +++ b/src/njs.h Fri Mar 15 23:14:39 2024 -0700 @@ -129,6 +129,18 @@ typedef enum { typedef enum { +#define njs_object_enum_kind(flags) (flags & 7) + NJS_ENUM_KEYS = 1, + NJS_ENUM_VALUES = 2, + NJS_ENUM_BOTH = 4, +#define njs_object_enum(flags) (flags & (NJS_ENUM_STRING | NJS_ENUM_SYMBOL)) + NJS_ENUM_STRING = 8, + NJS_ENUM_SYMBOL = 16, + NJS_ENUM_ENUMERABLE_ONLY = 32, +} njs_object_enum_t; + + +typedef enum { /* * Extern property type. */ @@ -497,6 +509,10 @@ NJS_EXPORT njs_int_t njs_vm_object_alloc ...); NJS_EXPORT njs_value_t *njs_vm_object_keys(njs_vm_t *vm, njs_value_t *value, njs_value_t *retval); +NJS_EXPORT njs_value_t *njs_vm_value_enumerate(njs_vm_t *vm, njs_value_t *value, + uint32_t flags, njs_value_t *retval); +NJS_EXPORT njs_value_t *njs_vm_value_own_enumerate(njs_vm_t *vm, + njs_value_t *value, uint32_t flags, njs_value_t *retval); NJS_EXPORT njs_value_t *njs_vm_object_prop(njs_vm_t *vm, njs_value_t *value, const njs_str_t *key, njs_opaque_value_t *retval); NJS_EXPORT njs_int_t njs_vm_object_prop_set(njs_vm_t *vm, njs_value_t *value, diff -r b0e42db5ca84 -r 1aa2bb15c966 src/njs_value.h --- a/src/njs_value.h Fri Mar 15 22:47:50 2024 -0700 +++ b/src/njs_value.h Fri Mar 15 23:14:39 2024 -0700 @@ -313,18 +313,6 @@ struct njs_object_type_init_s { typedef enum { -#define njs_object_enum_kind(flags) (flags & 7) - NJS_ENUM_KEYS = 1, - NJS_ENUM_VALUES = 2, - NJS_ENUM_BOTH = 4, -#define njs_object_enum(flags) (flags & (NJS_ENUM_STRING | NJS_ENUM_SYMBOL)) - NJS_ENUM_STRING = 8, - NJS_ENUM_SYMBOL = 16, - NJS_ENUM_ENUMERABLE_ONLY = 32, -} njs_object_enum_t; - - -typedef enum { NJS_PROPERTY = 0, NJS_ACCESSOR, NJS_PROPERTY_REF, diff -r b0e42db5ca84 -r 1aa2bb15c966 src/njs_vm.c --- a/src/njs_vm.c Fri Mar 15 22:47:50 2024 -0700 +++ b/src/njs_vm.c Fri Mar 15 23:14:39 2024 -0700 @@ -1114,6 +1114,94 @@ njs_vm_exception_string(njs_vm_t *vm, nj } +njs_value_t * +njs_vm_value_enumerate(njs_vm_t *vm, njs_value_t *value, uint32_t flags, + njs_value_t *retval) +{ + ssize_t length; + njs_int_t ret; + njs_value_t *val; + njs_array_t *keys; + njs_rbtree_t *variables; + njs_rbtree_node_t *rb_node; + njs_variable_node_t *node; + const njs_lexer_entry_t *lex_entry; + + static const njs_str_t njs_this_str = njs_str("this"); + + keys = njs_value_enumerate(vm, value, flags); + if (njs_slow_path(keys == NULL)) { + return NULL; + } + + if (!njs_values_same(value, &vm->global_value) + || vm->global_scope == NULL) + { + goto done; + } + + /* TODO: workaround for values in global object. */ + + variables = &vm->global_scope->variables; + rb_node = njs_rbtree_min(variables); + + while (njs_rbtree_is_there_successor(variables, rb_node)) { + node = (njs_variable_node_t *) rb_node; + + lex_entry = njs_lexer_entry(node->variable->unique_id); + if (njs_slow_path(lex_entry == NULL)) { + return NULL; + } + + if (njs_strstr_eq(&lex_entry->name, &njs_this_str)) { + rb_node = njs_rbtree_node_successor(variables, rb_node); + continue; + } + + length = njs_utf8_length(lex_entry->name.start, lex_entry->name.length); + if (njs_slow_path(length < 0)) { + return NULL; + } + + val = njs_array_push(vm, keys); + if (njs_slow_path(value == NULL)) { + return NULL; + } + + ret = njs_string_new(vm, val, lex_entry->name.start, + lex_entry->name.length, length); + if (njs_slow_path(ret != NJS_OK)) { + return NULL; + } + + rb_node = njs_rbtree_node_successor(variables, rb_node); + } + +done: + + njs_set_array(retval, keys); + + return retval; +} + + +njs_value_t * +njs_vm_value_own_enumerate(njs_vm_t *vm, njs_value_t *value, uint32_t flags, + njs_value_t *retval) +{ + njs_array_t *keys; + + keys = njs_value_own_enumerate(vm, value, flags); + if (njs_slow_path(keys == NULL)) { + return NULL; + } + + njs_set_array(retval, keys); + + return retval; +} + + njs_int_t njs_vm_object_alloc(njs_vm_t *vm, njs_value_t *retval, ...) { From xeioex at nginx.com Tue Mar 19 05:56:58 2024 From: xeioex at nginx.com (=?utf-8?q?Dmitry_Volyntsev?=) Date: Tue, 19 Mar 2024 05:56:58 +0000 Subject: [njs] Shell: completions are refactored and moved out of the core. Message-ID: details: https://hg.nginx.org/njs/rev/fe94552843d7 branches: changeset: 2301:fe94552843d7 user: Dmitry Volyntsev date: Mon Mar 18 22:15:48 2024 -0700 description: Shell: completions are refactored and moved out of the core. diffstat: external/njs_shell.c | 275 +++++++++++++++++++++++++++++++------------- src/njs.h | 2 - src/njs_builtin.c | 294 +----------------------------------------------- test/shell_test_njs.exp | 26 +-- 4 files changed, 201 insertions(+), 396 deletions(-) diffs (737 lines): diff -r 1aa2bb15c966 -r fe94552843d7 external/njs_shell.c --- a/external/njs_shell.c Fri Mar 15 23:14:39 2024 -0700 +++ b/external/njs_shell.c Mon Mar 18 22:15:48 2024 -0700 @@ -83,15 +83,7 @@ typedef enum { typedef struct { size_t index; - size_t length; - njs_arr_t *completions; njs_arr_t *suffix_completions; - njs_rbtree_node_t *node; - - enum { - NJS_COMPLETION_SUFFIX = 0, - NJS_COMPLETION_GLOBAL - } phase; } njs_completion_t; @@ -157,7 +149,6 @@ struct njs_engine_s { njs_vm_t *vm; njs_opaque_value_t value; - njs_completion_t completion; } njs; #if (NJS_HAVE_QUICKJS) struct { @@ -174,9 +165,11 @@ struct njs_engine_s { njs_int_t (*process_events)(njs_engine_t *engine); njs_int_t (*destroy)(njs_engine_t *engine); njs_int_t (*output)(njs_engine_t *engine, njs_int_t ret); + njs_arr_t *(*complete)(njs_engine_t *engine, njs_str_t *ex); unsigned type; njs_mp_t *pool; + njs_completion_t completion; }; typedef struct { @@ -1361,11 +1354,6 @@ njs_engine_njs_init(njs_engine_t *engine return NJS_ERROR; } - engine->u.njs.completion.completions = njs_vm_completions(vm, NULL); - if (engine->u.njs.completion.completions == NULL) { - return NJS_ERROR; - } - if (opts->unhandled_rejection) { njs_vm_set_rejection_tracker(vm, njs_rejection_tracker, njs_vm_external_ptr(vm)); @@ -1455,6 +1443,179 @@ njs_engine_njs_output(njs_engine_t *engi } +static njs_arr_t * +njs_object_completions(njs_vm_t *vm, njs_value_t *object, njs_str_t *expression) +{ + u_char *prefix; + size_t len, prefix_len; + int64_t k, n, length; + njs_int_t ret; + njs_arr_t *array; + njs_str_t *completion, key; + njs_value_t *keys; + njs_opaque_value_t *start, retval, prototype; + + prefix = expression->start + expression->length; + + while (prefix > expression->start && *prefix != '.') { + prefix--; + } + + if (prefix != expression->start) { + prefix++; + } + + prefix_len = prefix - expression->start; + len = expression->length - prefix_len; + + array = njs_arr_create(njs_vm_memory_pool(vm), 8, sizeof(njs_str_t)); + if (njs_slow_path(array == NULL)) { + goto fail; + } + + while (!njs_value_is_null(object)) { + keys = njs_vm_value_enumerate(vm, object, NJS_ENUM_KEYS + | NJS_ENUM_STRING, + njs_value_arg(&retval)); + if (njs_slow_path(keys == NULL)) { + goto fail; + } + + (void) njs_vm_array_length(vm, keys, &length); + + start = (njs_opaque_value_t *) njs_vm_array_start(vm, keys); + if (start == NULL) { + goto fail; + } + + + for (n = 0; n < length; n++) { + ret = njs_vm_value_to_string(vm, &key, njs_value_arg(start)); + if (njs_slow_path(ret != NJS_OK)) { + goto fail; + } + + start++; + + if (len > key.length || njs_strncmp(key.start, prefix, len) != 0) { + continue; + } + + for (k = 0; k < array->items; k++) { + completion = njs_arr_item(array, k); + + if ((completion->length - prefix_len - 1) == key.length + && njs_strncmp(&completion->start[prefix_len], + key.start, key.length) + == 0) + { + break; + } + } + + if (k != array->items) { + continue; + } + + completion = njs_arr_add(array); + if (njs_slow_path(completion == NULL)) { + goto fail; + } + + completion->length = prefix_len + key.length + 1; + completion->start = njs_mp_alloc(njs_vm_memory_pool(vm), + completion->length); + if (njs_slow_path(completion->start == NULL)) { + goto fail; + } + + + njs_sprintf(completion->start, + completion->start + completion->length, + "%*s%V%Z", prefix_len, expression->start, &key); + } + + ret = njs_vm_prototype(vm, object, njs_value_arg(&prototype)); + if (njs_slow_path(ret != NJS_OK)) { + goto fail; + } + + object = njs_value_arg(&prototype); + } + + return array; + +fail: + + if (array != NULL) { + njs_arr_destroy(array); + } + + return NULL; +} + + +static njs_arr_t * +njs_engine_njs_complete(njs_engine_t *engine, njs_str_t *expression) +{ + u_char *p, *start, *end; + njs_vm_t *vm; + njs_int_t ret; + njs_bool_t global; + njs_opaque_value_t value, key, retval; + + vm = engine->u.njs.vm; + + p = expression->start; + end = p + expression->length; + + global = 1; + (void) njs_vm_global(vm, njs_value_arg(&value)); + + while (p < end && *p != '.') { p++; } + + if (p == end) { + goto done; + } + + p = expression->start; + + for ( ;; ) { + + start = (*p == '.' && p < end) ? ++p: p; + + if (p == end) { + break; + } + + while (p < end && *p != '.') { p++; } + + ret = njs_vm_value_string_set(vm, njs_value_arg(&key), start, + p - start); + if (njs_slow_path(ret != NJS_OK)) { + return NULL; + } + + ret = njs_value_property(vm, njs_value_arg(&value), njs_value_arg(&key), + njs_value_arg(&retval)); + if (njs_slow_path(ret != NJS_OK)) { + if (ret == NJS_DECLINED && !global) { + goto done; + } + + return NULL; + } + + global = 0; + njs_value_assign(&value, &retval); + } + +done: + + return njs_object_completions(vm, njs_value_arg(&value), expression); +} + + static njs_int_t njs_engine_njs_process_events(njs_engine_t *engine) { @@ -2984,6 +3145,7 @@ njs_create_engine(njs_opts_t *opts) engine->process_events = njs_engine_njs_process_events; engine->destroy = njs_engine_njs_destroy; engine->output = njs_engine_njs_output; + engine->complete = njs_engine_njs_complete; break; #ifdef NJS_HAVE_QUICKJS @@ -3413,91 +3575,38 @@ njs_editline_init(void) } -/* editline frees the buffer every time. */ -#define njs_editline(s) strndup((char *) (s)->start, (s)->length) - -#define njs_completion(c, i) &(((njs_str_t *) (c)->start)[i]) - -#define njs_next_phase(c) \ - (c)->index = 0; \ - (c)->phase++; \ - goto next; - static char * njs_completion_generator(const char *text, int state) { njs_str_t expression, *suffix; - njs_vm_t *vm; + njs_engine_t *engine; njs_completion_t *cmpl; - if (njs_console.engine == NULL - || njs_console.engine->type != NJS_ENGINE_NJS) - { + engine = njs_console.engine; + if (engine->complete == NULL) { return NULL; } - vm = njs_console.engine->u.njs.vm; - cmpl = &njs_console.engine->u.njs.completion; + cmpl = &engine->completion; if (state == 0) { - cmpl->phase = NJS_COMPLETION_SUFFIX; cmpl->index = 0; - cmpl->length = njs_strlen(text); - cmpl->suffix_completions = NULL; - } - -next: - - switch (cmpl->phase) { - case NJS_COMPLETION_SUFFIX: - if (cmpl->length == 0) { - njs_next_phase(cmpl); - } - + expression.start = (u_char *) text; + expression.length = njs_strlen(text); + + cmpl->suffix_completions = engine->complete(engine, &expression); if (cmpl->suffix_completions == NULL) { - expression.start = (u_char *) text; - expression.length = cmpl->length; - - cmpl->suffix_completions = njs_vm_completions(vm, &expression); - if (cmpl->suffix_completions == NULL) { - njs_next_phase(cmpl); - } - } - - for ( ;; ) { - if (cmpl->index >= cmpl->suffix_completions->items) { - njs_next_phase(cmpl); - } - - suffix = njs_completion(cmpl->suffix_completions, cmpl->index++); - - return njs_editline(suffix); - } - - case NJS_COMPLETION_GLOBAL: - if (cmpl->suffix_completions != NULL) { - /* No global completions if suffixes were found. */ - njs_next_phase(cmpl); - } - - for ( ;; ) { - if (cmpl->index >= cmpl->completions->items) { - break; - } - - suffix = njs_completion(cmpl->completions, cmpl->index++); - - if (suffix->start[0] == '.' || suffix->length < cmpl->length) { - continue; - } - - if (njs_strncmp(text, suffix->start, cmpl->length) == 0) { - return njs_editline(suffix); - } + return NULL; } } - return NULL; + if (cmpl->index == cmpl->suffix_completions->items) { + return NULL; + } + + suffix = njs_arr_item(cmpl->suffix_completions, cmpl->index++); + + return strndup((char *) suffix->start, suffix->length); } #endif diff -r 1aa2bb15c966 -r fe94552843d7 src/njs.h --- a/src/njs.h Fri Mar 15 23:14:39 2024 -0700 +++ b/src/njs.h Mon Mar 18 22:15:48 2024 -0700 @@ -316,8 +316,6 @@ NJS_EXPORT njs_int_t njs_vm_pending(njs_ NJS_EXPORT void njs_vm_set_rejection_tracker(njs_vm_t *vm, njs_rejection_tracker_t rejection_tracker, void *opaque); -NJS_EXPORT void *njs_vm_completions(njs_vm_t *vm, njs_str_t *expression); - /* * Runs the specified function with provided arguments. * NJS_OK successful run. diff -r 1aa2bb15c966 -r fe94552843d7 src/njs_builtin.c --- a/src/njs_builtin.c Fri Mar 15 23:14:39 2024 -0700 +++ b/src/njs_builtin.c Mon Mar 18 22:15:48 2024 -0700 @@ -25,12 +25,7 @@ typedef struct { static njs_int_t njs_global_this_prop_handler(njs_vm_t *vm, njs_object_prop_t *self, njs_value_t *global, njs_value_t *setval, njs_value_t *retval); -static njs_arr_t *njs_vm_expression_completions(njs_vm_t *vm, - njs_str_t *expression); -static njs_arr_t *njs_vm_global_var_completions(njs_vm_t *vm, - njs_str_t *expression); -static njs_arr_t *njs_object_completions(njs_vm_t *vm, njs_value_t *object, - njs_str_t *expression); + static njs_int_t njs_env_hash_init(njs_vm_t *vm, njs_lvlhsh_t *hash, char **environment); @@ -415,293 +410,6 @@ njs_builtin_traverse(njs_vm_t *vm, njs_t } -static njs_arr_t * -njs_builtin_completions(njs_vm_t *vm) -{ - njs_arr_t *array; - njs_str_t *completion; - njs_int_t ret; - njs_lvlhsh_each_t lhe; - njs_builtin_traverse_t ctx; - const njs_object_prop_t *prop; - - array = njs_arr_create(vm->mem_pool, 64, sizeof(njs_str_t)); - if (njs_slow_path(array == NULL)) { - return NULL; - } - - ret = njs_lexer_keywords(array); - if (njs_slow_path(ret != NJS_OK)) { - return NULL; - } - - /* Global object completions. */ - - ctx.type = NJS_BUILTIN_TRAVERSE_KEYS; - njs_lvlhsh_init(&ctx.keys); - - ret = njs_object_traverse(vm, njs_object(&vm->global_value), &ctx, - njs_builtin_traverse); - if (njs_slow_path(ret != NJS_OK)) { - return NULL; - } - - njs_lvlhsh_each_init(&lhe, &njs_object_hash_proto); - - for ( ;; ) { - prop = njs_lvlhsh_each(&ctx.keys, &lhe); - - if (prop == NULL) { - break; - } - - completion = njs_arr_add(array); - if (njs_slow_path(completion == NULL)) { - return NULL; - } - - njs_string_get(&prop->name, completion); - } - - return array; -} - - -void * -njs_vm_completions(njs_vm_t *vm, njs_str_t *expression) -{ - u_char *p, *end; - - if (expression == NULL) { - return njs_builtin_completions(vm); - } - - p = expression->start; - end = p + expression->length; - - while (p < end && *p != '.') { p++; } - - if (p == end) { - return njs_vm_global_var_completions(vm, expression); - } - - return njs_vm_expression_completions(vm, expression); -} - - -static njs_arr_t * -njs_vm_global_var_completions(njs_vm_t *vm, njs_str_t *expression) -{ - njs_str_t *completion; - njs_arr_t *array; - njs_rbtree_t *variables; - njs_rbtree_node_t *node; - njs_variable_node_t *vnode; - const njs_lexer_entry_t *lex_entry; - - variables = (vm->global_scope != NULL) ? &vm->global_scope->variables - : NULL; - if (njs_slow_path(variables == NULL)) { - return NULL; - } - - array = njs_arr_create(vm->mem_pool, 8, sizeof(njs_str_t)); - if (njs_slow_path(array == NULL)) { - return NULL; - } - - node = njs_rbtree_min(variables); - - while (njs_rbtree_is_there_successor(variables, node)) { - vnode = (njs_variable_node_t *) node; - - node = njs_rbtree_node_successor(variables, node); - - lex_entry = njs_lexer_entry(vnode->key); - if (lex_entry == NULL) { - continue; - } - - if (lex_entry->name.length >= expression->length - && njs_strncmp(expression->start, lex_entry->name.start, - expression->length) == 0) - { - completion = njs_arr_add(array); - if (njs_slow_path(completion == NULL)) { - return NULL; - } - - *completion = lex_entry->name; - } - } - - return array; -} - - -static njs_arr_t * -njs_vm_expression_completions(njs_vm_t *vm, njs_str_t *expression) -{ - u_char *p, *end; - njs_int_t ret; - njs_value_t *value; - njs_variable_t *var; - njs_rbtree_node_t *node; - njs_object_prop_t *prop; - njs_lvlhsh_query_t lhq; - njs_variable_node_t var_node; - - p = expression->start; - end = p + expression->length; - - lhq.key.start = p; - - while (p < end && *p != '.') { p++; } - - lhq.proto = &njs_lexer_hash_proto; - lhq.key.length = p - lhq.key.start; - lhq.key_hash = njs_djb_hash(lhq.key.start, lhq.key.length); - - ret = njs_lvlhsh_find(&vm->shared->keywords_hash, &lhq); - if (njs_slow_path(ret != NJS_OK)) { - return NULL; - } - - var_node.key = (uintptr_t) lhq.value; - - node = njs_rbtree_find(&vm->global_scope->variables, &var_node.node); - if (njs_slow_path(node == NULL)) { - return NULL; - } - - var = ((njs_variable_node_t *) node)->variable; - value = njs_scope_value(vm, var->index); - - if (!njs_is_object(value)) { - return NULL; - } - - lhq.proto = &njs_object_hash_proto; - - for ( ;; ) { - - if (p == end) { - break; - } - - lhq.key.start = ++p; - - while (p < end && *p != '.') { p++; } - - lhq.key.length = p - lhq.key.start; - lhq.key_hash = njs_djb_hash(lhq.key.start, lhq.key.length); - - ret = njs_lvlhsh_find(njs_object_hash(value), &lhq); - if (njs_slow_path(ret != NJS_OK)) { - if (ret == NJS_DECLINED) { - break; - } - - return NULL; - } - - prop = lhq.value; - - if (njs_is_accessor_descriptor(prop) || - !njs_is_object(njs_prop_value(prop))) - { - return NULL; - } - - value = njs_prop_value(prop); - } - - return njs_object_completions(vm, value, expression); -} - - -static njs_arr_t * -njs_object_completions(njs_vm_t *vm, njs_value_t *object, njs_str_t *expression) -{ - u_char *prefix; - double num; - size_t len; - njs_arr_t *array; - njs_str_t *completion, key; - njs_uint_t n; - njs_array_t *keys; - njs_value_type_t type; - - prefix = expression->start + expression->length; - - while (prefix > expression->start && *prefix != '.') { - prefix--; - } - - prefix++; - len = expression->length - (prefix - expression->start); - - array = NULL; - type = object->type; - - if (type == NJS_ARRAY || type == NJS_TYPED_ARRAY) { - object->type = NJS_OBJECT; - } - - keys = njs_value_enumerate(vm, object, NJS_ENUM_KEYS | NJS_ENUM_STRING); - if (njs_slow_path(keys == NULL)) { - goto done; - } - - array = njs_arr_create(vm->mem_pool, 8, sizeof(njs_str_t)); - if (njs_slow_path(array == NULL)) { - goto done; - } - - for (n = 0; n < keys->length; n++) { - njs_string_get(&keys->start[n], &key); - - if (len != 0 - && njs_strncmp(key.start, prefix, njs_min(len, key.length)) != 0) - { - continue; - } - - num = njs_key_to_index(&keys->start[n]); - - if (!njs_key_is_integer_index(num, &keys->start[n])) { - completion = njs_arr_add(array); - if (njs_slow_path(completion == NULL)) { - njs_arr_destroy(array); - array = NULL; - goto done; - } - - completion->length = (prefix - expression->start) + key.length + 1; - completion->start = njs_mp_alloc(vm->mem_pool, completion->length); - if (njs_slow_path(completion->start == NULL)) { - njs_arr_destroy(array); - array = NULL; - goto done; - } - - njs_sprintf(completion->start, - completion->start + completion->length, - "%*s%V%Z", prefix - expression->start, - expression->start, &key); - } - } - -done: - - if (type == NJS_ARRAY || type == NJS_TYPED_ARRAY) { - object->type = type; - } - - return array; -} - - typedef struct { njs_str_t name; njs_function_native_t native; diff -r 1aa2bb15c966 -r fe94552843d7 test/shell_test_njs.exp --- a/test/shell_test_njs.exp Fri Mar 15 23:14:39 2024 -0700 +++ b/test/shell_test_njs.exp Mon Mar 18 22:15:48 2024 -0700 @@ -46,23 +46,15 @@ njs_test { "a *= 2\r\n2\r\n>> "} } -# Global completions, no -# Disabled: readline does not support it in callback mode -# njs_test { -# {"\t\tn" -# "\a\r\nDisplay all*possibilities? (y or n)*>> "} -# } +# Global completions -# Global completions, yes njs_test { - {"\t\ty" - "\a\r\nDisplay all*possibilities? (y or n)*Array"} + {"\t\t" + "$262*"} } # Global completions, single partial match -# \a* is WORKAROUND for libedit-20170329.3.1-r3 -# which inserts '\rESC[6G' after '\a'. njs_test { {"O\t" "O\a*bject"} @@ -87,13 +79,13 @@ njs_test { njs_test { {"Type\t" "Type\a*Error"} - {"\t\t" - "TypeError.length"} + {".\t\t" + "TypeError.__proto__"} } njs_test { {"TypeError.\t\t" - "TypeError.length*TypeError.prototype"} + "TypeError.__proto__*TypeError.prototype"} } njs_test { @@ -106,8 +98,8 @@ njs_test { njs_test { {"JS\t" "JS\a*ON"} - {"\t\t" - "JSON.parse*JSON.stringify"} + {".\t\t" + "JSON.__proto__*JSON.stringify"} } # Global completions, no matches @@ -134,8 +126,6 @@ njs_test { "AA*AAA*"} } -# z*z is WORKAROUND for libedit-20170329.3.1-r3 -# which inserts bogus '\a' between 'z' njs_test { {"var zz = 1\r\n" "var zz = 1\r\nundefined\r\n>> "} From xeioex at nginx.com Tue Mar 19 05:57:00 2024 From: xeioex at nginx.com (=?utf-8?q?Dmitry_Volyntsev?=) Date: Tue, 19 Mar 2024 05:57:00 +0000 Subject: [njs] Shell: added completions for QuickJS engine. Message-ID: details: https://hg.nginx.org/njs/rev/85313a068147 branches: changeset: 2302:85313a068147 user: Dmitry Volyntsev date: Mon Mar 18 22:24:57 2024 -0700 description: Shell: added completions for QuickJS engine. diffstat: external/njs_shell.c | 202 +++++++++++++++++++++++++++++++++++++++++++++++- test/shell_test.exp | 147 ++++++++++++++++++++++++++++++++++ test/shell_test_njs.exp | 148 ----------------------------------- 3 files changed, 345 insertions(+), 152 deletions(-) diffs (541 lines): diff -r fe94552843d7 -r 85313a068147 external/njs_shell.c --- a/external/njs_shell.c Mon Mar 18 22:15:48 2024 -0700 +++ b/external/njs_shell.c Mon Mar 18 22:24:57 2024 -0700 @@ -3106,6 +3106,203 @@ njs_engine_qjs_output(njs_engine_t *engi return NJS_OK; } + +static njs_arr_t * +njs_qjs_object_completions(njs_engine_t *engine, JSContext *ctx, + JSValueConst object, njs_str_t *expression) +{ + u_char *prefix; + size_t len, prefix_len; + JSValue prototype; + uint32_t k, n, length; + njs_int_t ret; + njs_arr_t *array; + njs_str_t *completion, key; + JSPropertyEnum *ptab; + + prefix = expression->start + expression->length; + + while (prefix > expression->start && *prefix != '.') { + prefix--; + } + + if (prefix != expression->start) { + prefix++; + } + + ptab = NULL; + key.start = NULL; + prefix_len = prefix - expression->start; + len = expression->length - prefix_len; + + array = njs_arr_create(engine->pool, 8, sizeof(njs_str_t)); + if (njs_slow_path(array == NULL)) { + goto fail; + } + + while (!JS_IsNull(object)) { + ret = JS_GetOwnPropertyNames(ctx, &ptab, &length, object, + JS_GPN_STRING_MASK); + if (ret < 0) { + goto fail; + } + + for (n = 0; n < length; n++) { + key.start = (u_char *) JS_AtomToCString(ctx, ptab[n].atom); + JS_FreeAtom(ctx, ptab[n].atom); + if (njs_slow_path(key.start == NULL)) { + goto fail; + } + + key.length = njs_strlen(key.start); + + if (len > key.length || njs_strncmp(key.start, prefix, len) != 0) { + goto next; + } + + for (k = 0; k < array->items; k++) { + completion = njs_arr_item(array, k); + + if ((completion->length - prefix_len - 1) == key.length + && njs_strncmp(&completion->start[prefix_len], + key.start, key.length) + == 0) + { + goto next; + } + } + + completion = njs_arr_add(array); + if (njs_slow_path(completion == NULL)) { + goto fail; + } + + completion->length = prefix_len + key.length + 1; + completion->start = njs_mp_alloc(engine->pool, completion->length); + if (njs_slow_path(completion->start == NULL)) { + goto fail; + } + + njs_sprintf(completion->start, + completion->start + completion->length, + "%*s%V%Z", prefix_len, expression->start, &key); + +next: + + JS_FreeCString(ctx, (const char *) key.start); + } + + js_free_rt(JS_GetRuntime(ctx), ptab); + + prototype = JS_GetPrototype(ctx, object); + if (JS_IsException(prototype)) { + goto fail; + } + + JS_FreeValue(ctx, object); + object = prototype; + } + + return array; + +fail: + + if (array != NULL) { + njs_arr_destroy(array); + } + + if (key.start != NULL) { + JS_FreeCString(ctx, (const char *) key.start); + } + + if (ptab != NULL) { + js_free_rt(JS_GetRuntime(ctx), ptab); + } + + JS_FreeValue(ctx, object); + + return NULL; +} + + +static njs_arr_t * +njs_engine_qjs_complete(njs_engine_t *engine, njs_str_t *expression) +{ + u_char *p, *start, *end; + JSAtom key; + JSValue value, retval; + njs_arr_t *arr; + JSContext *ctx; + njs_bool_t global; + + ctx = engine->u.qjs.ctx; + + p = expression->start; + end = p + expression->length; + + global = 1; + value = JS_GetGlobalObject(ctx); + + while (p < end && *p != '.') { p++; } + + if (p == end) { + goto done; + } + + p = expression->start; + + for ( ;; ) { + + start = (*p == '.' && p < end) ? ++p: p; + + if (p == end) { + break; + } + + while (p < end && *p != '.') { p++; } + + key = JS_NewAtomLen(ctx, (char *) start, p - start); + if (key == JS_ATOM_NULL) { + goto fail; + } + + retval = JS_GetProperty(ctx, value, key); + + JS_FreeAtom(ctx, key); + + if (JS_IsUndefined(retval)) { + if (global) { + goto fail; + } + + goto done; + } + + if (JS_IsException(retval)) { + goto fail; + } + + JS_FreeValue(ctx, value); + value = retval; + global = 0; + } + +done: + + arr = njs_qjs_object_completions(engine, ctx, JS_DupValue(ctx, value), + expression); + + JS_FreeValue(ctx, value); + + return arr; + +fail: + + JS_FreeValue(ctx, value); + + return NULL; +} + #endif @@ -3163,6 +3360,7 @@ njs_create_engine(njs_opts_t *opts) engine->process_events = njs_engine_qjs_process_events; engine->destroy = njs_engine_qjs_destroy; engine->output = njs_engine_qjs_output; + engine->complete = njs_engine_qjs_complete; break; #endif @@ -3583,10 +3781,6 @@ njs_completion_generator(const char *tex njs_completion_t *cmpl; engine = njs_console.engine; - if (engine->complete == NULL) { - return NULL; - } - cmpl = &engine->completion; if (state == 0) { diff -r fe94552843d7 -r 85313a068147 test/shell_test.exp --- a/test/shell_test.exp Mon Mar 18 22:15:48 2024 -0700 +++ b/test/shell_test.exp Mon Mar 18 22:24:57 2024 -0700 @@ -58,6 +58,153 @@ njs_test { "2\r\n>> "} } +# Global completions + +njs_test { + {"\t\t" + "$262*"} +} + +# Global completions, single partial match + +njs_test { + {"O\t" + "O\a*bject"} +} + +njs_test { + {"Mat\t" + "Mat\a*h"} +} + + njs_test { + {"conso\t" + "conso\a*le"} + } + +# Global completions, multiple partial match +njs_test { + {"cons\t\t" + "console*const"} +} + +njs_test { + {"Type\t" + "Type\a*Error"} + {".\t\t" + "TypeError.__proto__"} +} + +njs_test { + {"TypeError.\t\t" + "TypeError.__proto__*TypeError.prototype"} +} + +njs_test { + {"Object.ge\t" + "Object.ge\a*t"} + {"\t\t" + "Object.getOwnPropertyDescriptor*Object.getPrototypeOf"} +} + +njs_test { + {"JS\t" + "JS\a*ON"} + {".\t\t" + "JSON.__*JSON.stringify"} +} + +njs_test { + {"decodeURI\t\t" + "decodeURI*decodeURIComponent"} + {"C\t" + "decodeURIC\a*omponent"} +} + +# Global completions, no matches +njs_test { + {"1.\t\t" + "1."} +} + +njs_test { + {"1..\t\t" + "1.."} +} + +njs_test { + {"'abc'.\t\t" + "'abc'."} +} + +# Global completions, global vars +njs_test { + {"var AA = 1; var AAA = 2\r\n" + "var AA = 1; var AAA = 2\r\nundefined\r\n>> "} + {"AA\t\t" + "AA*AAA*"} +} + +njs_test { + {"var zz = 1\r\n" + "var zz = 1\r\nundefined\r\n>> "} + {"1 + z\t\r\n" + "1 + z*z*\r\n2"} +} + +njs_test { + {"unknown_var\t\t" + "unknown_var"} +} + +njs_test { + {"unknown_var.\t\t" + "unknown_var."} +} + +# An object's level completions +njs_test { + {"var o = {zz:1, zb:2}\r\n" + "var o = {zz:1, zb:2}\r\nundefined\r\n>> "} + {"o.z\t\t" + "o.zb*o.zz"} +} + +njs_test { + {"var d = new Date()\r\n" + "var d = new Date()\r\nundefined\r\n>> "} + {"d.to\t\t" + "d.toDateString*d.toLocaleDateString*d.toString"} +} + +njs_test { + {"var o = {a:new Date()}\r\n" + "var o = {a:new Date()}\r\nundefined\r\n>> "} + {"o.a.to\t\t" + "o.a.toDateString*o.a.toLocaleDateString*o.a.toString"} +} + +njs_test { + {"var o = {a:1,b:2,333:'t'}\r\n" + "var o = {a:1,b:2,333:'t'}\r\nundefined\r\n>> "} + {"o.3\t\t" + "o.3"} +} + +njs_test { + {"var a = Array(5000000); a.aab = 1; a.aac = 2\r\n" + "var a = Array(5000000); a.aab = 1; a.aac = 2\r\n2\r\n>> "} + {"a.\t\t" + "a.aab*"} +} + +njs_test { + {"var a = new Uint8Array([5,6,7,8,8]); a.aab = 1; a.aac = 2\r\n" + "var a = new Uint8Array(\\\[5,6,7,8,8]); a.aab = 1; a.aac = 2\r\n2\r\n>> "} + {"a.\t\t" + "a.aab*"} +} + # console object njs_test { {"console[Symbol.toStringTag]\r\n" diff -r fe94552843d7 -r 85313a068147 test/shell_test_njs.exp --- a/test/shell_test_njs.exp Mon Mar 18 22:15:48 2024 -0700 +++ b/test/shell_test_njs.exp Mon Mar 18 22:24:57 2024 -0700 @@ -38,154 +38,6 @@ njs_test { "njs.version\r\n\*\.\*\.\*"} } -# simple multi line interaction -njs_test { - {"var a = 1\r\n" - "var a = 1\r\nundefined\r\n>> "} - {"a *= 2\r\n" - "a *= 2\r\n2\r\n>> "} -} - -# Global completions - -njs_test { - {"\t\t" - "$262*"} -} - -# Global completions, single partial match - -njs_test { - {"O\t" - "O\a*bject"} -} - -njs_test { - {"Ma\t" - "Ma\a*th"} -} - - njs_test { - {"conso\t" - "conso\a*le"} - } - -# Global completions, multiple partial match -njs_test { - {"cons\t\t" - "console*const"} -} - -njs_test { - {"Type\t" - "Type\a*Error"} - {".\t\t" - "TypeError.__proto__"} -} - -njs_test { - {"TypeError.\t\t" - "TypeError.__proto__*TypeError.prototype"} -} - -njs_test { - {"Object.g\t" - "Object.g\a*et"} - {"\t\t" - "Object.getOwnPropertyDescriptor*Object.getPrototypeOf"} -} - -njs_test { - {"JS\t" - "JS\a*ON"} - {".\t\t" - "JSON.__proto__*JSON.stringify"} -} - -# Global completions, no matches -njs_test { - {"1.\t\t" - "1."} -} - -njs_test { - {"1..\t\t" - "1.."} -} - -njs_test { - {"'abc'.\t\t" - "'abc'."} -} - -# Global completions, global vars -njs_test { - {"var AA = 1; var AAA = 2\r\n" - "var AA = 1; var AAA = 2\r\nundefined\r\n>> "} - {"AA\t\t" - "AA*AAA*"} -} - -njs_test { - {"var zz = 1\r\n" - "var zz = 1\r\nundefined\r\n>> "} - {"1 + z\t\r\n" - "1 + z*z*\r\n2"} -} - -njs_test { - {"unknown_var\t\t" - "unknown_var"} -} - -njs_test { - {"unknown_var.\t\t" - "unknown_var."} -} - -# An object's level completions -njs_test { - {"var o = {zz:1, zb:2}\r\n" - "var o = {zz:1, zb:2}\r\nundefined\r\n>> "} - {"o.z\t\t" - "o.zb*o.zz"} -} - -njs_test { - {"var d = new Date()\r\n" - "var d = new Date()\r\nundefined\r\n>> "} - {"d.to\t\t" - "d.toDateString*d.toLocaleDateString*d.toString"} -} - -njs_test { - {"var o = {a:new Date()}\r\n" - "var o = {a:new Date()}\r\nundefined\r\n>> "} - {"o.a.to\t\t" - "o.a.toDateString*o.a.toLocaleDateString*o.a.toString"} -} - -njs_test { - {"var o = {a:1,b:2,333:'t'}\r\n" - "var o = {a:1,b:2,333:'t'}\r\nundefined\r\n>> "} - {"o.3\t\t" - "o.3"} -} - -njs_test { - {"var a = Array(5000000); a.aab = 1; a.aac = 2\r\n" - "var a = Array(5000000); a.aab = 1; a.aac = 2\r\n2\r\n>> "} - {"a.\t\t" - "a.aab*"} -} - -njs_test { - {"var a = new Uint8Array([5,6,7,8,8]); a.aab = 1; a.aac = 2\r\n" - "var a = new Uint8Array(\\\[5,6,7,8,8]); a.aab = 1; a.aac = 2\r\n2\r\n>> "} - {"a.\t\t" - "a.aab*"} -} - # console dump njs_test { {"console.dump()\r\n" From pluknet at nginx.com Wed Mar 20 12:13:47 2024 From: pluknet at nginx.com (Sergey Kandaurov) Date: Wed, 20 Mar 2024 16:13:47 +0400 Subject: [PATCH] macOS: detect cache line size at runtime In-Reply-To: References: Message-ID: On Wed, Feb 28, 2024 at 01:24:06AM +0000, Piotr Sikora via nginx-devel wrote: > # HG changeset patch > # User Piotr Sikora > # Date 1708977640 0 > # Mon Feb 26 20:00:40 2024 +0000 > # Branch patch015 > # Node ID f58bc1041ebca635517b919d58b49923bf24f76d > # Parent 570e97dddeeddb79c71587aa8a10150b64404beb Hello, Note that a compile time cache line size was previously improved in 8010:35afae4b3dff, to have a reasonable default of 64. > macOS: detect cache line size at runtime. I prefer not to introduce more ad-hoc prefixes in the log summary. Something like moving the "macOS" part to the end should be fine. > > Notably, Apple Silicon CPUs have 128 byte cache line size, > which is twice the default configured for generic aarch64. > > Signed-off-by: Piotr Sikora > > diff -r 570e97dddeed -r f58bc1041ebc src/os/unix/ngx_darwin_init.c > --- a/src/os/unix/ngx_darwin_init.c Mon Feb 26 20:00:38 2024 +0000 > +++ b/src/os/unix/ngx_darwin_init.c Mon Feb 26 20:00:40 2024 +0000 > @@ -11,6 +11,7 @@ > > char ngx_darwin_kern_ostype[16]; > char ngx_darwin_kern_osrelease[128]; > +int64_t ngx_darwin_hw_cachelinesize; > int ngx_darwin_hw_ncpu; > int ngx_darwin_kern_ipc_somaxconn; > u_long ngx_darwin_net_inet_tcp_sendspace; style: this breaks a perfect indentation of two spaces after type; further, it appears to be unsorted by type; I'd put it after u_long Regarding ngx_darwin_net_inet_tcp_sendspace, which is u_long here, it appears to be inherited from ngx_freebsd_init.c unmodified in 0.7.7. However, tcp_sendspace is sysctl-exported using integer OID format in both XNU and FreeBSD modern versions. Restoring its type there, back to before 673:b80f94fa2197, might further affect the sorting. As originally introduced in the BSD kernel, tcp_sendspace was int. It was changed to u_long in 4.3BSD-Tahoe by Mike Karels (together with soreserve), though with introduction of the (old style) sysctl interface in FreeBSD 2.0.5, it started to be exported as type int (with appropriate XXX). It continued to be exported as int using the new style, until its OID format was change to SYSCTL_ULONG in FreeBSD 7.0 "to match the type of the variable they are exporting". Apparently, this was the reason of updating ngx_freebsd_net_inet_tcp_sendspace to u_long in nginx 0.3.58 (further unsorting declarations with regard to sysctls[] order): | *) Bugfix: nginx could not run on 64-bit FreeBSD 7.0-CURRENT. Later, the OID format was changed back to int in FreeBSD 9.2, as part of the sysctl virtualization work, with the following reason: |A long is not necessary as the TCP window is limited to 2**30 but ngx_freebsd_net_inet_tcp_sendspace remained u_long, apparently to keep nginx running on older FreeBSD version. In XNU, tcp_sendspace interface remained the same as imported from around FreeBSD 4.x, until in xnu-1456.1.26 (Snow Leopard), its type was corrected to u_int32_t, and its sysctl OID format was changed to "unsigned int". Anyway, ngx_darwin_net_inet_tcp_sendspace is not currently used in the code, we can leave it for now. See the relevant discussion on incomplete work here: https://mailman.nginx.org/pipermail/nginx-devel/2023-May/PSFQLA6TJSL7B2RKJWL6ORA54IOPO5SM.html > @@ -44,6 +45,10 @@ > > > sysctl_t sysctls[] = { > + { "hw.cachelinesize", > + &ngx_darwin_hw_cachelinesize, > + sizeof(ngx_darwin_hw_cachelinesize), 0 }, > + JFTR, this is well sorted regarding the order of declarations, but it will need to be adjusted after fixing the sorting above. > { "hw.ncpu", > &ngx_darwin_hw_ncpu, > sizeof(ngx_darwin_hw_ncpu), 0 }, > @@ -155,6 +160,7 @@ > return NGX_ERROR; > } > > + ngx_cacheline_size = ngx_darwin_hw_cachelinesize; > ngx_ncpu = ngx_darwin_hw_ncpu; > > if (ngx_darwin_kern_ipc_somaxconn > 32767) { > diff -r 570e97dddeed -r f58bc1041ebc src/os/unix/ngx_posix_init.c > --- a/src/os/unix/ngx_posix_init.c Mon Feb 26 20:00:38 2024 +0000 > +++ b/src/os/unix/ngx_posix_init.c Mon Feb 26 20:00:40 2024 +0000 > @@ -51,7 +51,10 @@ > } > > ngx_pagesize = getpagesize(); > - ngx_cacheline_size = NGX_CPU_CACHE_LINE; > + > + if (ngx_cacheline_size == 0) { > + ngx_cacheline_size = NGX_CPU_CACHE_LINE; > + } > > for (n = ngx_pagesize; n >>= 1; ngx_pagesize_shift++) { /* void */ } > This makes the following slight update to the patch. If you're okey with it, I will commit it then. # HG changeset patch # User Piotr Sikora # Date 1710886608 -14400 # Wed Mar 20 02:16:48 2024 +0400 # Node ID d6fc8103a60d5fe2ca291a3ee943a93725df420c # Parent e773dcc47d1004babe8383b73c0731a734b70ca4 Detect cache line size at runtime on macOS. Notably, Apple Silicon CPUs have 128 byte cache line size, which is twice the default configured for generic aarch64. Signed-off-by: Piotr Sikora diff --git a/src/os/unix/ngx_darwin_init.c b/src/os/unix/ngx_darwin_init.c --- a/src/os/unix/ngx_darwin_init.c +++ b/src/os/unix/ngx_darwin_init.c @@ -9,11 +9,12 @@ #include -char ngx_darwin_kern_ostype[16]; -char ngx_darwin_kern_osrelease[128]; -int ngx_darwin_hw_ncpu; -int ngx_darwin_kern_ipc_somaxconn; -u_long ngx_darwin_net_inet_tcp_sendspace; +char ngx_darwin_kern_ostype[16]; +char ngx_darwin_kern_osrelease[128]; +int ngx_darwin_hw_ncpu; +int ngx_darwin_kern_ipc_somaxconn; +u_long ngx_darwin_net_inet_tcp_sendspace; +int64_t ngx_darwin_hw_cachelinesize; ngx_uint_t ngx_debug_malloc; @@ -56,6 +57,10 @@ sysctl_t sysctls[] = { &ngx_darwin_kern_ipc_somaxconn, sizeof(ngx_darwin_kern_ipc_somaxconn), 0 }, + { "hw.cachelinesize", + &ngx_darwin_hw_cachelinesize, + sizeof(ngx_darwin_hw_cachelinesize), 0 }, + { NULL, NULL, 0, 0 } }; @@ -155,6 +160,7 @@ ngx_os_specific_init(ngx_log_t *log) return NGX_ERROR; } + ngx_cacheline_size = ngx_darwin_hw_cachelinesize; ngx_ncpu = ngx_darwin_hw_ncpu; if (ngx_darwin_kern_ipc_somaxconn > 32767) { diff --git a/src/os/unix/ngx_posix_init.c b/src/os/unix/ngx_posix_init.c --- a/src/os/unix/ngx_posix_init.c +++ b/src/os/unix/ngx_posix_init.c @@ -51,7 +51,10 @@ ngx_os_init(ngx_log_t *log) } ngx_pagesize = getpagesize(); - ngx_cacheline_size = NGX_CPU_CACHE_LINE; + + if (ngx_cacheline_size == 0) { + ngx_cacheline_size = NGX_CPU_CACHE_LINE; + } for (n = ngx_pagesize; n >>= 1; ngx_pagesize_shift++) { /* void */ } From pluknet at nginx.com Wed Mar 20 14:57:14 2024 From: pluknet at nginx.com (Sergey Kandaurov) Date: Wed, 20 Mar 2024 18:57:14 +0400 Subject: [PATCH] Configure: link libcrypt when a feature using it is detected In-Reply-To: <570e97dddeeddb79c715.1709083399@ip-172-31-36-66.ec2.internal> References: <570e97dddeeddb79c715.1709083399@ip-172-31-36-66.ec2.internal> Message-ID: <67E7F428-19EB-4370-B117-78FF9D6BEC18@nginx.com> > On 28 Feb 2024, at 05:23, Piotr Sikora via nginx-devel wrote: > > # HG changeset patch > # User Piotr Sikora > # Date 1708977638 0 > # Mon Feb 26 20:00:38 2024 +0000 > # Branch patch014 > # Node ID 570e97dddeeddb79c71587aa8a10150b64404beb > # Parent cdc173477ea99fd6c952a85e5cd11db66452076a > Configure: link libcrypt when a feature using it is detected. > > Previously, this worked only because libcrypt was added in a separate > test for crypt() in auto/unix. Hello, The crypt_r() test was added specifically for a missing crypt_r() on uclibc, which is rare these days: uclibc has just crypt() in libcrypt. Anyway, crypt_r() is expected to be in the same library as crypt(). That's how it worked in practice: testing crypt_r() uses libcrypt to pass in all cases, and testing crypt() adds actual -lcrypt as needed. Currently, it may fail rather theoretically: e.g. crypt_r() is present in libcrypt, but crypt() is not (it might be in libc, instead). In this case, given the crypt_r() precedence, linking nginx would fail due to a missing linking with libcrypt required for crypt_r(). Aside from uclibc, I think the patch is good to improve consistency. I'd rewrote the description to be more specific: : Configure: fixed Linux crypt_r() test to add libcrypt. : : Previously, the resulting binary was successfully linked : because libcrypt was added in a separate test for crypt(). > > Signed-off-by: Piotr Sikora > > diff -r cdc173477ea9 -r 570e97dddeed auto/os/linux > --- a/auto/os/linux Mon Feb 26 20:00:37 2024 +0000 > +++ b/auto/os/linux Mon Feb 26 20:00:38 2024 +0000 > @@ -228,6 +228,10 @@ > crypt_r(\"key\", \"salt\", &cd);" > . auto/feature > > +if [ $ngx_found = yes ]; then > + CRYPT_LIB="-lcrypt" > +fi > + > > ngx_include="sys/vfs.h"; . auto/include > -- Sergey Kandaurov From xeioex at nginx.com Thu Mar 21 01:29:06 2024 From: xeioex at nginx.com (=?utf-8?q?Dmitry_Volyntsev?=) Date: Thu, 21 Mar 2024 01:29:06 +0000 Subject: [njs] Modules: fixed clear() method of a shared dictionary without timeout. Message-ID: details: https://hg.nginx.org/njs/rev/f632fe16ba05 branches: changeset: 2303:f632fe16ba05 user: Dmitry Volyntsev date: Tue Mar 19 21:05:51 2024 -0700 description: Modules: fixed clear() method of a shared dictionary without timeout. Previously, the code did not unlock the rwlock when a dict was empty. The issue was introduced in 4a15613f4e8b (0.8.3). This closes issue #699 on Github. diffstat: nginx/ngx_js_shared_dict.c | 4 +++- nginx/t/js_shared_dict.t | 1 + 2 files changed, 4 insertions(+), 1 deletions(-) diffs (32 lines): diff -r 85313a068147 -r f632fe16ba05 nginx/ngx_js_shared_dict.c --- a/nginx/ngx_js_shared_dict.c Mon Mar 18 22:24:57 2024 -0700 +++ b/nginx/ngx_js_shared_dict.c Tue Mar 19 21:05:51 2024 -0700 @@ -479,7 +479,7 @@ njs_js_ext_shared_dict_clear(njs_vm_t *v rbtree = &dict->sh->rbtree; if (rbtree->root == rbtree->sentinel) { - return NJS_OK; + goto done; } for (rn = ngx_rbtree_min(rbtree->root, rbtree->sentinel); @@ -494,6 +494,8 @@ njs_js_ext_shared_dict_clear(njs_vm_t *v } } +done: + ngx_rwlock_unlock(&dict->sh->rwlock); njs_value_undefined_set(retval); diff -r 85313a068147 -r f632fe16ba05 nginx/t/js_shared_dict.t --- a/nginx/t/js_shared_dict.t Mon Mar 18 22:24:57 2024 -0700 +++ b/nginx/t/js_shared_dict.t Tue Mar 19 21:05:51 2024 -0700 @@ -266,6 +266,7 @@ EOF function set_clear(r) { var dict = ngx.shared.no_timeout; + dict.clear(); dict.set("test", "test value"); dict.set("test1", "test1 value"); dict.clear(); From piotr at aviatrix.com Thu Mar 21 06:42:49 2024 From: piotr at aviatrix.com (Piotr Sikora) Date: Thu, 21 Mar 2024 06:42:49 +0000 Subject: [PATCH] Core: free connections and read/write events at shutdown In-Reply-To: References: <59780B99-BFAD-4DD6-8887-3C87DDC53AC6@nginx.com> Message-ID: Hi Sergey, > While I agree that false positives do not allow to run LeakSanitizer > in a clean fashion, I don't think it is nginx which should be fixed. > Rather, sanitizer analysis could be improved instead to prevent FPs. Patches welcome? > Meanwhile, leak sanitizer can be used with suppressions as appropriate > to run tests cleanly. For example, this allows to suppress memory leak > reports for allocations made during worker process init, such as cycle > connections and read/write events: > > $ cat suppr.txt > leak:ngx_worker_process_init > $ LSAN_OPTIONS=suppressions=suppr.txt prove -r Right, but that's an unnecessary step that prevents potential contributors from using LeakSanitizer with NGINX. Also, I don't think that you're using those tools either, likely because of those few false positives. Best regards, Piotr Sikora From piotr at aviatrix.com Thu Mar 21 06:44:58 2024 From: piotr at aviatrix.com (Piotr Sikora) Date: Thu, 21 Mar 2024 06:44:58 +0000 Subject: [PATCH] Geo: fix uninitialized memory access In-Reply-To: <20240314155515.f3c6bttkxzoo2unz@N00W24XTQX> References: <20240314155515.f3c6bttkxzoo2unz@N00W24XTQX> Message-ID: Hi Roman, > Thanks for the patch, looks valid, except we no longer need to explicitly > initialize fields to zero. Right, I was going back-and-forth between which version I should send. > Also, I think we need more details about the > uninitialized memory access. See updated patch. LGTM, thanks! Best regards, Piotr Sikora From piotr at aviatrix.com Thu Mar 21 06:53:54 2024 From: piotr at aviatrix.com (Piotr Sikora) Date: Thu, 21 Mar 2024 06:53:54 +0000 Subject: [PATCH] Core: fix conversion of IPv4-mapped IPv6 addresses In-Reply-To: References: <5584232259d28489efba.1709083309@ip-172-31-36-66.ec2.internal> Message-ID: Hi Sergey, > The "shift" remark doesn't describe a problem in details. It's not a remark, it's the name of the UndefinedBehaviorSanitizer check that caught the issue [1]. > @@ -507,7 +507,7 @@ ngx_cidr_match(struct sockaddr *sa, ngx_ > > p = inaddr6->s6_addr; > > - inaddr = p[12] << 24; > + inaddr = (in_addr_t) p[12] << 24; > inaddr += p[13] << 16; > inaddr += p[14] << 8; > inaddr += p[15]; While this minimizes the diff and silences the error at hand, I find my version more readable. But you're obviously welcome to commit either version. [1] https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html Best regards, Piotr Sikora From piotr at aviatrix.com Thu Mar 21 07:01:54 2024 From: piotr at aviatrix.com (Piotr Sikora) Date: Thu, 21 Mar 2024 07:01:54 +0000 Subject: [PATCH 1 of 2] SSL: add support for AWS-LC In-Reply-To: <20240318135536.bnqzgmqemz67pxgo@N00W24XTQX> References: <5e923992006199748e79.1709083334@ip-172-31-36-66.ec2.internal> <20240318135536.bnqzgmqemz67pxgo@N00W24XTQX> Message-ID: Hi Roman, > It looks like this library is not super popular, but the patch is relatively > large. Perhaps it's not as widely used as the forks that started ~10 years ago, but it's basically a version of BoringSSL that's more suitable to use with NGINX than BoringSSL itself: - it ships releases and it's versioned, - it supports OCSP stapling, - it supports multiple TLS certificates, - it supports big endian platforms supported by NGINX. Also, the patch is pretty small. > Also, compiling nginx with -DOPENSSL_IS_BORINGSSL should probably solve > the issue. For the time being, probably, but AWS folks are actively developing it, so I'd expect it to led to issues sooner rather than later. Best regards, Piotr Sikora From piotr at aviatrix.com Thu Mar 21 07:17:06 2024 From: piotr at aviatrix.com (Piotr Sikora) Date: Thu, 21 Mar 2024 07:17:06 +0000 Subject: [PATCH] Core: fix build without libcrypt In-Reply-To: <20240318121300.522yn5bbwnd2ext4@N00W24XTQX> References: <20240318121300.522yn5bbwnd2ext4@N00W24XTQX> Message-ID: Hi Roman, > Can you provide an example of a system where this fix is needed? 1. When linking against musl-libc on a system where libcrypt is linked against glibc and thus not linkable. 2. When cross-compiling, for similar reasons. Best regards, Piotr Sikora From piotr at aviatrix.com Thu Mar 21 07:19:27 2024 From: piotr at aviatrix.com (Piotr Sikora) Date: Thu, 21 Mar 2024 07:19:27 +0000 Subject: [PATCH] macOS: detect cache line size at runtime In-Reply-To: References: Message-ID: Hi Sergey, > I prefer not to introduce more ad-hoc prefixes in the log summary. > Something like moving the "macOS" part to the end should be fine. That's fine with me. > style: this breaks a perfect indentation of two spaces after type; > further, it appears to be unsorted by type; I'd put it after u_long Good catch, thanks! > This makes the following slight update to the patch. > If you're okey with it, I will commit it then. LGTM. Best regards, Piotr Sikora From piotr at aviatrix.com Thu Mar 21 07:23:00 2024 From: piotr at aviatrix.com (Piotr Sikora) Date: Thu, 21 Mar 2024 07:23:00 +0000 Subject: [PATCH] Configure: link libcrypt when a feature using it is detected In-Reply-To: <67E7F428-19EB-4370-B117-78FF9D6BEC18@nginx.com> References: <570e97dddeeddb79c715.1709083399@ip-172-31-36-66.ec2.internal> <67E7F428-19EB-4370-B117-78FF9D6BEC18@nginx.com> Message-ID: Hi Sergey, > I'd rewrote the description to be more specific: > > : Configure: fixed Linux crypt_r() test to add libcrypt. > : > : Previously, the resulting binary was successfully linked > : because libcrypt was added in a separate test for crypt(). That's fine with me. Best regards, Piotr Sikora From nejc.lovrencic at gmail.com Thu Mar 21 07:30:39 2024 From: nejc.lovrencic at gmail.com (=?iso-8859-1?q?Nejc_Lovrencic?=) Date: Thu, 21 Mar 2024 08:30:39 +0100 Subject: [PATCH] proxy_cache_lock: Make lock timer configurable Message-ID: <8d38e6642e82bb219bb5.1711006239@1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa> # HG changeset patch # User Nejc Lovrencic # Date 1711005111 -3600 # Thu Mar 21 08:11:51 2024 +0100 # Node ID 8d38e6642e82bb219bb5b586f1dcca5222a036e8 # Parent 89bff782528a91ad123b63b624f798e6fd9c8e68 proxy_cache_lock: Make lock timer configurable Default timer is set to 500ms. This in a worst-case scenario adds 500ms latency to MISS requests. This commit adds proxy_cache_lock_wait_time directive, which makes the timer configurable. diff -r 89bff782528a -r 8d38e6642e82 src/http/modules/ngx_http_proxy_module.c --- a/src/http/modules/ngx_http_proxy_module.c Wed Feb 14 20:03:00 2024 +0400 +++ b/src/http/modules/ngx_http_proxy_module.c Thu Mar 21 08:11:51 2024 +0100 @@ -592,6 +592,12 @@ offsetof(ngx_http_proxy_loc_conf_t, upstream.cache_lock_age), NULL }, + { ngx_string("proxy_cache_lock_wait_time"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1, + ngx_conf_set_msec_slot, + NGX_HTTP_LOC_CONF_OFFSET, + offsetof(ngx_http_proxy_loc_conf_t, upstream.cache_lock_wait_time), + NULL }, { ngx_string("proxy_cache_revalidate"), NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_FLAG, ngx_conf_set_flag_slot, @@ -3390,6 +3396,7 @@ conf->upstream.cache_lock = NGX_CONF_UNSET; conf->upstream.cache_lock_timeout = NGX_CONF_UNSET_MSEC; conf->upstream.cache_lock_age = NGX_CONF_UNSET_MSEC; + conf->upstream.cache_lock_wait_time = NGX_CONF_UNSET_MSEC; conf->upstream.cache_revalidate = NGX_CONF_UNSET; conf->upstream.cache_convert_head = NGX_CONF_UNSET; conf->upstream.cache_background_update = NGX_CONF_UNSET; @@ -3705,6 +3712,9 @@ ngx_conf_merge_msec_value(conf->upstream.cache_lock_age, prev->upstream.cache_lock_age, 5000); + ngx_conf_merge_msec_value(conf->upstream.cache_lock_wait_time, + prev->upstream.cache_lock_wait_time, 500); + ngx_conf_merge_value(conf->upstream.cache_revalidate, prev->upstream.cache_revalidate, 0); diff -r 89bff782528a -r 8d38e6642e82 src/http/ngx_http_cache.h --- a/src/http/ngx_http_cache.h Wed Feb 14 20:03:00 2024 +0400 +++ b/src/http/ngx_http_cache.h Thu Mar 21 08:11:51 2024 +0100 @@ -103,6 +103,7 @@ ngx_msec_t lock_timeout; ngx_msec_t lock_age; ngx_msec_t lock_time; + ngx_msec_t lock_wait_time; ngx_msec_t wait_time; ngx_event_t wait_event; diff -r 89bff782528a -r 8d38e6642e82 src/http/ngx_http_file_cache.c --- a/src/http/ngx_http_file_cache.c Wed Feb 14 20:03:00 2024 +0400 +++ b/src/http/ngx_http_file_cache.c Thu Mar 21 08:11:51 2024 +0100 @@ -452,7 +452,10 @@ timer = c->wait_time - now; - ngx_add_timer(&c->wait_event, (timer > 500) ? 500 : timer); + ngx_log_debug2(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, + "http file cache lock timer tm:%M lwt:%M", timer, c->lock_wait_time); + + ngx_add_timer(&c->wait_event, (timer > c->lock_wait_time) ? c->lock_wait_time : timer); r->main->blocked++; @@ -531,7 +534,7 @@ ngx_shmtx_unlock(&cache->shpool->mutex); if (wait) { - ngx_add_timer(&c->wait_event, (timer > 500) ? 500 : timer); + ngx_add_timer(&c->wait_event, (timer > c->lock_wait_time) ? c->lock_wait_time : timer); return NGX_AGAIN; } diff -r 89bff782528a -r 8d38e6642e82 src/http/ngx_http_upstream.c --- a/src/http/ngx_http_upstream.c Wed Feb 14 20:03:00 2024 +0400 +++ b/src/http/ngx_http_upstream.c Thu Mar 21 08:11:51 2024 +0100 @@ -894,6 +894,7 @@ c->lock = u->conf->cache_lock; c->lock_timeout = u->conf->cache_lock_timeout; c->lock_age = u->conf->cache_lock_age; + c->lock_wait_time = u->conf->cache_lock_wait_time; u->cache_status = NGX_HTTP_CACHE_MISS; } diff -r 89bff782528a -r 8d38e6642e82 src/http/ngx_http_upstream.h --- a/src/http/ngx_http_upstream.h Wed Feb 14 20:03:00 2024 +0400 +++ b/src/http/ngx_http_upstream.h Thu Mar 21 08:11:51 2024 +0100 @@ -204,6 +204,7 @@ ngx_flag_t cache_lock; ngx_msec_t cache_lock_timeout; ngx_msec_t cache_lock_age; + ngx_msec_t cache_lock_wait_time; ngx_flag_t cache_revalidate; ngx_flag_t cache_convert_head; From arut at nginx.com Thu Mar 21 13:40:48 2024 From: arut at nginx.com (=?utf-8?q?Roman_Arutyunyan?=) Date: Thu, 21 Mar 2024 13:40:48 +0000 Subject: [nginx] Version bump. Message-ID: details: https://hg.nginx.org/nginx/rev/d6f8a2c31cde branches: changeset: 9214:d6f8a2c31cde user: Roman Arutyunyan date: Thu Mar 21 17:06:21 2024 +0400 description: Version bump. diffstat: src/core/nginx.h | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diffs (14 lines): diff -r 89bff782528a -r d6f8a2c31cde src/core/nginx.h --- a/src/core/nginx.h Wed Feb 14 20:03:00 2024 +0400 +++ b/src/core/nginx.h Thu Mar 21 17:06:21 2024 +0400 @@ -9,8 +9,8 @@ #define _NGINX_H_INCLUDED_ -#define nginx_version 1025004 -#define NGINX_VERSION "1.25.4" +#define nginx_version 1025005 +#define NGINX_VERSION "1.25.5" #define NGINX_VER "nginx/" NGINX_VERSION #ifdef NGX_BUILD From arut at nginx.com Thu Mar 21 13:40:51 2024 From: arut at nginx.com (=?utf-8?q?Roman_Arutyunyan?=) Date: Thu, 21 Mar 2024 13:40:51 +0000 Subject: [nginx] Stream: socket peek in preread phase. Message-ID: details: https://hg.nginx.org/nginx/rev/cf890df37bb6 branches: changeset: 9215:cf890df37bb6 user: Roman Arutyunyan date: Wed Dec 13 18:04:55 2023 +0400 description: Stream: socket peek in preread phase. Previously, preread buffer was always read out from socket, which made it impossible to terminate SSL on the connection without introducing additional SSL BIOs. The following patches will rely on this. Now, when possible, recv(MSG_PEEK) is used instead, which keeps data in socket. It's called if SSL is not already terminated and if an egde-triggered event method is used. For epoll, EPOLLRDHUP support is also required. diffstat: src/stream/ngx_stream_core_module.c | 189 +++++++++++++++++++++++++++-------- 1 files changed, 146 insertions(+), 43 deletions(-) diffs (229 lines): diff -r d6f8a2c31cde -r cf890df37bb6 src/stream/ngx_stream_core_module.c --- a/src/stream/ngx_stream_core_module.c Thu Mar 21 17:06:21 2024 +0400 +++ b/src/stream/ngx_stream_core_module.c Wed Dec 13 18:04:55 2023 +0400 @@ -10,6 +10,11 @@ #include +static ngx_uint_t ngx_stream_preread_can_peek(ngx_connection_t *c); +static ngx_int_t ngx_stream_preread_peek(ngx_stream_session_t *s, + ngx_stream_phase_handler_t *ph); +static ngx_int_t ngx_stream_preread(ngx_stream_session_t *s, + ngx_stream_phase_handler_t *ph); static ngx_int_t ngx_stream_core_preconfiguration(ngx_conf_t *cf); static void *ngx_stream_core_create_main_conf(ngx_conf_t *cf); static char *ngx_stream_core_init_main_conf(ngx_conf_t *cf, void *conf); @@ -203,8 +208,6 @@ ngx_int_t ngx_stream_core_preread_phase(ngx_stream_session_t *s, ngx_stream_phase_handler_t *ph) { - size_t size; - ssize_t n; ngx_int_t rc; ngx_connection_t *c; ngx_stream_core_srv_conf_t *cscf; @@ -217,56 +220,33 @@ ngx_stream_core_preread_phase(ngx_stream if (c->read->timedout) { rc = NGX_STREAM_OK; + goto done; + } - } else if (c->read->timer_set) { - rc = NGX_AGAIN; + if (!c->read->timer_set) { + rc = ph->handler(s); - } else { - rc = ph->handler(s); + if (rc != NGX_AGAIN) { + goto done; + } } - while (rc == NGX_AGAIN) { - + if (c->buffer == NULL) { + c->buffer = ngx_create_temp_buf(c->pool, cscf->preread_buffer_size); if (c->buffer == NULL) { - c->buffer = ngx_create_temp_buf(c->pool, cscf->preread_buffer_size); - if (c->buffer == NULL) { - rc = NGX_ERROR; - break; - } + rc = NGX_ERROR; + goto done; } - - size = c->buffer->end - c->buffer->last; - - if (size == 0) { - ngx_log_error(NGX_LOG_ERR, c->log, 0, "preread buffer full"); - rc = NGX_STREAM_BAD_REQUEST; - break; - } + } - if (c->read->eof) { - rc = NGX_STREAM_OK; - break; - } - - if (!c->read->ready) { - break; - } - - n = c->recv(c, c->buffer->last, size); + if (ngx_stream_preread_can_peek(c)) { + rc = ngx_stream_preread_peek(s, ph); - if (n == NGX_ERROR || n == 0) { - rc = NGX_STREAM_OK; - break; - } + } else { + rc = ngx_stream_preread(s, ph); + } - if (n == NGX_AGAIN) { - break; - } - - c->buffer->last += n; - - rc = ph->handler(s); - } +done: if (rc == NGX_AGAIN) { if (ngx_handle_read_event(c->read, 0) != NGX_OK) { @@ -311,6 +291,129 @@ ngx_stream_core_preread_phase(ngx_stream } +static ngx_uint_t +ngx_stream_preread_can_peek(ngx_connection_t *c) +{ +#if (NGX_STREAM_SSL) + if (c->ssl) { + return 0; + } +#endif + + if ((ngx_event_flags & NGX_USE_CLEAR_EVENT) == 0) { + return 0; + } + +#if (NGX_HAVE_KQUEUE) + if (ngx_event_flags & NGX_USE_KQUEUE_EVENT) { + return 1; + } +#endif + +#if (NGX_HAVE_EPOLLRDHUP) + if ((ngx_event_flags & NGX_USE_EPOLL_EVENT) && ngx_use_epoll_rdhup) { + return 1; + } +#endif + + return 0; +} + + +static ngx_int_t +ngx_stream_preread_peek(ngx_stream_session_t *s, ngx_stream_phase_handler_t *ph) +{ + ssize_t n; + ngx_int_t rc; + ngx_err_t err; + ngx_connection_t *c; + + c = s->connection; + + n = recv(c->fd, (char *) c->buffer->last, + c->buffer->end - c->buffer->last, MSG_PEEK); + + err = ngx_socket_errno; + + ngx_log_debug1(NGX_LOG_DEBUG_STREAM, c->log, 0, "stream recv(): %z", n); + + if (n == -1) { + if (err == NGX_EAGAIN) { + c->read->ready = 0; + return NGX_AGAIN; + } + + ngx_connection_error(c, err, "recv() failed"); + return NGX_STREAM_OK; + } + + if (n == 0) { + return NGX_STREAM_OK; + } + + c->buffer->last += n; + + rc = ph->handler(s); + + if (rc != NGX_AGAIN) { + c->buffer->last = c->buffer->pos; + return rc; + } + + if (c->buffer->last == c->buffer->end) { + ngx_log_error(NGX_LOG_ERR, c->log, 0, "preread buffer full"); + return NGX_STREAM_BAD_REQUEST; + } + + if (c->read->pending_eof) { + return NGX_STREAM_OK; + } + + c->buffer->last = c->buffer->pos; + + return NGX_AGAIN; +} + + +static ngx_int_t +ngx_stream_preread(ngx_stream_session_t *s, ngx_stream_phase_handler_t *ph) +{ + ssize_t n; + ngx_int_t rc; + ngx_connection_t *c; + + c = s->connection; + + while (c->read->ready) { + + n = c->recv(c, c->buffer->last, c->buffer->end - c->buffer->last); + + if (n == NGX_AGAIN) { + return NGX_AGAIN; + } + + if (n == NGX_ERROR || n == 0) { + return NGX_STREAM_OK; + } + + c->buffer->last += n; + + rc = ph->handler(s); + + if (rc != NGX_AGAIN) { + return rc; + } + + if (c->buffer->last == c->buffer->end) { + ngx_log_error(NGX_LOG_ERR, c->log, 0, "preread buffer full"); + return NGX_STREAM_BAD_REQUEST; + } + } + + return NGX_AGAIN; +} + + ngx_int_t ngx_stream_core_content_phase(ngx_stream_session_t *s, ngx_stream_phase_handler_t *ph) From arut at nginx.com Thu Mar 21 13:40:54 2024 From: arut at nginx.com (=?utf-8?q?Roman_Arutyunyan?=) Date: Thu, 21 Mar 2024 13:40:54 +0000 Subject: [nginx] Stream: virtual servers. Message-ID: details: https://hg.nginx.org/nginx/rev/e28b044908cb branches: changeset: 9216:e28b044908cb user: Roman Arutyunyan date: Thu Dec 14 21:58:39 2023 +0400 description: Stream: virtual servers. Server name is taken either from ngx_stream_ssl_module or ngx_stream_ssl_preread_module. The change adds "default_server" parameter to the "listen" directive, as well as the following directives: "server_names_hash_max_size", "server_names_hash_bucket_size", "server_name" and "ssl_reject_handshake". diffstat: src/stream/ngx_stream.c | 902 ++++++++++++++++++++++------ src/stream/ngx_stream.h | 160 +++- src/stream/ngx_stream_core_module.c | 494 ++++++++++++--- src/stream/ngx_stream_handler.c | 10 +- src/stream/ngx_stream_ssl_module.c | 224 ++++++- src/stream/ngx_stream_ssl_module.h | 5 +- src/stream/ngx_stream_ssl_preread_module.c | 57 +- 7 files changed, 1457 insertions(+), 395 deletions(-) diffs (truncated from 2452 to 1000 lines): diff -r cf890df37bb6 -r e28b044908cb src/stream/ngx_stream.c --- a/src/stream/ngx_stream.c Wed Dec 13 18:04:55 2023 +0400 +++ b/src/stream/ngx_stream.c Thu Dec 14 21:58:39 2023 +0400 @@ -16,16 +16,34 @@ static ngx_int_t ngx_stream_init_phases( ngx_stream_core_main_conf_t *cmcf); static ngx_int_t ngx_stream_init_phase_handlers(ngx_conf_t *cf, ngx_stream_core_main_conf_t *cmcf); -static ngx_int_t ngx_stream_add_ports(ngx_conf_t *cf, ngx_array_t *ports, - ngx_stream_listen_t *listen); -static char *ngx_stream_optimize_servers(ngx_conf_t *cf, ngx_array_t *ports); + +static ngx_int_t ngx_stream_add_addresses(ngx_conf_t *cf, + ngx_stream_core_srv_conf_t *cscf, ngx_stream_conf_port_t *port, + ngx_stream_listen_opt_t *lsopt); +static ngx_int_t ngx_stream_add_address(ngx_conf_t *cf, + ngx_stream_core_srv_conf_t *cscf, ngx_stream_conf_port_t *port, + ngx_stream_listen_opt_t *lsopt); +static ngx_int_t ngx_stream_add_server(ngx_conf_t *cf, + ngx_stream_core_srv_conf_t *cscf, ngx_stream_conf_addr_t *addr); + +static ngx_int_t ngx_stream_optimize_servers(ngx_conf_t *cf, + ngx_stream_core_main_conf_t *cmcf, ngx_array_t *ports); +static ngx_int_t ngx_stream_server_names(ngx_conf_t *cf, + ngx_stream_core_main_conf_t *cmcf, ngx_stream_conf_addr_t *addr); +static ngx_int_t ngx_stream_cmp_conf_addrs(const void *one, const void *two); +static int ngx_libc_cdecl ngx_stream_cmp_dns_wildcards(const void *one, + const void *two); + +static ngx_int_t ngx_stream_init_listening(ngx_conf_t *cf, + ngx_stream_conf_port_t *port); +static ngx_listening_t *ngx_stream_add_listening(ngx_conf_t *cf, + ngx_stream_conf_addr_t *addr); static ngx_int_t ngx_stream_add_addrs(ngx_conf_t *cf, ngx_stream_port_t *stport, ngx_stream_conf_addr_t *addr); #if (NGX_HAVE_INET6) static ngx_int_t ngx_stream_add_addrs6(ngx_conf_t *cf, ngx_stream_port_t *stport, ngx_stream_conf_addr_t *addr); #endif -static ngx_int_t ngx_stream_cmp_conf_addrs(const void *one, const void *two); ngx_uint_t ngx_stream_max_module; @@ -74,10 +92,8 @@ static char * ngx_stream_block(ngx_conf_t *cf, ngx_command_t *cmd, void *conf) { char *rv; - ngx_uint_t i, m, mi, s; + ngx_uint_t mi, m, s; ngx_conf_t pcf; - ngx_array_t ports; - ngx_stream_listen_t *listen; ngx_stream_module_t *module; ngx_stream_conf_ctx_t *ctx; ngx_stream_core_srv_conf_t **cscfp; @@ -251,21 +267,13 @@ ngx_stream_block(ngx_conf_t *cf, ngx_com return NGX_CONF_ERROR; } - if (ngx_array_init(&ports, cf->temp_pool, 4, sizeof(ngx_stream_conf_port_t)) - != NGX_OK) - { + /* optimize the lists of ports, addresses and server names */ + + if (ngx_stream_optimize_servers(cf, cmcf, cmcf->ports) != NGX_OK) { return NGX_CONF_ERROR; } - listen = cmcf->listen.elts; - - for (i = 0; i < cmcf->listen.nelts; i++) { - if (ngx_stream_add_ports(cf, &ports, &listen[i]) != NGX_OK) { - return NGX_CONF_ERROR; - } - } - - return ngx_stream_optimize_servers(cf, &ports); + return NGX_CONF_OK; } @@ -377,73 +385,295 @@ ngx_stream_init_phase_handlers(ngx_conf_ } -static ngx_int_t -ngx_stream_add_ports(ngx_conf_t *cf, ngx_array_t *ports, - ngx_stream_listen_t *listen) +ngx_int_t +ngx_stream_add_listen(ngx_conf_t *cf, ngx_stream_core_srv_conf_t *cscf, + ngx_stream_listen_opt_t *lsopt) { - in_port_t p; - ngx_uint_t i; - struct sockaddr *sa; - ngx_stream_conf_port_t *port; - ngx_stream_conf_addr_t *addr; + in_port_t p; + ngx_uint_t i; + struct sockaddr *sa; + ngx_stream_conf_port_t *port; + ngx_stream_core_main_conf_t *cmcf; + + cmcf = ngx_stream_conf_get_module_main_conf(cf, ngx_stream_core_module); - sa = listen->sockaddr; + if (cmcf->ports == NULL) { + cmcf->ports = ngx_array_create(cf->temp_pool, 2, + sizeof(ngx_stream_conf_port_t)); + if (cmcf->ports == NULL) { + return NGX_ERROR; + } + } + + sa = lsopt->sockaddr; p = ngx_inet_get_port(sa); - port = ports->elts; - for (i = 0; i < ports->nelts; i++) { + port = cmcf->ports->elts; + for (i = 0; i < cmcf->ports->nelts; i++) { - if (p == port[i].port - && listen->type == port[i].type - && sa->sa_family == port[i].family) + if (p != port[i].port + || lsopt->type != port[i].type + || sa->sa_family != port[i].family) { - /* a port is already in the port list */ + continue; + } - port = &port[i]; - goto found; - } + /* a port is already in the port list */ + + return ngx_stream_add_addresses(cf, cscf, &port[i], lsopt); } /* add a port to the port list */ - port = ngx_array_push(ports); + port = ngx_array_push(cmcf->ports); if (port == NULL) { return NGX_ERROR; } port->family = sa->sa_family; - port->type = listen->type; + port->type = lsopt->type; port->port = p; + port->addrs.elts = NULL; + + return ngx_stream_add_address(cf, cscf, port, lsopt); +} + + +static ngx_int_t +ngx_stream_add_addresses(ngx_conf_t *cf, ngx_stream_core_srv_conf_t *cscf, + ngx_stream_conf_port_t *port, ngx_stream_listen_opt_t *lsopt) +{ + ngx_uint_t i, default_server, proxy_protocol, + protocols, protocols_prev; + ngx_stream_conf_addr_t *addr; +#if (NGX_STREAM_SSL) + ngx_uint_t ssl; +#endif + + /* + * we cannot compare whole sockaddr struct's as kernel + * may fill some fields in inherited sockaddr struct's + */ + + addr = port->addrs.elts; + + for (i = 0; i < port->addrs.nelts; i++) { + + if (ngx_cmp_sockaddr(lsopt->sockaddr, lsopt->socklen, + addr[i].opt.sockaddr, + addr[i].opt.socklen, 0) + != NGX_OK) + { + continue; + } + + /* the address is already in the address list */ + + if (ngx_stream_add_server(cf, cscf, &addr[i]) != NGX_OK) { + return NGX_ERROR; + } + + /* preserve default_server bit during listen options overwriting */ + default_server = addr[i].opt.default_server; + + proxy_protocol = lsopt->proxy_protocol || addr[i].opt.proxy_protocol; + protocols = lsopt->proxy_protocol; + protocols_prev = addr[i].opt.proxy_protocol; + +#if (NGX_STREAM_SSL) + ssl = lsopt->ssl || addr[i].opt.ssl; + protocols |= lsopt->ssl << 1; + protocols_prev |= addr[i].opt.ssl << 1; +#endif + + if (lsopt->set) { + + if (addr[i].opt.set) { + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, + "duplicate listen options for %V", + &addr[i].opt.addr_text); + return NGX_ERROR; + } + + addr[i].opt = *lsopt; + } + + /* check the duplicate "default" server for this address:port */ - if (ngx_array_init(&port->addrs, cf->temp_pool, 2, - sizeof(ngx_stream_conf_addr_t)) - != NGX_OK) - { - return NGX_ERROR; + if (lsopt->default_server) { + + if (default_server) { + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, + "a duplicate default server for %V", + &addr[i].opt.addr_text); + return NGX_ERROR; + } + + default_server = 1; + addr[i].default_server = cscf; + } + + /* check for conflicting protocol options */ + + if ((protocols | protocols_prev) != protocols_prev) { + + /* options added */ + + if ((addr[i].opt.set && !lsopt->set) + || addr[i].protocols_changed + || (protocols | protocols_prev) != protocols) + { + ngx_conf_log_error(NGX_LOG_WARN, cf, 0, + "protocol options redefined for %V", + &addr[i].opt.addr_text); + } + + addr[i].protocols = protocols_prev; + addr[i].protocols_set = 1; + addr[i].protocols_changed = 1; + + } else if ((protocols_prev | protocols) != protocols) { + + /* options removed */ + + if (lsopt->set + || (addr[i].protocols_set && protocols != addr[i].protocols)) + { + ngx_conf_log_error(NGX_LOG_WARN, cf, 0, + "protocol options redefined for %V", + &addr[i].opt.addr_text); + } + + addr[i].protocols = protocols; + addr[i].protocols_set = 1; + addr[i].protocols_changed = 1; + + } else { + + /* the same options */ + + if ((lsopt->set && addr[i].protocols_changed) + || (addr[i].protocols_set && protocols != addr[i].protocols)) + { + ngx_conf_log_error(NGX_LOG_WARN, cf, 0, + "protocol options redefined for %V", + &addr[i].opt.addr_text); + } + + addr[i].protocols = protocols; + addr[i].protocols_set = 1; + } + + addr[i].opt.default_server = default_server; + addr[i].opt.proxy_protocol = proxy_protocol; +#if (NGX_STREAM_SSL) + addr[i].opt.ssl = ssl; +#endif + + return NGX_OK; } -found: + /* add the address to the addresses list that bound to this port */ + + return ngx_stream_add_address(cf, cscf, port, lsopt); +} + + +/* + * add the server address, the server names and the server core module + * configurations to the port list + */ + +static ngx_int_t +ngx_stream_add_address(ngx_conf_t *cf, ngx_stream_core_srv_conf_t *cscf, + ngx_stream_conf_port_t *port, ngx_stream_listen_opt_t *lsopt) +{ + ngx_stream_conf_addr_t *addr; + + if (port->addrs.elts == NULL) { + if (ngx_array_init(&port->addrs, cf->temp_pool, 4, + sizeof(ngx_stream_conf_addr_t)) + != NGX_OK) + { + return NGX_ERROR; + } + } addr = ngx_array_push(&port->addrs); if (addr == NULL) { return NGX_ERROR; } - addr->opt = *listen; + addr->opt = *lsopt; + addr->protocols = 0; + addr->protocols_set = 0; + addr->protocols_changed = 0; + addr->hash.buckets = NULL; + addr->hash.size = 0; + addr->wc_head = NULL; + addr->wc_tail = NULL; +#if (NGX_PCRE) + addr->nregex = 0; + addr->regex = NULL; +#endif + addr->default_server = cscf; + addr->servers.elts = NULL; + + return ngx_stream_add_server(cf, cscf, addr); +} + + +/* add the server core module configuration to the address:port */ + +static ngx_int_t +ngx_stream_add_server(ngx_conf_t *cf, ngx_stream_core_srv_conf_t *cscf, + ngx_stream_conf_addr_t *addr) +{ + ngx_uint_t i; + ngx_stream_core_srv_conf_t **server; + + if (addr->servers.elts == NULL) { + if (ngx_array_init(&addr->servers, cf->temp_pool, 4, + sizeof(ngx_stream_core_srv_conf_t *)) + != NGX_OK) + { + return NGX_ERROR; + } + + } else { + server = addr->servers.elts; + for (i = 0; i < addr->servers.nelts; i++) { + if (server[i] == cscf) { + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, + "a duplicate listen %V", + &addr->opt.addr_text); + return NGX_ERROR; + } + } + } + + server = ngx_array_push(&addr->servers); + if (server == NULL) { + return NGX_ERROR; + } + + *server = cscf; return NGX_OK; } -static char * -ngx_stream_optimize_servers(ngx_conf_t *cf, ngx_array_t *ports) +static ngx_int_t +ngx_stream_optimize_servers(ngx_conf_t *cf, ngx_stream_core_main_conf_t *cmcf, + ngx_array_t *ports) { - ngx_uint_t i, p, last, bind_wildcard; - ngx_listening_t *ls; - ngx_stream_port_t *stport; - ngx_stream_conf_port_t *port; - ngx_stream_conf_addr_t *addr; - ngx_stream_core_srv_conf_t *cscf; + ngx_uint_t p, a; + ngx_stream_conf_port_t *port; + ngx_stream_conf_addr_t *addr; + + if (ports == NULL) { + return NGX_OK; + } port = ports->elts; for (p = 0; p < ports->nelts; p++) { @@ -451,175 +681,191 @@ ngx_stream_optimize_servers(ngx_conf_t * ngx_sort(port[p].addrs.elts, (size_t) port[p].addrs.nelts, sizeof(ngx_stream_conf_addr_t), ngx_stream_cmp_conf_addrs); - addr = port[p].addrs.elts; - last = port[p].addrs.nelts; - /* - * if there is the binding to the "*:port" then we need to bind() - * to the "*:port" only and ignore the other bindings + * check whether all name-based servers have the same + * configuration as a default server for given address:port */ - if (addr[last - 1].opt.wildcard) { - addr[last - 1].opt.bind = 1; - bind_wildcard = 1; + addr = port[p].addrs.elts; + for (a = 0; a < port[p].addrs.nelts; a++) { - } else { - bind_wildcard = 0; + if (addr[a].servers.nelts > 1 +#if (NGX_PCRE) + || addr[a].default_server->captures +#endif + ) + { + if (ngx_stream_server_names(cf, cmcf, &addr[a]) != NGX_OK) { + return NGX_ERROR; + } + } } - i = 0; - - while (i < last) { - - if (bind_wildcard && !addr[i].opt.bind) { - i++; - continue; - } - - ls = ngx_create_listening(cf, addr[i].opt.sockaddr, - addr[i].opt.socklen); - if (ls == NULL) { - return NGX_CONF_ERROR; - } - - ls->addr_ntop = 1; - ls->handler = ngx_stream_init_connection; - ls->pool_size = 256; - ls->type = addr[i].opt.type; - - cscf = addr->opt.ctx->srv_conf[ngx_stream_core_module.ctx_index]; - - ls->logp = cscf->error_log; - ls->log.data = &ls->addr_text; - ls->log.handler = ngx_accept_log_error; - - ls->backlog = addr[i].opt.backlog; - ls->rcvbuf = addr[i].opt.rcvbuf; - ls->sndbuf = addr[i].opt.sndbuf; - - ls->wildcard = addr[i].opt.wildcard; - - ls->keepalive = addr[i].opt.so_keepalive; -#if (NGX_HAVE_KEEPALIVE_TUNABLE) - ls->keepidle = addr[i].opt.tcp_keepidle; - ls->keepintvl = addr[i].opt.tcp_keepintvl; - ls->keepcnt = addr[i].opt.tcp_keepcnt; -#endif - -#if (NGX_HAVE_INET6) - ls->ipv6only = addr[i].opt.ipv6only; -#endif - -#if (NGX_HAVE_TCP_FASTOPEN) - ls->fastopen = addr[i].opt.fastopen; -#endif - -#if (NGX_HAVE_REUSEPORT) - ls->reuseport = addr[i].opt.reuseport; -#endif - - stport = ngx_palloc(cf->pool, sizeof(ngx_stream_port_t)); - if (stport == NULL) { - return NGX_CONF_ERROR; - } - - ls->servers = stport; - - stport->naddrs = i + 1; - - switch (ls->sockaddr->sa_family) { -#if (NGX_HAVE_INET6) - case AF_INET6: - if (ngx_stream_add_addrs6(cf, stport, addr) != NGX_OK) { - return NGX_CONF_ERROR; - } - break; -#endif - default: /* AF_INET */ - if (ngx_stream_add_addrs(cf, stport, addr) != NGX_OK) { - return NGX_CONF_ERROR; - } - break; - } - - addr++; - last--; + if (ngx_stream_init_listening(cf, &port[p]) != NGX_OK) { + return NGX_ERROR; } } - return NGX_CONF_OK; -} - - -static ngx_int_t -ngx_stream_add_addrs(ngx_conf_t *cf, ngx_stream_port_t *stport, - ngx_stream_conf_addr_t *addr) -{ - ngx_uint_t i; - struct sockaddr_in *sin; - ngx_stream_in_addr_t *addrs; - - stport->addrs = ngx_pcalloc(cf->pool, - stport->naddrs * sizeof(ngx_stream_in_addr_t)); - if (stport->addrs == NULL) { - return NGX_ERROR; - } - - addrs = stport->addrs; - - for (i = 0; i < stport->naddrs; i++) { - - sin = (struct sockaddr_in *) addr[i].opt.sockaddr; - addrs[i].addr = sin->sin_addr.s_addr; - - addrs[i].conf.ctx = addr[i].opt.ctx; -#if (NGX_STREAM_SSL) - addrs[i].conf.ssl = addr[i].opt.ssl; -#endif - addrs[i].conf.proxy_protocol = addr[i].opt.proxy_protocol; - addrs[i].conf.addr_text = addr[i].opt.addr_text; - } - return NGX_OK; } -#if (NGX_HAVE_INET6) - static ngx_int_t -ngx_stream_add_addrs6(ngx_conf_t *cf, ngx_stream_port_t *stport, +ngx_stream_server_names(ngx_conf_t *cf, ngx_stream_core_main_conf_t *cmcf, ngx_stream_conf_addr_t *addr) { - ngx_uint_t i; - struct sockaddr_in6 *sin6; - ngx_stream_in6_addr_t *addrs6; + ngx_int_t rc; + ngx_uint_t n, s; + ngx_hash_init_t hash; + ngx_hash_keys_arrays_t ha; + ngx_stream_server_name_t *name; + ngx_stream_core_srv_conf_t **cscfp; +#if (NGX_PCRE) + ngx_uint_t regex, i; - stport->addrs = ngx_pcalloc(cf->pool, - stport->naddrs * sizeof(ngx_stream_in6_addr_t)); - if (stport->addrs == NULL) { + regex = 0; +#endif + + ngx_memzero(&ha, sizeof(ngx_hash_keys_arrays_t)); + + ha.temp_pool = ngx_create_pool(NGX_DEFAULT_POOL_SIZE, cf->log); + if (ha.temp_pool == NULL) { return NGX_ERROR; } - addrs6 = stport->addrs; + ha.pool = cf->pool; + + if (ngx_hash_keys_array_init(&ha, NGX_HASH_LARGE) != NGX_OK) { + goto failed; + } + + cscfp = addr->servers.elts; + + for (s = 0; s < addr->servers.nelts; s++) { + + name = cscfp[s]->server_names.elts; + + for (n = 0; n < cscfp[s]->server_names.nelts; n++) { - for (i = 0; i < stport->naddrs; i++) { +#if (NGX_PCRE) + if (name[n].regex) { + regex++; + continue; + } +#endif - sin6 = (struct sockaddr_in6 *) addr[i].opt.sockaddr; - addrs6[i].addr6 = sin6->sin6_addr; + rc = ngx_hash_add_key(&ha, &name[n].name, name[n].server, + NGX_HASH_WILDCARD_KEY); + + if (rc == NGX_ERROR) { + goto failed; + } - addrs6[i].conf.ctx = addr[i].opt.ctx; -#if (NGX_STREAM_SSL) - addrs6[i].conf.ssl = addr[i].opt.ssl; -#endif - addrs6[i].conf.proxy_protocol = addr[i].opt.proxy_protocol; - addrs6[i].conf.addr_text = addr[i].opt.addr_text; + if (rc == NGX_DECLINED) { + ngx_log_error(NGX_LOG_EMERG, cf->log, 0, + "invalid server name or wildcard \"%V\" on %V", + &name[n].name, &addr->opt.addr_text); + goto failed; + } + + if (rc == NGX_BUSY) { + ngx_log_error(NGX_LOG_WARN, cf->log, 0, + "conflicting server name \"%V\" on %V, ignored", + &name[n].name, &addr->opt.addr_text); + } + } + } + + hash.key = ngx_hash_key_lc; + hash.max_size = cmcf->server_names_hash_max_size; + hash.bucket_size = cmcf->server_names_hash_bucket_size; + hash.name = "server_names_hash"; + hash.pool = cf->pool; + + if (ha.keys.nelts) { + hash.hash = &addr->hash; + hash.temp_pool = NULL; + + if (ngx_hash_init(&hash, ha.keys.elts, ha.keys.nelts) != NGX_OK) { + goto failed; + } } - return NGX_OK; -} + if (ha.dns_wc_head.nelts) { + + ngx_qsort(ha.dns_wc_head.elts, (size_t) ha.dns_wc_head.nelts, + sizeof(ngx_hash_key_t), ngx_stream_cmp_dns_wildcards); + + hash.hash = NULL; + hash.temp_pool = ha.temp_pool; + + if (ngx_hash_wildcard_init(&hash, ha.dns_wc_head.elts, + ha.dns_wc_head.nelts) + != NGX_OK) + { + goto failed; + } + + addr->wc_head = (ngx_hash_wildcard_t *) hash.hash; + } + + if (ha.dns_wc_tail.nelts) { + + ngx_qsort(ha.dns_wc_tail.elts, (size_t) ha.dns_wc_tail.nelts, + sizeof(ngx_hash_key_t), ngx_stream_cmp_dns_wildcards); + + hash.hash = NULL; + hash.temp_pool = ha.temp_pool; + + if (ngx_hash_wildcard_init(&hash, ha.dns_wc_tail.elts, + ha.dns_wc_tail.nelts) + != NGX_OK) + { + goto failed; + } + + addr->wc_tail = (ngx_hash_wildcard_t *) hash.hash; + } + + ngx_destroy_pool(ha.temp_pool); + +#if (NGX_PCRE) + + if (regex == 0) { + return NGX_OK; + } + + addr->nregex = regex; + addr->regex = ngx_palloc(cf->pool, + regex * sizeof(ngx_stream_server_name_t)); + if (addr->regex == NULL) { + return NGX_ERROR; + } + + i = 0; + + for (s = 0; s < addr->servers.nelts; s++) { + + name = cscfp[s]->server_names.elts; + + for (n = 0; n < cscfp[s]->server_names.nelts; n++) { + if (name[n].regex) { + addr->regex[i++] = name[n]; + } + } + } #endif + return NGX_OK; + +failed: + + ngx_destroy_pool(ha.temp_pool); + + return NGX_ERROR; +} + static ngx_int_t ngx_stream_cmp_conf_addrs(const void *one, const void *two) @@ -630,12 +876,12 @@ ngx_stream_cmp_conf_addrs(const void *on second = (ngx_stream_conf_addr_t *) two; if (first->opt.wildcard) { - /* a wildcard must be the last resort, shift it to the end */ + /* a wildcard address must be the last resort, shift it to the end */ return 1; } if (second->opt.wildcard) { - /* a wildcard must be the last resort, shift it to the end */ + /* a wildcard address must be the last resort, shift it to the end */ return -1; } @@ -653,3 +899,265 @@ ngx_stream_cmp_conf_addrs(const void *on return 0; } + + +static int ngx_libc_cdecl +ngx_stream_cmp_dns_wildcards(const void *one, const void *two) +{ + ngx_hash_key_t *first, *second; + + first = (ngx_hash_key_t *) one; + second = (ngx_hash_key_t *) two; + + return ngx_dns_strcmp(first->key.data, second->key.data); +} + + +static ngx_int_t +ngx_stream_init_listening(ngx_conf_t *cf, ngx_stream_conf_port_t *port) +{ + ngx_uint_t i, last, bind_wildcard; + ngx_listening_t *ls; + ngx_stream_port_t *stport; + ngx_stream_conf_addr_t *addr; + + addr = port->addrs.elts; + last = port->addrs.nelts; + + /* + * If there is a binding to an "*:port" then we need to bind() to + * the "*:port" only and ignore other implicit bindings. The bindings + * have been already sorted: explicit bindings are on the start, then + * implicit bindings go, and wildcard binding is in the end. + */ + + if (addr[last - 1].opt.wildcard) { + addr[last - 1].opt.bind = 1; + bind_wildcard = 1; + + } else { + bind_wildcard = 0; + } + + i = 0; + + while (i < last) { + + if (bind_wildcard && !addr[i].opt.bind) { + i++; + continue; + } + + ls = ngx_stream_add_listening(cf, &addr[i]); + if (ls == NULL) { + return NGX_ERROR; + } + + stport = ngx_pcalloc(cf->pool, sizeof(ngx_stream_port_t)); + if (stport == NULL) { + return NGX_ERROR; + } + + ls->servers = stport; + + stport->naddrs = i + 1; + + switch (ls->sockaddr->sa_family) { + +#if (NGX_HAVE_INET6) + case AF_INET6: + if (ngx_stream_add_addrs6(cf, stport, addr) != NGX_OK) { + return NGX_ERROR; + } + break; +#endif + default: /* AF_INET */ + if (ngx_stream_add_addrs(cf, stport, addr) != NGX_OK) { + return NGX_ERROR; + } + break; + } + + addr++; + last--; + } + + return NGX_OK; +} + + +static ngx_listening_t * +ngx_stream_add_listening(ngx_conf_t *cf, ngx_stream_conf_addr_t *addr) +{ + ngx_listening_t *ls; + ngx_stream_core_srv_conf_t *cscf; + + ls = ngx_create_listening(cf, addr->opt.sockaddr, addr->opt.socklen); + if (ls == NULL) { + return NULL; + } + + ls->addr_ntop = 1; + + ls->handler = ngx_stream_init_connection; + + ls->pool_size = 256; + + cscf = addr->default_server; + + ls->logp = cscf->error_log; + ls->log.data = &ls->addr_text; + ls->log.handler = ngx_accept_log_error; + + ls->type = addr->opt.type; + ls->backlog = addr->opt.backlog; + ls->rcvbuf = addr->opt.rcvbuf; + ls->sndbuf = addr->opt.sndbuf; + + ls->keepalive = addr->opt.so_keepalive; +#if (NGX_HAVE_KEEPALIVE_TUNABLE) + ls->keepidle = addr->opt.tcp_keepidle; + ls->keepintvl = addr->opt.tcp_keepintvl; + ls->keepcnt = addr->opt.tcp_keepcnt; +#endif + +#if (NGX_HAVE_INET6) + ls->ipv6only = addr->opt.ipv6only; +#endif + +#if (NGX_HAVE_TCP_FASTOPEN) + ls->fastopen = addr->opt.fastopen; +#endif + +#if (NGX_HAVE_REUSEPORT) + ls->reuseport = addr->opt.reuseport; +#endif + + ls->wildcard = addr->opt.wildcard; + + return ls; +} + + +static ngx_int_t +ngx_stream_add_addrs(ngx_conf_t *cf, ngx_stream_port_t *stport, + ngx_stream_conf_addr_t *addr) +{ + ngx_uint_t i; + struct sockaddr_in *sin; + ngx_stream_in_addr_t *addrs; + ngx_stream_virtual_names_t *vn; + + stport->addrs = ngx_pcalloc(cf->pool, + stport->naddrs * sizeof(ngx_stream_in_addr_t)); + if (stport->addrs == NULL) { + return NGX_ERROR; + } + + addrs = stport->addrs; + + for (i = 0; i < stport->naddrs; i++) { + + sin = (struct sockaddr_in *) addr[i].opt.sockaddr; + addrs[i].addr = sin->sin_addr.s_addr; + addrs[i].conf.default_server = addr[i].default_server; +#if (NGX_STREAM_SSL) + addrs[i].conf.ssl = addr[i].opt.ssl; +#endif + addrs[i].conf.proxy_protocol = addr[i].opt.proxy_protocol; + + if (addr[i].hash.buckets == NULL + && (addr[i].wc_head == NULL + || addr[i].wc_head->hash.buckets == NULL) + && (addr[i].wc_tail == NULL + || addr[i].wc_tail->hash.buckets == NULL) +#if (NGX_PCRE) + && addr[i].nregex == 0 +#endif + ) + { + continue; + } + + vn = ngx_palloc(cf->pool, sizeof(ngx_stream_virtual_names_t)); + if (vn == NULL) { + return NGX_ERROR; + } + + addrs[i].conf.virtual_names = vn; + + vn->names.hash = addr[i].hash; + vn->names.wc_head = addr[i].wc_head; + vn->names.wc_tail = addr[i].wc_tail; +#if (NGX_PCRE) + vn->nregex = addr[i].nregex; + vn->regex = addr[i].regex; +#endif + } + + return NGX_OK; +} + + +#if (NGX_HAVE_INET6) + +static ngx_int_t +ngx_stream_add_addrs6(ngx_conf_t *cf, ngx_stream_port_t *stport, + ngx_stream_conf_addr_t *addr) +{ + ngx_uint_t i; + struct sockaddr_in6 *sin6; + ngx_stream_in6_addr_t *addrs6; + ngx_stream_virtual_names_t *vn; + + stport->addrs = ngx_pcalloc(cf->pool, + stport->naddrs * sizeof(ngx_stream_in6_addr_t)); + if (stport->addrs == NULL) { + return NGX_ERROR; + } + + addrs6 = stport->addrs; + + for (i = 0; i < stport->naddrs; i++) { + + sin6 = (struct sockaddr_in6 *) addr[i].opt.sockaddr; + addrs6[i].addr6 = sin6->sin6_addr; + addrs6[i].conf.default_server = addr[i].default_server; +#if (NGX_STREAM_SSL) + addrs6[i].conf.ssl = addr[i].opt.ssl; +#endif + addrs6[i].conf.proxy_protocol = addr[i].opt.proxy_protocol; + + if (addr[i].hash.buckets == NULL + && (addr[i].wc_head == NULL + || addr[i].wc_head->hash.buckets == NULL) + && (addr[i].wc_tail == NULL + || addr[i].wc_tail->hash.buckets == NULL) +#if (NGX_PCRE) + && addr[i].nregex == 0 +#endif From arut at nginx.com Thu Mar 21 13:40:57 2024 From: arut at nginx.com (=?utf-8?q?Roman_Arutyunyan?=) Date: Thu, 21 Mar 2024 13:40:57 +0000 Subject: [nginx] Stream: ngx_stream_pass_module. Message-ID: details: https://hg.nginx.org/nginx/rev/913518341c20 branches: changeset: 9217:913518341c20 user: Roman Arutyunyan date: Wed Feb 21 17:36:02 2024 +0400 description: Stream: ngx_stream_pass_module. The module allows to pass connections from Stream to other modules such as HTTP or Mail, as well as back to Stream. Previously, this was only possible with proxying. Connections with preread buffer read out from socket cannot be passed. The module allows selective SSL termination based on SNI. stream { server { listen 8000 default_server; ssl_preread on; ... } server { listen 8000; server_name foo.example.com; pass 127.0.0.1:8001; # to HTTP } server { listen 8000; server_name bar.example.com; ... } } http { server { listen 8001 ssl; ... location / { root html; } } } diffstat: auto/modules | 10 + auto/options | 3 + src/stream/ngx_stream_pass_module.c | 276 ++++++++++++++++++++++++++++++++++++ 3 files changed, 289 insertions(+), 0 deletions(-) diffs (327 lines): diff -r e28b044908cb -r 913518341c20 auto/modules --- a/auto/modules Thu Dec 14 21:58:39 2023 +0400 +++ b/auto/modules Wed Feb 21 17:36:02 2024 +0400 @@ -1166,6 +1166,16 @@ if [ $STREAM != NO ]; then . auto/module fi + if [ $STREAM_PASS = YES ]; then + ngx_module_name=ngx_stream_pass_module + ngx_module_deps= + ngx_module_srcs=src/stream/ngx_stream_pass_module.c + ngx_module_libs= + ngx_module_link=$STREAM_PASS + + . auto/module + fi + if [ $STREAM_SET = YES ]; then ngx_module_name=ngx_stream_set_module ngx_module_deps= diff -r e28b044908cb -r 913518341c20 auto/options --- a/auto/options Thu Dec 14 21:58:39 2023 +0400 +++ b/auto/options Wed Feb 21 17:36:02 2024 +0400 @@ -127,6 +127,7 @@ STREAM_GEOIP=NO STREAM_MAP=YES STREAM_SPLIT_CLIENTS=YES STREAM_RETURN=YES +STREAM_PASS=YES STREAM_SET=YES STREAM_UPSTREAM_HASH=YES STREAM_UPSTREAM_LEAST_CONN=YES @@ -337,6 +338,7 @@ use the \"--with-mail_ssl_module\" optio --without-stream_split_clients_module) STREAM_SPLIT_CLIENTS=NO ;; --without-stream_return_module) STREAM_RETURN=NO ;; + --without-stream_pass_module) STREAM_PASS=NO ;; --without-stream_set_module) STREAM_SET=NO ;; --without-stream_upstream_hash_module) STREAM_UPSTREAM_HASH=NO ;; @@ -556,6 +558,7 @@ cat << END --without-stream_split_clients_module disable ngx_stream_split_clients_module --without-stream_return_module disable ngx_stream_return_module + --without-stream_pass_module disable ngx_stream_pass_module --without-stream_set_module disable ngx_stream_set_module --without-stream_upstream_hash_module disable ngx_stream_upstream_hash_module diff -r e28b044908cb -r 913518341c20 src/stream/ngx_stream_pass_module.c --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/src/stream/ngx_stream_pass_module.c Wed Feb 21 17:36:02 2024 +0400 @@ -0,0 +1,276 @@ + +/* + * Copyright (C) Roman Arutyunyan + * Copyright (C) Nginx, Inc. + */ + + +#include +#include +#include + + +typedef struct { + ngx_addr_t *addr; + ngx_stream_complex_value_t *addr_value; +} ngx_stream_pass_srv_conf_t; + + +static void ngx_stream_pass_handler(ngx_stream_session_t *s); +static ngx_int_t ngx_stream_pass_match(ngx_listening_t *ls, ngx_addr_t *addr); +static void *ngx_stream_pass_create_srv_conf(ngx_conf_t *cf); +static char *ngx_stream_pass(ngx_conf_t *cf, ngx_command_t *cmd, void *conf); + + +static ngx_command_t ngx_stream_pass_commands[] = { + + { ngx_string("pass"), + NGX_STREAM_SRV_CONF|NGX_CONF_TAKE1, + ngx_stream_pass, + NGX_STREAM_SRV_CONF_OFFSET, + 0, + NULL }, + + ngx_null_command +}; + + +static ngx_stream_module_t ngx_stream_pass_module_ctx = { + NULL, /* preconfiguration */ + NULL, /* postconfiguration */ + + NULL, /* create main configuration */ + NULL, /* init main configuration */ + + ngx_stream_pass_create_srv_conf, /* create server configuration */ + NULL /* merge server configuration */ +}; + + +ngx_module_t ngx_stream_pass_module = { + NGX_MODULE_V1, + &ngx_stream_pass_module_ctx, /* module context */ + ngx_stream_pass_commands, /* module directives */ + NGX_STREAM_MODULE, /* module type */ + NULL, /* init master */ + NULL, /* init module */ + NULL, /* init process */ + NULL, /* init thread */ + NULL, /* exit thread */ + NULL, /* exit process */ + NULL, /* exit master */ + NGX_MODULE_V1_PADDING +}; + + +static void +ngx_stream_pass_handler(ngx_stream_session_t *s) +{ + ngx_url_t u; + ngx_str_t url; + ngx_addr_t *addr; + ngx_uint_t i; + ngx_listening_t *ls; + ngx_connection_t *c; + ngx_stream_pass_srv_conf_t *pscf; + + c = s->connection; + + c->log->action = "passing connection to port"; + + if (c->buffer && c->buffer->pos != c->buffer->last) { + ngx_log_error(NGX_LOG_ERR, c->log, 0, + "cannot pass connection with preread data"); + goto failed; + } + + pscf = ngx_stream_get_module_srv_conf(s, ngx_stream_pass_module); + + addr = pscf->addr; + + if (addr == NULL) { + if (ngx_stream_complex_value(s, pscf->addr_value, &url) != NGX_OK) { + goto failed; + } + + ngx_memzero(&u, sizeof(ngx_url_t)); + + u.url = url; + u.no_resolve = 1; + + if (ngx_parse_url(c->pool, &u) != NGX_OK) { + if (u.err) { + ngx_log_error(NGX_LOG_ERR, c->log, 0, + "%s in pass \"%V\"", u.err, &u.url); + } + + goto failed; + } + + if (u.naddrs == 0) { + ngx_log_error(NGX_LOG_ERR, c->log, 0, + "no addresses in pass \"%V\"", &u.url); + goto failed; + } + + if (u.no_port) { + ngx_log_error(NGX_LOG_ERR, c->log, 0, + "no port in pass \"%V\"", &u.url); + goto failed; + } + + addr = &u.addrs[0]; + } + + ngx_log_debug1(NGX_LOG_DEBUG_STREAM, c->log, 0, + "stream pass addr: \"%V\"", &addr->name); + + ls = ngx_cycle->listening.elts; + + for (i = 0; i < ngx_cycle->listening.nelts; i++) { + + if (ngx_stream_pass_match(&ls[i], addr) != NGX_OK) { + continue; + } + + c->listening = &ls[i]; + + c->data = NULL; + c->buffer = NULL; + + *c->log = c->listening->log; + c->log->handler = NULL; + c->log->data = NULL; + + c->local_sockaddr = addr->sockaddr; + c->local_socklen = addr->socklen; + + c->listening->handler(c); + + return; + } + + ngx_log_error(NGX_LOG_ERR, c->log, 0, + "port not found for \"%V\"", &addr->name); + + ngx_stream_finalize_session(s, NGX_STREAM_OK); + + return; + +failed: + + ngx_stream_finalize_session(s, NGX_STREAM_INTERNAL_SERVER_ERROR); +} + + +static ngx_int_t +ngx_stream_pass_match(ngx_listening_t *ls, ngx_addr_t *addr) +{ + if (!ls->wildcard) { + return ngx_cmp_sockaddr(ls->sockaddr, ls->socklen, + addr->sockaddr, addr->socklen, 1); + } + + if (ls->sockaddr->sa_family == addr->sockaddr->sa_family + && ngx_inet_get_port(ls->sockaddr) == ngx_inet_get_port(addr->sockaddr)) + { + return NGX_OK; + } + + return NGX_DECLINED; +} + + +static void * +ngx_stream_pass_create_srv_conf(ngx_conf_t *cf) +{ + ngx_stream_pass_srv_conf_t *conf; + + conf = ngx_pcalloc(cf->pool, sizeof(ngx_stream_pass_srv_conf_t)); + if (conf == NULL) { + return NULL; + } + + /* + * set by ngx_pcalloc(): + * + * conf->addr = NULL; + * conf->addr_value = NULL; + */ + + return conf; +} + + +static char * +ngx_stream_pass(ngx_conf_t *cf, ngx_command_t *cmd, void *conf) +{ + ngx_stream_pass_srv_conf_t *pscf = conf; + + ngx_url_t u; + ngx_str_t *value, *url; + ngx_stream_complex_value_t cv; + ngx_stream_core_srv_conf_t *cscf; + ngx_stream_compile_complex_value_t ccv; + + if (pscf->addr || pscf->addr_value) { + return "is duplicate"; + } + + cscf = ngx_stream_conf_get_module_srv_conf(cf, ngx_stream_core_module); + + cscf->handler = ngx_stream_pass_handler; + + value = cf->args->elts; + + url = &value[1]; + + ngx_memzero(&ccv, sizeof(ngx_stream_compile_complex_value_t)); + + ccv.cf = cf; + ccv.value = url; + ccv.complex_value = &cv; + + if (ngx_stream_compile_complex_value(&ccv) != NGX_OK) { + return NGX_CONF_ERROR; + } + + if (cv.lengths) { + pscf->addr_value = ngx_palloc(cf->pool, + sizeof(ngx_stream_complex_value_t)); + if (pscf->addr_value == NULL) { + return NGX_CONF_ERROR; + } + + *pscf->addr_value = cv; + + return NGX_CONF_OK; + } + + ngx_memzero(&u, sizeof(ngx_url_t)); + + u.url = *url; + u.no_resolve = 1; + + if (ngx_parse_url(cf->pool, &u) != NGX_OK) { + if (u.err) { + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, + "%s in \"%V\" of the \"pass\" directive", + u.err, &u.url); + } + + return NGX_CONF_ERROR; + } + + if (u.naddrs == 0) { + return "has no addresses"; + } + + if (u.no_port) { + return "has no port"; + } + + pscf->addr = &u.addrs[0]; + + return NGX_CONF_OK; +} From arut at nginx.com Thu Mar 21 14:56:07 2024 From: arut at nginx.com (Roman Arutyunyan) Date: Thu, 21 Mar 2024 18:56:07 +0400 Subject: [PATCH] Avoiding mixed socket families in PROXY protocol v1 (ticket #2594) In-Reply-To: <6leyh7kuhr5uwqf6mm4xvafnfkcunnlbe7klwjsbdif5do5ojh@3evbd5fwq5qx> References: <2f12c929527b2337c15e.1705920594@arut-laptop> <20240122154801.ycda4ie442ipzw6n@N00W24XTQX> <20240221132920.chmms5v3aekvmc2i@N00W24XTQX> <20240222151726.3dzpvvanswdqhbkh@N00W24XTQX> <20240311124415.67cz4puapxh266is@N00W24XTQX> <6leyh7kuhr5uwqf6mm4xvafnfkcunnlbe7klwjsbdif5do5ojh@3evbd5fwq5qx> Message-ID: <20240321145607.qvexxg2fpzbhujo5@N00W24XTQX> Hi, On Wed, Mar 13, 2024 at 09:08:43PM +0400, Sergey Kandaurov wrote: > On Mon, Mar 11, 2024 at 04:44:15PM +0400, Roman Arutyunyan wrote: > > Hi, > > > > On Wed, Mar 06, 2024 at 06:50:26PM +0400, Sergey Kandaurov wrote: > > > On Thu, Feb 22, 2024 at 07:17:26PM +0400, Roman Arutyunyan wrote: > > > > Hi, > > > > > > > > On Thu, Feb 22, 2024 at 01:59:25AM +0000, J Carter wrote: > > > > > Hello Roman, > > > > > > > > > > On Wed, 21 Feb 2024 17:29:52 +0400 > > > > > Roman Arutyunyan wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > On Wed, Jan 24, 2024 at 12:03:06AM +0300, Maxim Dounin wrote: > > > > > > > [..] > > > > > > > Also, as suggested, using the server address as obtained via PROXY > > > > > > > protocol from the client might be a better solution as long as the > > > > > > > client address was set via PROXY protocol (regardless of whether > > > > > > > address families match or not), and what users expect from the > > > > > > > "proty_protocol on;" when chaining stream proxies in the first > > > > > > > place. > > > > > > > > > > > Checking whether the address used in PROXY writer is in fact the address > > > > > > that was passed in the PROXY header, is complicated. This will either require > > > > > > setting a flag when PROXY address is set by realip, which is ugly. > > > > > > Another approach is checking if the client address written to a PROXY header > > > > > > matches the client address in the received PROXY header. However since > > > > > > currently PROXY protocol addresses are stored as text, and not all addresses > > > > > > have unique text repersentations, this approach would require refactoring all > > > > > > PROXY protocol code + realip modules to switch from text to sockaddr. > > > > > > Checking for the realip context should do the trick, it is created > > > once the client address was successfully substituted. We could then > > > parse the text repr of server address in the PROXY header and use it > > > in PROXY writer. This will allow not to break API. > > > Something like this should work called from the stream proxy module, > > > it allows to preserve an original server address in the PROXY header > > > keeping both addresses unmutated > > > (which seems to be the best way to solve this problem). > > > > > > : extern ngx_module_t ngx_stream_realip_module; > > > : > > > : if (ngx_stream_get_module_ctx(s, ngx_stream_realip_module)) { > > > : > > > : local_sockaddr = c->local_sockaddr; > > > : local_socklen = c->local_socklen; > > > : > > > : if (ngx_parse_addr(c->pool, &addr, c->proxy_protocol->dst_addr.data, > > > : c->proxy_protocol->dst_addr.len) > > > : != NGX_OK) > > > : { > > > : ngx_stream_proxy_finalize(s, NGX_STREAM_INTERNAL_SERVER_ERROR); > > > : return; > > > : } > > > : > > > : ngx_inet_set_port(addr.sockaddr, c->proxy_protocol->dst_port); > > > : > > > : c->local_sockaddr = addr.sockaddr; > > > : c->local_socklen = addr.socklen; > > > : } > > > : > > > : # call ngx_proxy_protocol_write(), other existing stuff > > > : > > > : if (ngx_stream_get_module_ctx(s, ngx_stream_realip_module)) { > > > : c->local_sockaddr = local_sockaddr; > > > : c->local_socklen = local_socklen; > > > : } > > > > > > Can't say I am excited from this code, though. > > > > Neither am I. > > > > > Another way is stick to the INADDR_ANY plan removing the server > > > address part from the PROXY header if not applicable. > > > > > > > > > > > > > > > I suggest that we follow the first plan (INADDR_ANY etc). > > > > > > I agree in general, but see below. > > > > > > > > > > > > > > > > [...] > > > > > > > > > > > > Updated patch attached. > > > > > > > > > > > > > # HG changeset patch > > > > # User Roman Arutyunyan > > > > # Date 1708522464 -14400 > > > > # Wed Feb 21 17:34:24 2024 +0400 > > > > # Node ID 2d9bb7b49d64576fa29a673133129f16de3cfbbe > > > > # Parent 44da04c2d4db94ad4eefa84b299e07c5fa4a00b9 > > > > Avoiding mixed socket families in PROXY protocol v1 (ticket #2010). > > > > > > > > When using realip module, remote and local addresses of a connection can belong > > > > to different address families. This previously resulted in generating PROXY > > > > protocol headers like this: > > > > > > > > PROXY TCP4 127.0.0.1 unix:/tmp/nginx1.sock 55544 0 > > > > > > Well, UNIX-domain socket addresses are somewhat rather exsotic, > > > especially in the PROXY protocol. I'd use an example with IPv4/IPv6. > > > > > > > > > > > The PROXY protocol v1 specification does not allow mixed families. The change > > > > substitutes server address with zero address in this case: > > > > > > > > PROXY TCP4 127.0.0.1 0.0.0.0 55544 0 > > > > > > > > As an alternative, "PROXY UNKNOWN" header could be used, which unlike this > > > > header does not contain any useful information about the client. > > > > > > > > Also, the above mentioned format for unix socket address is not specified in > > > > PROXY protocol v1 and is a by-product of internal nginx representation of it. > > > > The change eliminates such addresses from PROXY protocol headers as well. > > > > > > > > diff --git a/src/core/ngx_proxy_protocol.c b/src/core/ngx_proxy_protocol.c > > > > --- a/src/core/ngx_proxy_protocol.c > > > > +++ b/src/core/ngx_proxy_protocol.c > > > > @@ -279,7 +279,13 @@ ngx_proxy_protocol_read_port(u_char *p, > > > > u_char * > > > > ngx_proxy_protocol_write(ngx_connection_t *c, u_char *buf, u_char *last) > > > > { > > > > - ngx_uint_t port, lport; > > > > + socklen_t local_socklen; > > > > + ngx_uint_t port, lport; > > > > + struct sockaddr *local_sockaddr; > > > > + struct sockaddr_in sin; > > > > +#if (NGX_HAVE_INET6) > > > > + struct sockaddr_in6 sin6; > > > > +#endif > > > > > > > > if (last - buf < NGX_PROXY_PROTOCOL_V1_MAX_HEADER) { > > > > ngx_log_error(NGX_LOG_ALERT, c->log, 0, > > > > @@ -312,11 +318,35 @@ ngx_proxy_protocol_write(ngx_connection_ > > > > > > > > *buf++ = ' '; > > > > > > > > - buf += ngx_sock_ntop(c->local_sockaddr, c->local_socklen, buf, last - buf, > > > > - 0); > > > > + if (c->sockaddr->sa_family == c->local_sockaddr->sa_family) { > > > > + local_sockaddr = c->local_sockaddr; > > > > + local_socklen = c->local_socklen; > > > > + > > > > + } else { > > > > + switch (c->sockaddr->sa_family) { > > > > + > > > > +#if (NGX_HAVE_INET6) > > > > + case AF_INET6: > > > > + ngx_memzero(&sin6, sizeof(struct sockaddr_in6)); > > > > + sin6.sin6_family = AF_INET6; > > > > + local_sockaddr = (struct sockaddr *) &sin6; > > > > + local_socklen = sizeof(struct sockaddr_in6); > > > > + break; > > > > +#endif > > > > + > > > > + default: /* AF_INET */ > > > > + ngx_memzero(&sin, sizeof(struct sockaddr)); > > > > > > Note that although wildcard adresses are typically implemented > > > using all-zeroes, this is not obligated by standards and hence > > > is an implementation detail. POSIX mandates just the following: > > > > > > : The header shall define the following symbolic > > > : constant for use as a local address in the structure passed to bind(): > > > : > > > : INADDR_ANY > > > : IPv4 wildcard address. > > > > > > (Note: it is "IPv4 local host address." in previous editions.) > > > > > > : The header shall declare the following external > > > : variable: > > > : > > > : const struct in6_addr in6addr_any > > > : > > > : This variable is initialized by the system to contain the wildcard > > > : IPv6 address. The header also defines the > > > : IN6ADDR_ANY_INIT macro. This macro must be constant at compile time > > > : and can be used to initialize a variable of type struct in6_addr to > > > : the IPv6 wildcard address. > > > > > > : RATIONALE > > > : > > > : The INADDR_ANY and INADDR_BROADCAST values are byte-order-neutral > > > : and thus their byte order is not specified. Many implementations > > > : have additional constants as extensions, such as INADDR_LOOPBACK, > > > : that are not byte-order-neutral. Traditionally, these constants are > > > : in host byte order, requiring the use of htonl() when using them in > > > : a sockaddr_in structure. > > > > > > So it seems quite legitimate to use a different binary representation. > > > To be on the safe side (and improve code readability), it makes sense > > > to consistently set wildcard addresses explicitly, using INADDR_ANY > > > and in6addr_any. It is not so important in this patch though, because > > > addresses do not cross the kernel, used solely for ngx_sock_ntop(). > > > > > > > + sin.sin_family = AF_INET; > > > > + local_sockaddr = (struct sockaddr *) &sin; > > > > + local_socklen = sizeof(struct sockaddr_in); > > > > + break; > > > > + } > > > > + } > > > > + > > > > + buf += ngx_sock_ntop(local_sockaddr, local_socklen, buf, last - buf, 0); > > > > > > > > port = ngx_inet_get_port(c->sockaddr); > > > > - lport = ngx_inet_get_port(c->local_sockaddr); > > > > + lport = ngx_inet_get_port(local_sockaddr); > > > > > > > > return ngx_slprintf(buf, last, " %ui %ui" CRLF, port, lport); > > > > } > > > > > > Picking INADDR_ANY (and IPv6 equivalent) to substitute non-applicable > > > addresses due to mismatching address family may be unnecessarily > > > strict hiding addresses that can be represented in a different family. > > > > > > IPv4 address can be represented in the IPv4-mapped IPv6 address format. > > > IPv6 address can be represented in the IPv4 format if it's IPv4-mapped. > > > > What if it's not? > > This is not a silver bullet. > If not, such address is replaced with INADDR_ANY (or IPv6 equivalent). > > [..] > > > I agree we can implement a few more workarounds and make it work. > > It all looks like we need a normal solution even if the patch will be bigger. > > Attached is a patch converting PROXY protocol addresses from text to binary > > representation. IMHO we should've done this long ago. Once it's done, > > the mixed family patch will be trivial, attached as well. > > > > -- > > Roman Arutyunyan > > > # HG changeset patch > > # User Roman Arutyunyan > > # Date 1709913161 -14400 > > # Fri Mar 08 19:52:41 2024 +0400 > > # Node ID 07c8c445746c6c74250bad7b42cd26dbb0ed6dd3 > > # Parent 2ed3f57dca0a664340bca2236c7d614902db4180 > > Storing PROXY protocol addresses as sockaddr. > > > > Previously they were stored as text. This was beneficial for v1 where the > > Nitpicking on log description: > > s/was/is/ ? > > Using text format is still beneficial for v1. > > > addresses do arrive as text, however counterproductive for v2, where they > > - using emphasising "do" verb here looks unreasonable > - inconsistent comma before "where" > > > arrive as binary. > > > > The change allows more flexibility when checking whether PROXY protocol > > address was set as client address. This will be needed in the followup changes. > > > > Also, PROXY protocol debugging is improved. Now TLVs are logged after parsing. > > While logging certainly looks more convenient now, because all TLVs are > logged at once, this somewhat duplicates logging of $proxy_protocol_tlv_ > variable evaluation, see ngx_proxy_protocol_get_tlv(). > IMHO, it makes sense to remove it from there. > > > > > diff --git a/src/core/ngx_proxy_protocol.c b/src/core/ngx_proxy_protocol.c > > --- a/src/core/ngx_proxy_protocol.c > > +++ b/src/core/ngx_proxy_protocol.c > > @@ -67,9 +67,12 @@ typedef struct { > > > > > > static u_char *ngx_proxy_protocol_read_addr(ngx_connection_t *c, u_char *p, > > - u_char *last, ngx_str_t *addr); > > + u_char *last, ngx_addr_t *addr); > > static u_char *ngx_proxy_protocol_read_port(u_char *p, u_char *last, > > - in_port_t *port, u_char sep); > > + struct sockaddr *sa, u_char sep); > > +#if (NGX_DEBUG) > > +static void ngx_proxy_protocol_debug(ngx_connection_t *c, ngx_uint_t version); > > I suggest to rename it to ngx_proxy_protocol_log(), > for consistency with ngx_ssl_handshake_log(), > and place below all callers including ngx_proxy_protocol_v2_read(). > > > +#endif > > static u_char *ngx_proxy_protocol_v2_read(ngx_connection_t *c, u_char *buf, > > u_char *last); > > static ngx_int_t ngx_proxy_protocol_lookup_tlv(ngx_connection_t *c, > > @@ -101,6 +104,7 @@ ngx_proxy_protocol_read(ngx_connection_t > > { > > size_t len; > > u_char *p; > > + ngx_addr_t addr; > > ngx_proxy_protocol_t *pp; > > > > static const u_char signature[] = "\r\n\r\n\0\r\nQUIT\n"; > > @@ -141,22 +145,28 @@ ngx_proxy_protocol_read(ngx_connection_t > > return NULL; > > } > > > > - p = ngx_proxy_protocol_read_addr(c, p, last, &pp->src_addr); > > + p = ngx_proxy_protocol_read_addr(c, p, last, &addr); > > if (p == NULL) { > > goto invalid; > > } > > > > - p = ngx_proxy_protocol_read_addr(c, p, last, &pp->dst_addr); > > + pp->src_sockaddr = addr.sockaddr; > > + pp->src_socklen = addr.socklen; > > + > > + p = ngx_proxy_protocol_read_addr(c, p, last, &addr); > > if (p == NULL) { > > goto invalid; > > } > > > > - p = ngx_proxy_protocol_read_port(p, last, &pp->src_port, ' '); > > + pp->dst_sockaddr = addr.sockaddr; > > + pp->dst_socklen = addr.socklen; > > + > > + p = ngx_proxy_protocol_read_port(p, last, pp->src_sockaddr, ' '); > > if (p == NULL) { > > goto invalid; > > } > > > > - p = ngx_proxy_protocol_read_port(p, last, &pp->dst_port, CR); > > + p = ngx_proxy_protocol_read_port(p, last, pp->dst_sockaddr, CR); > > if (p == NULL) { > > goto invalid; > > } > > @@ -169,11 +179,11 @@ ngx_proxy_protocol_read(ngx_connection_t > > goto invalid; > > } > > > > - ngx_log_debug4(NGX_LOG_DEBUG_CORE, c->log, 0, > > - "PROXY protocol src: %V %d, dst: %V %d", > > - &pp->src_addr, pp->src_port, &pp->dst_addr, pp->dst_port); > > + c->proxy_protocol = pp; > > > > - c->proxy_protocol = pp; > > +#if (NGX_DEBUG) > > + ngx_proxy_protocol_debug(c, 1); > > +#endif > > > > return p; > > > > @@ -202,7 +212,7 @@ invalid: > > > > static u_char * > > ngx_proxy_protocol_read_addr(ngx_connection_t *c, u_char *p, u_char *last, > > - ngx_str_t *addr) > > + ngx_addr_t *addr) > > { > > size_t len; > > u_char ch, *pos; > > @@ -231,20 +241,16 @@ ngx_proxy_protocol_read_addr(ngx_connect > > > > len = p - pos - 1; > > > > - addr->data = ngx_pnalloc(c->pool, len); > > - if (addr->data == NULL) { > > + if (ngx_parse_addr(c->pool, addr, pos, len) != NGX_OK) { > > return NULL; > > } > > > > - ngx_memcpy(addr->data, pos, len); > > - addr->len = len; > > - > > return p; > > } > > > > > > static u_char * > > -ngx_proxy_protocol_read_port(u_char *p, u_char *last, in_port_t *port, > > +ngx_proxy_protocol_read_port(u_char *p, u_char *last, struct sockaddr *sa, > > u_char sep) > > { > > size_t len; > > @@ -270,11 +276,68 @@ ngx_proxy_protocol_read_port(u_char *p, > > return NULL; > > } > > > > - *port = (in_port_t) n; > > + ngx_inet_set_port(sa, n); > > "n" still needs to be casted to "in_port_t". > > This used to fix build on MSVC in the past. > See ngx_parse_addr_port(), which is very similar. > > > > > return p; > > } > > style: two blank lines here > > > > > +#if (NGX_DEBUG) > > + > > +static void > > +ngx_proxy_protocol_debug(ngx_connection_t *c, ngx_uint_t version) > > +{ > > + u_char *p; > > + size_t n, len; > > + ngx_str_t src_addr, dst_addr; > > + ngx_proxy_protocol_tlv_t *tlv; > > + u_char src_text[NGX_SOCKADDR_STRLEN]; > > + u_char dst_text[NGX_SOCKADDR_STRLEN]; > > + > > + if ((c->log->log_level & NGX_LOG_DEBUG_CORE) == 0) { > > + return; > > + } > > + > > + src_addr.data = src_text; > > + src_addr.len = ngx_sock_ntop(c->proxy_protocol->src_sockaddr, > > + c->proxy_protocol->src_socklen, > > + src_text, NGX_SOCKADDR_STRLEN, 1); > > + > > + dst_addr.data = dst_text; > > + dst_addr.len = ngx_sock_ntop(c->proxy_protocol->dst_sockaddr, > > + c->proxy_protocol->dst_socklen, > > + dst_text, NGX_SOCKADDR_STRLEN, 1); > > + > > + ngx_log_debug3(NGX_LOG_DEBUG_CORE, c->log, 0, > > + "PROXY protocol v%ui %V %V", version, &src_addr, &dst_addr); > > + > > + p = c->proxy_protocol->tlvs.data; > > + n = c->proxy_protocol->tlvs.len; > > + > > + for ( ;; ) { > > + if (n < sizeof(ngx_proxy_protocol_tlv_t)) { > > + break; > > + } > > + > > + tlv = (ngx_proxy_protocol_tlv_t *) p; > > + len = ngx_proxy_protocol_parse_uint16(tlv->len); > > + > > + p += sizeof(ngx_proxy_protocol_tlv_t); > > + n -= sizeof(ngx_proxy_protocol_tlv_t); > > + > > + if (n < len) { > > + break; > > + } > > + > > + ngx_log_debug3(NGX_LOG_DEBUG_CORE, c->log, 0, > > + "PROXY protocol tlv:%02xi %*xs", > > + (ngx_uint_t) tlv->type, len, p); > > + > > + p += len; > > + n -= len; > > + } > > +} > > + > > +#endif > > and there > > > > > u_char * > > ngx_proxy_protocol_write(ngx_connection_t *c, u_char *buf, u_char *last) > > @@ -327,13 +390,13 @@ ngx_proxy_protocol_v2_read(ngx_connectio > > { > > u_char *end; > > size_t len; > > - socklen_t socklen; > > ngx_uint_t version, command, family, transport; > > - ngx_sockaddr_t src_sockaddr, dst_sockaddr; > > + struct sockaddr_in *sin; > > ngx_proxy_protocol_t *pp; > > ngx_proxy_protocol_header_t *header; > > ngx_proxy_protocol_inet_addrs_t *in; > > #if (NGX_HAVE_INET6) > > + struct sockaddr_in6 *sin6; > > ngx_proxy_protocol_inet6_addrs_t *in6; > > #endif > > > > @@ -394,18 +457,29 @@ ngx_proxy_protocol_v2_read(ngx_connectio > > > > in = (ngx_proxy_protocol_inet_addrs_t *) buf; > > > > - src_sockaddr.sockaddr_in.sin_family = AF_INET; > > - src_sockaddr.sockaddr_in.sin_port = 0; > > - ngx_memcpy(&src_sockaddr.sockaddr_in.sin_addr, in->src_addr, 4); > > + sin = ngx_pcalloc(c->pool, sizeof(struct sockaddr_in)); > > + if (sin == NULL) { > > + return NULL; > > + } > > + > > + sin->sin_family = AF_INET; > > + ngx_memcpy(&sin->sin_addr, in->src_addr, 4); > > + sin->sin_port = htons(ngx_proxy_protocol_parse_uint16(in->src_port)); > > + > > + pp->src_sockaddr = (struct sockaddr *) sin; > > + pp->src_socklen = sizeof(struct sockaddr_in); > > > > - dst_sockaddr.sockaddr_in.sin_family = AF_INET; > > - dst_sockaddr.sockaddr_in.sin_port = 0; > > - ngx_memcpy(&dst_sockaddr.sockaddr_in.sin_addr, in->dst_addr, 4); > > + sin = ngx_pcalloc(c->pool, sizeof(struct sockaddr_in)); > > + if (sin == NULL) { > > + return NULL; > > + } > > > > - pp->src_port = ngx_proxy_protocol_parse_uint16(in->src_port); > > - pp->dst_port = ngx_proxy_protocol_parse_uint16(in->dst_port); > > + sin->sin_family = AF_INET; > > + ngx_memcpy(&sin->sin_addr, in->dst_addr, 4); > > + sin->sin_port = htons(ngx_proxy_protocol_parse_uint16(in->dst_port)); > > > > - socklen = sizeof(struct sockaddr_in); > > + pp->dst_sockaddr = (struct sockaddr *) sin; > > + pp->dst_socklen = sizeof(struct sockaddr_in); > > > > buf += sizeof(ngx_proxy_protocol_inet_addrs_t); > > > > @@ -421,18 +495,29 @@ ngx_proxy_protocol_v2_read(ngx_connectio > > > > in6 = (ngx_proxy_protocol_inet6_addrs_t *) buf; > > > > - src_sockaddr.sockaddr_in6.sin6_family = AF_INET6; > > - src_sockaddr.sockaddr_in6.sin6_port = 0; > > - ngx_memcpy(&src_sockaddr.sockaddr_in6.sin6_addr, in6->src_addr, 16); > > + sin6 = ngx_pcalloc(c->pool, sizeof(struct sockaddr_in6)); > > + if (sin6 == NULL) { > > + return NULL; > > + } > > + > > + sin6->sin6_family = AF_INET6; > > + ngx_memcpy(&sin6->sin6_addr, in6->src_addr, 16); > > + sin6->sin6_port = htons(ngx_proxy_protocol_parse_uint16(in6->src_port)); > > + > > + pp->src_sockaddr = (struct sockaddr *) sin6; > > + pp->src_socklen = sizeof(struct sockaddr_in6); > > > > - dst_sockaddr.sockaddr_in6.sin6_family = AF_INET6; > > - dst_sockaddr.sockaddr_in6.sin6_port = 0; > > - ngx_memcpy(&dst_sockaddr.sockaddr_in6.sin6_addr, in6->dst_addr, 16); > > + sin6 = ngx_pcalloc(c->pool, sizeof(struct sockaddr_in6)); > > + if (sin6 == NULL) { > > + return NULL; > > + } > > > > - pp->src_port = ngx_proxy_protocol_parse_uint16(in6->src_port); > > - pp->dst_port = ngx_proxy_protocol_parse_uint16(in6->dst_port); > > + sin6->sin6_family = AF_INET6; > > + ngx_memcpy(&sin6->sin6_addr, in6->dst_addr, 16); > > + sin6->sin6_port = htons(ngx_proxy_protocol_parse_uint16(in6->dst_port)); > > > > - socklen = sizeof(struct sockaddr_in6); > > + pp->dst_sockaddr = (struct sockaddr *) sin6; > > + pp->dst_socklen = sizeof(struct sockaddr_in6); > > > > buf += sizeof(ngx_proxy_protocol_inet6_addrs_t); > > > > @@ -447,26 +532,6 @@ ngx_proxy_protocol_v2_read(ngx_connectio > > return end; > > } > > > > - pp->src_addr.data = ngx_pnalloc(c->pool, NGX_SOCKADDR_STRLEN); > > - if (pp->src_addr.data == NULL) { > > - return NULL; > > - } > > - > > - pp->src_addr.len = ngx_sock_ntop(&src_sockaddr.sockaddr, socklen, > > - pp->src_addr.data, NGX_SOCKADDR_STRLEN, 0); > > - > > - pp->dst_addr.data = ngx_pnalloc(c->pool, NGX_SOCKADDR_STRLEN); > > - if (pp->dst_addr.data == NULL) { > > - return NULL; > > - } > > - > > - pp->dst_addr.len = ngx_sock_ntop(&dst_sockaddr.sockaddr, socklen, > > - pp->dst_addr.data, NGX_SOCKADDR_STRLEN, 0); > > - > > - ngx_log_debug4(NGX_LOG_DEBUG_CORE, c->log, 0, > > - "PROXY protocol v2 src: %V %d, dst: %V %d", > > - &pp->src_addr, pp->src_port, &pp->dst_addr, pp->dst_port); > > - > > if (buf < end) { > > pp->tlvs.data = ngx_pnalloc(c->pool, end - buf); > > if (pp->tlvs.data == NULL) { > > @@ -479,6 +544,10 @@ ngx_proxy_protocol_v2_read(ngx_connectio > > > > c->proxy_protocol = pp; > > > > +#if (NGX_DEBUG) > > + ngx_proxy_protocol_debug(c, 2); > > +#endif > > + > > return end; > > } > > > > diff --git a/src/core/ngx_proxy_protocol.h b/src/core/ngx_proxy_protocol.h > > --- a/src/core/ngx_proxy_protocol.h > > +++ b/src/core/ngx_proxy_protocol.h > > @@ -18,10 +18,12 @@ > > > > > > struct ngx_proxy_protocol_s { > > - ngx_str_t src_addr; > > - ngx_str_t dst_addr; > > - in_port_t src_port; > > - in_port_t dst_port; > > + struct sockaddr *src_sockaddr; > > + struct sockaddr *dst_sockaddr; > > + > > + socklen_t src_socklen; > > + socklen_t dst_socklen; > > + > > ngx_str_t tlvs; > > }; > > > > diff --git a/src/http/modules/ngx_http_realip_module.c b/src/http/modules/ngx_http_realip_module.c > > --- a/src/http/modules/ngx_http_realip_module.c > > +++ b/src/http/modules/ngx_http_realip_module.c > > @@ -152,6 +152,8 @@ ngx_http_realip_handler(ngx_http_request > > return NGX_DECLINED; > > } > > > > + c = r->connection; > > + > > switch (rlcf->type) { > > > > case NGX_HTTP_REALIP_XREALIP: > > @@ -179,14 +181,18 @@ ngx_http_realip_handler(ngx_http_request > > > > case NGX_HTTP_REALIP_PROXY: > > > > - if (r->connection->proxy_protocol == NULL) { > > + if (c->proxy_protocol == NULL) { > > return NGX_DECLINED; > > } > > > > - value = &r->connection->proxy_protocol->src_addr; > > - xfwd = NULL; > > + if (ngx_cidr_match(c->sockaddr, rlcf->from) != NGX_OK) { > > + return NGX_DECLINED; > > + } > > > > - break; > > + addr.sockaddr = c->proxy_protocol->src_sockaddr; > > + addr.socklen = c->proxy_protocol->src_socklen; > > + > > + return ngx_http_realip_set_addr(r, &addr); > > > > default: /* NGX_HTTP_REALIP_HEADER */ > > > > @@ -225,8 +231,6 @@ ngx_http_realip_handler(ngx_http_request > > > > found: > > > > - c = r->connection; > > - > > addr.sockaddr = c->sockaddr; > > addr.socklen = c->socklen; > > /* addr.name = c->addr_text; */ > > @@ -235,10 +239,6 @@ found: > > rlcf->recursive) > > != NGX_DECLINED) > > { > > - if (rlcf->type == NGX_HTTP_REALIP_PROXY) { > > - ngx_inet_set_port(addr.sockaddr, c->proxy_protocol->src_port); > > - } > > - > > return ngx_http_realip_set_addr(r, &addr); > > } > > > > diff --git a/src/http/ngx_http_variables.c b/src/http/ngx_http_variables.c > > --- a/src/http/ngx_http_variables.c > > +++ b/src/http/ngx_http_variables.c > > @@ -201,20 +201,16 @@ static ngx_http_variable_t ngx_http_cor > > { ngx_string("remote_port"), NULL, ngx_http_variable_remote_port, 0, 0, 0 }, > > > > { ngx_string("proxy_protocol_addr"), NULL, > > - ngx_http_variable_proxy_protocol_addr, > > - offsetof(ngx_proxy_protocol_t, src_addr), 0, 0 }, > > + ngx_http_variable_proxy_protocol_addr, 0, 0, 0 }, > > > > { ngx_string("proxy_protocol_port"), NULL, > > - ngx_http_variable_proxy_protocol_port, > > - offsetof(ngx_proxy_protocol_t, src_port), 0, 0 }, > > + ngx_http_variable_proxy_protocol_port, 0, 0, 0 }, > > > > { ngx_string("proxy_protocol_server_addr"), NULL, > > - ngx_http_variable_proxy_protocol_addr, > > - offsetof(ngx_proxy_protocol_t, dst_addr), 0, 0 }, > > + ngx_http_variable_proxy_protocol_addr, 1, 0, 0 }, > > > > { ngx_string("proxy_protocol_server_port"), NULL, > > - ngx_http_variable_proxy_protocol_port, > > - offsetof(ngx_proxy_protocol_t, dst_port), 0, 0 }, > > + ngx_http_variable_proxy_protocol_port, 1, 0, 0 }, > > > > { ngx_string("proxy_protocol_tlv_"), NULL, > > ngx_http_variable_proxy_protocol_tlv, > > @@ -1340,7 +1336,8 @@ static ngx_int_t > > ngx_http_variable_proxy_protocol_addr(ngx_http_request_t *r, > > ngx_http_variable_value_t *v, uintptr_t data) > > { > > - ngx_str_t *addr; > > + u_char *p; > > + size_t n; > > ngx_proxy_protocol_t *pp; > > > > pp = r->connection->proxy_protocol; > > @@ -1349,13 +1346,25 @@ ngx_http_variable_proxy_protocol_addr(ng > > return NGX_OK; > > } > > > > - addr = (ngx_str_t *) ((char *) pp + data); > > - > > - v->len = addr->len; > > + p = ngx_pnalloc(r->connection->pool, NGX_SOCKADDR_STRLEN); > > NGX_SOCKADDR_STRLEN includes UNIX-domain sockets as well, which means > ~100 bytes memory over-allocation, with such addresses prohibited in > the PROXY protocol. It can be saved if allocate the actual size > returned from ngx_sock_ntop(). > > > + if (p == NULL) { > > + return NGX_ERROR; > > + } > > + > > + if (data) { > > + n = ngx_sock_ntop(pp->dst_sockaddr, pp->dst_socklen, > > + p, NGX_SOCKADDR_STRLEN, 0); > > + > > + } else { > > + n = ngx_sock_ntop(pp->src_sockaddr, pp->src_socklen, > > + p, NGX_SOCKADDR_STRLEN, 0); > > + } > > + > > + v->len = n; > > v->valid = 1; > > v->no_cacheable = 0; > > v->not_found = 0; > > - v->data = addr->data; > > + v->data = p; > > > > return NGX_OK; > > } > > @@ -1384,7 +1393,12 @@ ngx_http_variable_proxy_protocol_port(ng > > return NGX_ERROR; > > } > > > > - port = *(in_port_t *) ((char *) pp + data); > > + if (data) { > > + port = ngx_inet_get_port(pp->dst_sockaddr); > > + > > + } else { > > + port = ngx_inet_get_port(pp->src_sockaddr); > > + } > > > > if (port > 0 && port < 65536) { > > v->len = ngx_sprintf(v->data, "%ui", port) - v->data; > > What I don't get is why we parse zero port as valid (see ngx_atoi() > call in ngx_proxy_protocol_read_port()) but reject evaluating it as > such in variable handlers. IMHO, it makes sense to remove this > restriction and write the port value unconditionally, including "0", > especially that after storing port as binary it cannot be invalid. > > > diff --git a/src/mail/ngx_mail_auth_http_module.c b/src/mail/ngx_mail_auth_http_module.c > > --- a/src/mail/ngx_mail_auth_http_module.c > > +++ b/src/mail/ngx_mail_auth_http_module.c > > @@ -1249,11 +1249,11 @@ ngx_mail_auth_http_create_request(ngx_ma > > > > if (c->proxy_protocol) { > > len += sizeof("Proxy-Protocol-Addr: ") - 1 > > - + c->proxy_protocol->src_addr.len + sizeof(CRLF) - 1 > > + + NGX_SOCKADDR_STRLEN + sizeof(CRLF) - 1 > > See above about wasted memory. > > > + sizeof("Proxy-Protocol-Port: ") - 1 > > + sizeof("65535") - 1 + sizeof(CRLF) - 1 > > + sizeof("Proxy-Protocol-Server-Addr: ") - 1 > > - + c->proxy_protocol->dst_addr.len + sizeof(CRLF) - 1 > > + + NGX_SOCKADDR_STRLEN + sizeof(CRLF) - 1 > > + sizeof("Proxy-Protocol-Server-Port: ") - 1 > > + sizeof("65535") - 1 + sizeof(CRLF) - 1; > > } > > @@ -1352,21 +1352,23 @@ ngx_mail_auth_http_create_request(ngx_ma > > if (c->proxy_protocol) { > > b->last = ngx_cpymem(b->last, "Proxy-Protocol-Addr: ", > > sizeof("Proxy-Protocol-Addr: ") - 1); > > - b->last = ngx_copy(b->last, c->proxy_protocol->src_addr.data, > > - c->proxy_protocol->src_addr.len); > > + b->last += ngx_sock_ntop(c->proxy_protocol->src_sockaddr, > > + c->proxy_protocol->src_socklen, > > + b->last, b->end - b->last, 0); > > *b->last++ = CR; *b->last++ = LF; > > > > b->last = ngx_sprintf(b->last, "Proxy-Protocol-Port: %d" CRLF, > > - c->proxy_protocol->src_port); > > + (int) ngx_inet_get_port(c->proxy_protocol->src_sockaddr)); > > > > b->last = ngx_cpymem(b->last, "Proxy-Protocol-Server-Addr: ", > > sizeof("Proxy-Protocol-Server-Addr: ") - 1); > > - b->last = ngx_copy(b->last, c->proxy_protocol->dst_addr.data, > > - c->proxy_protocol->dst_addr.len); > > + b->last += ngx_sock_ntop(c->proxy_protocol->dst_sockaddr, > > + c->proxy_protocol->dst_socklen, > > + b->last, b->end - b->last, 0); > > *b->last++ = CR; *b->last++ = LF; > > > > b->last = ngx_sprintf(b->last, "Proxy-Protocol-Server-Port: %d" CRLF, > > - c->proxy_protocol->dst_port); > > + (int) ngx_inet_get_port(c->proxy_protocol->dst_sockaddr)); > > } > > > > if (s->auth_method == NGX_MAIL_AUTH_NONE) { > > diff --git a/src/mail/ngx_mail_realip_module.c b/src/mail/ngx_mail_realip_module.c > > --- a/src/mail/ngx_mail_realip_module.c > > +++ b/src/mail/ngx_mail_realip_module.c > > @@ -87,14 +87,8 @@ ngx_mail_realip_handler(ngx_mail_session > > return NGX_OK; > > } > > > > - if (ngx_parse_addr(c->pool, &addr, c->proxy_protocol->src_addr.data, > > - c->proxy_protocol->src_addr.len) > > - != NGX_OK) > > - { > > - return NGX_OK; > > - } > > - > > - ngx_inet_set_port(addr.sockaddr, c->proxy_protocol->src_port); > > + addr.sockaddr = c->proxy_protocol->src_sockaddr; > > + addr.socklen = c->proxy_protocol->src_socklen; > > > > return ngx_mail_realip_set_addr(s, &addr); > > } > > diff --git a/src/stream/ngx_stream_realip_module.c b/src/stream/ngx_stream_realip_module.c > > --- a/src/stream/ngx_stream_realip_module.c > > +++ b/src/stream/ngx_stream_realip_module.c > > @@ -116,14 +116,8 @@ ngx_stream_realip_handler(ngx_stream_ses > > return NGX_DECLINED; > > } > > > > - if (ngx_parse_addr(c->pool, &addr, c->proxy_protocol->src_addr.data, > > - c->proxy_protocol->src_addr.len) > > - != NGX_OK) > > - { > > - return NGX_DECLINED; > > - } > > - > > - ngx_inet_set_port(addr.sockaddr, c->proxy_protocol->src_port); > > + addr.sockaddr = c->proxy_protocol->src_sockaddr; > > + addr.socklen = c->proxy_protocol->src_socklen; > > > > return ngx_stream_realip_set_addr(s, &addr); > > } > > diff --git a/src/stream/ngx_stream_variables.c b/src/stream/ngx_stream_variables.c > > --- a/src/stream/ngx_stream_variables.c > > +++ b/src/stream/ngx_stream_variables.c > > @@ -66,20 +66,16 @@ static ngx_stream_variable_t ngx_stream > > ngx_stream_variable_remote_port, 0, 0, 0 }, > > > > { ngx_string("proxy_protocol_addr"), NULL, > > - ngx_stream_variable_proxy_protocol_addr, > > - offsetof(ngx_proxy_protocol_t, src_addr), 0, 0 }, > > + ngx_stream_variable_proxy_protocol_addr, 0, 0, 0 }, > > > > { ngx_string("proxy_protocol_port"), NULL, > > - ngx_stream_variable_proxy_protocol_port, > > - offsetof(ngx_proxy_protocol_t, src_port), 0, 0 }, > > + ngx_stream_variable_proxy_protocol_port, 0, 0, 0 }, > > > > { ngx_string("proxy_protocol_server_addr"), NULL, > > - ngx_stream_variable_proxy_protocol_addr, > > - offsetof(ngx_proxy_protocol_t, dst_addr), 0, 0 }, > > + ngx_stream_variable_proxy_protocol_addr, 1, 0, 0 }, > > > > { ngx_string("proxy_protocol_server_port"), NULL, > > - ngx_stream_variable_proxy_protocol_port, > > - offsetof(ngx_proxy_protocol_t, dst_port), 0, 0 }, > > + ngx_stream_variable_proxy_protocol_port, 1, 0, 0 }, > > > > { ngx_string("proxy_protocol_tlv_"), NULL, > > ngx_stream_variable_proxy_protocol_tlv, > > @@ -573,7 +569,8 @@ static ngx_int_t > > ngx_stream_variable_proxy_protocol_addr(ngx_stream_session_t *s, > > ngx_stream_variable_value_t *v, uintptr_t data) > > { > > - ngx_str_t *addr; > > + u_char *p; > > + size_t n; > > ngx_proxy_protocol_t *pp; > > > > pp = s->connection->proxy_protocol; > > @@ -582,13 +579,25 @@ ngx_stream_variable_proxy_protocol_addr( > > return NGX_OK; > > } > > > > - addr = (ngx_str_t *) ((char *) pp + data); > > + p = ngx_pnalloc(s->connection->pool, NGX_SOCKADDR_STRLEN); > > + if (p == NULL) { > > + return NGX_ERROR; > > + } > > > > - v->len = addr->len; > > + if (data) { > > + n = ngx_sock_ntop(pp->dst_sockaddr, pp->dst_socklen, > > + p, NGX_SOCKADDR_STRLEN, 0); > > + > > + } else { > > + n = ngx_sock_ntop(pp->src_sockaddr, pp->src_socklen, > > + p, NGX_SOCKADDR_STRLEN, 0); > > + } > > + > > + v->len = n; > > v->valid = 1; > > v->no_cacheable = 0; > > v->not_found = 0; > > - v->data = addr->data; > > + v->data = p; > > > > return NGX_OK; > > } > > @@ -617,7 +626,12 @@ ngx_stream_variable_proxy_protocol_port( > > return NGX_ERROR; > > } > > > > - port = *(in_port_t *) ((char *) pp + data); > > + if (data) { > > + port = ngx_inet_get_port(pp->dst_sockaddr); > > + > > + } else { > > + port = ngx_inet_get_port(pp->src_sockaddr); > > + } > > > > if (port > 0 && port < 65536) { > > v->len = ngx_sprintf(v->data, "%ui", port) - v->data; > > > # HG changeset patch > > # User Roman Arutyunyan > > # Date 1710081227 -14400 > > # Sun Mar 10 18:33:47 2024 +0400 > > # Node ID 6957c271a15c789be51d042de6835ce02141fe58 > > # Parent 07c8c445746c6c74250bad7b42cd26dbb0ed6dd3 > > Pass PROXY protocol header from client to upstream (ticket #2010). > > > > When creating a PROXY protocol header in mail or stream, server address is > > taken from c->local_sockaddr, which is normally the real server address of the > > connection. If prior to that client address was substituted by the realip > > module, then client and server addresses can have different address families, > > which breaks PROXY protocol syntax. The change fixes this by using server > > address from client PROXY protocol header as well. > > > > diff --git a/src/core/ngx_proxy_protocol.c b/src/core/ngx_proxy_protocol.c > > --- a/src/core/ngx_proxy_protocol.c > > +++ b/src/core/ngx_proxy_protocol.c > > @@ -375,11 +375,24 @@ ngx_proxy_protocol_write(ngx_connection_ > > > > *buf++ = ' '; > > > > The above ngx_connection_local_sockaddr() call is not needed now if > server address is obtained from the saved PROXY header, it can be put > under conditional. > > > - buf += ngx_sock_ntop(c->local_sockaddr, c->local_socklen, buf, last - buf, > > - 0); > > + if (c->proxy_protocol > > + && ngx_cmp_sockaddr(c->proxy_protocol->src_sockaddr, > > + c->proxy_protocol->src_socklen, > > + c->sockaddr, c->socklen, 1) > > + == NGX_OK) > > This relies on a weak assumption that matching client addresses > is the result of client address substitution by the realip module. > This allows to substitute server address in PROXY writer to a > client chosen value regardless of address replacement being allowed > with set_real_ip_from. > (That's why an explicit hint from realip might be beneficial.) This is a valid point. However keeping a hint from realip still feels like a bad solution. Also, passing client PROXY protocol header doesn't solve all the issues. For example, stream pass module may change server address, but keep client address. This also can happen in HTTP when using X-Forwarded-For etc. We do not currently write PROXY protocol headers in HTTP, but this can happen in future. All the above makes me think the previous solution with zero addr/port substitution is a better choice. I will update that patch. > > + { > > + buf += ngx_sock_ntop(c->proxy_protocol->dst_sockaddr, > > + c->proxy_protocol->dst_socklen, > > + buf, last - buf, 0); > > I'd add a blank line > > > + lport = ngx_inet_get_port(c->proxy_protocol->dst_sockaddr); > > + > > + } else { > > + buf += ngx_sock_ntop(c->local_sockaddr, c->local_socklen, > > + buf, last - buf, 0); > > As ngx_connection_local_sockaddr() calls ngx_sock_ntop() internally, > the result could be re-used here, to get rid of the 2nd explicit > ngx_sock_ntop() call (with all its implied overhead). > > > + lport = ngx_inet_get_port(c->local_sockaddr); > > + } > > > > port = ngx_inet_get_port(c->sockaddr); > > - lport = ngx_inet_get_port(c->local_sockaddr); > > > > return ngx_slprintf(buf, last, " %ui %ui" CRLF, port, lport); > > } > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > https://mailman.nginx.org/mailman/listinfo/nginx-devel -- Roman Arutyunyan From pluknet at nginx.com Fri Mar 22 11:17:08 2024 From: pluknet at nginx.com (=?utf-8?q?Sergey_Kandaurov?=) Date: Fri, 22 Mar 2024 11:17:08 +0000 Subject: [nginx] Stream: using ngx_stream_ssl_srv_conf_t *sscf naming convention. Message-ID: details: https://hg.nginx.org/nginx/rev/fa75fccf7fa0 branches: changeset: 9218:fa75fccf7fa0 user: Sergey Kandaurov date: Fri Mar 22 14:18:51 2024 +0400 description: Stream: using ngx_stream_ssl_srv_conf_t *sscf naming convention. Originally, the stream module was developed based on the mail module, following the existing style. Then it was diverged to closely follow the http module development. This change updates style to use sscf naming convention troughout the stream module, which matches the http module code style. No functional changes. diffstat: src/stream/ngx_stream_ssl_module.c | 192 ++++++++++++++++++------------------ src/stream/ngx_stream_ssl_module.h | 2 +- 2 files changed, 97 insertions(+), 97 deletions(-) diffs (544 lines): diff -r 913518341c20 -r fa75fccf7fa0 src/stream/ngx_stream_ssl_module.c --- a/src/stream/ngx_stream_ssl_module.c Wed Feb 21 17:36:02 2024 +0400 +++ b/src/stream/ngx_stream_ssl_module.c Fri Mar 22 14:18:51 2024 +0400 @@ -40,12 +40,12 @@ static ngx_int_t ngx_stream_ssl_variable ngx_stream_variable_value_t *v, uintptr_t data); static ngx_int_t ngx_stream_ssl_add_variables(ngx_conf_t *cf); -static void *ngx_stream_ssl_create_conf(ngx_conf_t *cf); -static char *ngx_stream_ssl_merge_conf(ngx_conf_t *cf, void *parent, +static void *ngx_stream_ssl_create_srv_conf(ngx_conf_t *cf); +static char *ngx_stream_ssl_merge_srv_conf(ngx_conf_t *cf, void *parent, void *child); static ngx_int_t ngx_stream_ssl_compile_certificates(ngx_conf_t *cf, - ngx_stream_ssl_conf_t *conf); + ngx_stream_ssl_srv_conf_t *conf); static char *ngx_stream_ssl_password_file(ngx_conf_t *cf, ngx_command_t *cmd, void *conf); @@ -90,21 +90,21 @@ static ngx_command_t ngx_stream_ssl_com NGX_STREAM_MAIN_CONF|NGX_STREAM_SRV_CONF|NGX_CONF_TAKE1, ngx_conf_set_msec_slot, NGX_STREAM_SRV_CONF_OFFSET, - offsetof(ngx_stream_ssl_conf_t, handshake_timeout), + offsetof(ngx_stream_ssl_srv_conf_t, handshake_timeout), NULL }, { ngx_string("ssl_certificate"), NGX_STREAM_MAIN_CONF|NGX_STREAM_SRV_CONF|NGX_CONF_TAKE1, ngx_conf_set_str_array_slot, NGX_STREAM_SRV_CONF_OFFSET, - offsetof(ngx_stream_ssl_conf_t, certificates), + offsetof(ngx_stream_ssl_srv_conf_t, certificates), NULL }, { ngx_string("ssl_certificate_key"), NGX_STREAM_MAIN_CONF|NGX_STREAM_SRV_CONF|NGX_CONF_TAKE1, ngx_conf_set_str_array_slot, NGX_STREAM_SRV_CONF_OFFSET, - offsetof(ngx_stream_ssl_conf_t, certificate_keys), + offsetof(ngx_stream_ssl_srv_conf_t, certificate_keys), NULL }, { ngx_string("ssl_password_file"), @@ -118,63 +118,63 @@ static ngx_command_t ngx_stream_ssl_com NGX_STREAM_MAIN_CONF|NGX_STREAM_SRV_CONF|NGX_CONF_TAKE1, ngx_conf_set_str_slot, NGX_STREAM_SRV_CONF_OFFSET, - offsetof(ngx_stream_ssl_conf_t, dhparam), + offsetof(ngx_stream_ssl_srv_conf_t, dhparam), NULL }, { ngx_string("ssl_ecdh_curve"), NGX_STREAM_MAIN_CONF|NGX_STREAM_SRV_CONF|NGX_CONF_TAKE1, ngx_conf_set_str_slot, NGX_STREAM_SRV_CONF_OFFSET, - offsetof(ngx_stream_ssl_conf_t, ecdh_curve), + offsetof(ngx_stream_ssl_srv_conf_t, ecdh_curve), NULL }, { ngx_string("ssl_protocols"), NGX_STREAM_MAIN_CONF|NGX_STREAM_SRV_CONF|NGX_CONF_1MORE, ngx_conf_set_bitmask_slot, NGX_STREAM_SRV_CONF_OFFSET, - offsetof(ngx_stream_ssl_conf_t, protocols), + offsetof(ngx_stream_ssl_srv_conf_t, protocols), &ngx_stream_ssl_protocols }, { ngx_string("ssl_ciphers"), NGX_STREAM_MAIN_CONF|NGX_STREAM_SRV_CONF|NGX_CONF_TAKE1, ngx_conf_set_str_slot, NGX_STREAM_SRV_CONF_OFFSET, - offsetof(ngx_stream_ssl_conf_t, ciphers), + offsetof(ngx_stream_ssl_srv_conf_t, ciphers), NULL }, { ngx_string("ssl_verify_client"), NGX_STREAM_MAIN_CONF|NGX_STREAM_SRV_CONF|NGX_CONF_TAKE1, ngx_conf_set_enum_slot, NGX_STREAM_SRV_CONF_OFFSET, - offsetof(ngx_stream_ssl_conf_t, verify), + offsetof(ngx_stream_ssl_srv_conf_t, verify), &ngx_stream_ssl_verify }, { ngx_string("ssl_verify_depth"), NGX_STREAM_MAIN_CONF|NGX_STREAM_SRV_CONF|NGX_CONF_TAKE1, ngx_conf_set_num_slot, NGX_STREAM_SRV_CONF_OFFSET, - offsetof(ngx_stream_ssl_conf_t, verify_depth), + offsetof(ngx_stream_ssl_srv_conf_t, verify_depth), NULL }, { ngx_string("ssl_client_certificate"), NGX_STREAM_MAIN_CONF|NGX_STREAM_SRV_CONF|NGX_CONF_TAKE1, ngx_conf_set_str_slot, NGX_STREAM_SRV_CONF_OFFSET, - offsetof(ngx_stream_ssl_conf_t, client_certificate), + offsetof(ngx_stream_ssl_srv_conf_t, client_certificate), NULL }, { ngx_string("ssl_trusted_certificate"), NGX_STREAM_MAIN_CONF|NGX_STREAM_SRV_CONF|NGX_CONF_TAKE1, ngx_conf_set_str_slot, NGX_STREAM_SRV_CONF_OFFSET, - offsetof(ngx_stream_ssl_conf_t, trusted_certificate), + offsetof(ngx_stream_ssl_srv_conf_t, trusted_certificate), NULL }, { ngx_string("ssl_prefer_server_ciphers"), NGX_STREAM_MAIN_CONF|NGX_STREAM_SRV_CONF|NGX_CONF_FLAG, ngx_conf_set_flag_slot, NGX_STREAM_SRV_CONF_OFFSET, - offsetof(ngx_stream_ssl_conf_t, prefer_server_ciphers), + offsetof(ngx_stream_ssl_srv_conf_t, prefer_server_ciphers), NULL }, { ngx_string("ssl_session_cache"), @@ -188,42 +188,42 @@ static ngx_command_t ngx_stream_ssl_com NGX_STREAM_MAIN_CONF|NGX_STREAM_SRV_CONF|NGX_CONF_FLAG, ngx_conf_set_flag_slot, NGX_STREAM_SRV_CONF_OFFSET, - offsetof(ngx_stream_ssl_conf_t, session_tickets), + offsetof(ngx_stream_ssl_srv_conf_t, session_tickets), NULL }, { ngx_string("ssl_session_ticket_key"), NGX_STREAM_MAIN_CONF|NGX_STREAM_SRV_CONF|NGX_CONF_TAKE1, ngx_conf_set_str_array_slot, NGX_STREAM_SRV_CONF_OFFSET, - offsetof(ngx_stream_ssl_conf_t, session_ticket_keys), + offsetof(ngx_stream_ssl_srv_conf_t, session_ticket_keys), NULL }, { ngx_string("ssl_session_timeout"), NGX_STREAM_MAIN_CONF|NGX_STREAM_SRV_CONF|NGX_CONF_TAKE1, ngx_conf_set_sec_slot, NGX_STREAM_SRV_CONF_OFFSET, - offsetof(ngx_stream_ssl_conf_t, session_timeout), + offsetof(ngx_stream_ssl_srv_conf_t, session_timeout), NULL }, { ngx_string("ssl_crl"), NGX_STREAM_MAIN_CONF|NGX_STREAM_SRV_CONF|NGX_CONF_TAKE1, ngx_conf_set_str_slot, NGX_STREAM_SRV_CONF_OFFSET, - offsetof(ngx_stream_ssl_conf_t, crl), + offsetof(ngx_stream_ssl_srv_conf_t, crl), NULL }, { ngx_string("ssl_conf_command"), NGX_STREAM_MAIN_CONF|NGX_STREAM_SRV_CONF|NGX_CONF_TAKE2, ngx_conf_set_keyval_slot, NGX_STREAM_SRV_CONF_OFFSET, - offsetof(ngx_stream_ssl_conf_t, conf_commands), + offsetof(ngx_stream_ssl_srv_conf_t, conf_commands), &ngx_stream_ssl_conf_command_post }, { ngx_string("ssl_reject_handshake"), NGX_STREAM_MAIN_CONF|NGX_STREAM_SRV_CONF|NGX_CONF_FLAG, ngx_conf_set_flag_slot, NGX_STREAM_SRV_CONF_OFFSET, - offsetof(ngx_stream_ssl_conf_t, reject_handshake), + offsetof(ngx_stream_ssl_srv_conf_t, reject_handshake), NULL }, { ngx_string("ssl_alpn"), @@ -244,8 +244,8 @@ static ngx_stream_module_t ngx_stream_s NULL, /* create main configuration */ NULL, /* init main configuration */ - ngx_stream_ssl_create_conf, /* create server configuration */ - ngx_stream_ssl_merge_conf /* merge server configuration */ + ngx_stream_ssl_create_srv_conf, /* create server configuration */ + ngx_stream_ssl_merge_srv_conf /* merge server configuration */ }; @@ -339,11 +339,11 @@ static ngx_str_t ngx_stream_ssl_sess_id_ static ngx_int_t ngx_stream_ssl_handler(ngx_stream_session_t *s) { - long rc; - X509 *cert; - ngx_int_t rv; - ngx_connection_t *c; - ngx_stream_ssl_conf_t *sslcf; + long rc; + X509 *cert; + ngx_int_t rv; + ngx_connection_t *c; + ngx_stream_ssl_srv_conf_t *sscf; if (!s->ssl) { return NGX_OK; @@ -351,23 +351,23 @@ ngx_stream_ssl_handler(ngx_stream_sessio c = s->connection; - sslcf = ngx_stream_get_module_srv_conf(s, ngx_stream_ssl_module); + sscf = ngx_stream_get_module_srv_conf(s, ngx_stream_ssl_module); if (c->ssl == NULL) { c->log->action = "SSL handshaking"; - rv = ngx_stream_ssl_init_connection(&sslcf->ssl, c); + rv = ngx_stream_ssl_init_connection(&sscf->ssl, c); if (rv != NGX_OK) { return rv; } } - if (sslcf->verify) { + if (sscf->verify) { rc = SSL_get_verify_result(c->ssl->connection); if (rc != X509_V_OK - && (sslcf->verify != 3 || !ngx_ssl_verify_error_optional(rc))) + && (sscf->verify != 3 || !ngx_ssl_verify_error_optional(rc))) { ngx_log_error(NGX_LOG_INFO, c->log, 0, "client SSL certificate verify error: (%l:%s)", @@ -378,7 +378,7 @@ ngx_stream_ssl_handler(ngx_stream_sessio return NGX_ERROR; } - if (sslcf->verify == 1) { + if (sscf->verify == 1) { cert = SSL_get_peer_certificate(c->ssl->connection); if (cert == NULL) { @@ -403,7 +403,7 @@ ngx_stream_ssl_init_connection(ngx_ssl_t { ngx_int_t rc; ngx_stream_session_t *s; - ngx_stream_ssl_conf_t *sslcf; + ngx_stream_ssl_srv_conf_t *sscf; ngx_stream_core_srv_conf_t *cscf; s = c->data; @@ -425,9 +425,9 @@ ngx_stream_ssl_init_connection(ngx_ssl_t } if (rc == NGX_AGAIN) { - sslcf = ngx_stream_get_module_srv_conf(s, ngx_stream_ssl_module); + sscf = ngx_stream_get_module_srv_conf(s, ngx_stream_ssl_module); - ngx_add_timer(c->read, sslcf->handshake_timeout); + ngx_add_timer(c->read, sscf->handshake_timeout); c->ssl->handler = ngx_stream_ssl_handshake_handler; @@ -470,7 +470,7 @@ ngx_stream_ssl_servername(ngx_ssl_conn_t const char *servername; ngx_connection_t *c; ngx_stream_session_t *s; - ngx_stream_ssl_conf_t *sscf; + ngx_stream_ssl_srv_conf_t *sscf; ngx_stream_core_srv_conf_t *cscf; c = ngx_ssl_get_connection(ssl_conn); @@ -625,7 +625,7 @@ ngx_stream_ssl_certificate(ngx_ssl_conn_ ngx_uint_t i, nelts; ngx_connection_t *c; ngx_stream_session_t *s; - ngx_stream_ssl_conf_t *sslcf; + ngx_stream_ssl_srv_conf_t *sscf; ngx_stream_complex_value_t *certs, *keys; c = ngx_ssl_get_connection(ssl_conn); @@ -636,11 +636,11 @@ ngx_stream_ssl_certificate(ngx_ssl_conn_ s = c->data; - sslcf = arg; + sscf = arg; - nelts = sslcf->certificate_values->nelts; - certs = sslcf->certificate_values->elts; - keys = sslcf->certificate_key_values->elts; + nelts = sscf->certificate_values->nelts; + certs = sscf->certificate_values->elts; + keys = sscf->certificate_key_values->elts; for (i = 0; i < nelts; i++) { @@ -659,7 +659,7 @@ ngx_stream_ssl_certificate(ngx_ssl_conn_ "ssl key: \"%s\"", key.data); if (ngx_ssl_connection_certificate(c, c->pool, &cert, &key, - sslcf->passwords) + sscf->passwords) != NGX_OK) { return 0; @@ -755,53 +755,53 @@ ngx_stream_ssl_add_variables(ngx_conf_t static void * -ngx_stream_ssl_create_conf(ngx_conf_t *cf) +ngx_stream_ssl_create_srv_conf(ngx_conf_t *cf) { - ngx_stream_ssl_conf_t *scf; + ngx_stream_ssl_srv_conf_t *sscf; - scf = ngx_pcalloc(cf->pool, sizeof(ngx_stream_ssl_conf_t)); - if (scf == NULL) { + sscf = ngx_pcalloc(cf->pool, sizeof(ngx_stream_ssl_srv_conf_t)); + if (sscf == NULL) { return NULL; } /* * set by ngx_pcalloc(): * - * scf->protocols = 0; - * scf->certificate_values = NULL; - * scf->dhparam = { 0, NULL }; - * scf->ecdh_curve = { 0, NULL }; - * scf->client_certificate = { 0, NULL }; - * scf->trusted_certificate = { 0, NULL }; - * scf->crl = { 0, NULL }; - * scf->alpn = { 0, NULL }; - * scf->ciphers = { 0, NULL }; - * scf->shm_zone = NULL; + * sscf->protocols = 0; + * sscf->certificate_values = NULL; + * sscf->dhparam = { 0, NULL }; + * sscf->ecdh_curve = { 0, NULL }; + * sscf->client_certificate = { 0, NULL }; + * sscf->trusted_certificate = { 0, NULL }; + * sscf->crl = { 0, NULL }; + * sscf->alpn = { 0, NULL }; + * sscf->ciphers = { 0, NULL }; + * sscf->shm_zone = NULL; */ - scf->handshake_timeout = NGX_CONF_UNSET_MSEC; - scf->certificates = NGX_CONF_UNSET_PTR; - scf->certificate_keys = NGX_CONF_UNSET_PTR; - scf->passwords = NGX_CONF_UNSET_PTR; - scf->conf_commands = NGX_CONF_UNSET_PTR; - scf->prefer_server_ciphers = NGX_CONF_UNSET; - scf->reject_handshake = NGX_CONF_UNSET; - scf->verify = NGX_CONF_UNSET_UINT; - scf->verify_depth = NGX_CONF_UNSET_UINT; - scf->builtin_session_cache = NGX_CONF_UNSET; - scf->session_timeout = NGX_CONF_UNSET; - scf->session_tickets = NGX_CONF_UNSET; - scf->session_ticket_keys = NGX_CONF_UNSET_PTR; + sscf->handshake_timeout = NGX_CONF_UNSET_MSEC; + sscf->certificates = NGX_CONF_UNSET_PTR; + sscf->certificate_keys = NGX_CONF_UNSET_PTR; + sscf->passwords = NGX_CONF_UNSET_PTR; + sscf->conf_commands = NGX_CONF_UNSET_PTR; + sscf->prefer_server_ciphers = NGX_CONF_UNSET; + sscf->reject_handshake = NGX_CONF_UNSET; + sscf->verify = NGX_CONF_UNSET_UINT; + sscf->verify_depth = NGX_CONF_UNSET_UINT; + sscf->builtin_session_cache = NGX_CONF_UNSET; + sscf->session_timeout = NGX_CONF_UNSET; + sscf->session_tickets = NGX_CONF_UNSET; + sscf->session_ticket_keys = NGX_CONF_UNSET_PTR; - return scf; + return sscf; } static char * -ngx_stream_ssl_merge_conf(ngx_conf_t *cf, void *parent, void *child) +ngx_stream_ssl_merge_srv_conf(ngx_conf_t *cf, void *parent, void *child) { - ngx_stream_ssl_conf_t *prev = parent; - ngx_stream_ssl_conf_t *conf = child; + ngx_stream_ssl_srv_conf_t *prev = parent; + ngx_stream_ssl_srv_conf_t *conf = child; ngx_pool_cleanup_t *cln; @@ -1010,7 +1010,7 @@ ngx_stream_ssl_merge_conf(ngx_conf_t *cf static ngx_int_t ngx_stream_ssl_compile_certificates(ngx_conf_t *cf, - ngx_stream_ssl_conf_t *conf) + ngx_stream_ssl_srv_conf_t *conf) { ngx_str_t *cert, *key; ngx_uint_t i, nelts; @@ -1099,19 +1099,19 @@ found: static char * ngx_stream_ssl_password_file(ngx_conf_t *cf, ngx_command_t *cmd, void *conf) { - ngx_stream_ssl_conf_t *scf = conf; + ngx_stream_ssl_srv_conf_t *sscf = conf; ngx_str_t *value; - if (scf->passwords != NGX_CONF_UNSET_PTR) { + if (sscf->passwords != NGX_CONF_UNSET_PTR) { return "is duplicate"; } value = cf->args->elts; - scf->passwords = ngx_ssl_read_password_file(cf, &value[1]); + sscf->passwords = ngx_ssl_read_password_file(cf, &value[1]); - if (scf->passwords == NULL) { + if (sscf->passwords == NULL) { return NGX_CONF_ERROR; } @@ -1122,7 +1122,7 @@ ngx_stream_ssl_password_file(ngx_conf_t static char * ngx_stream_ssl_session_cache(ngx_conf_t *cf, ngx_command_t *cmd, void *conf) { - ngx_stream_ssl_conf_t *scf = conf; + ngx_stream_ssl_srv_conf_t *sscf = conf; size_t len; ngx_str_t *value, name, size; @@ -1134,17 +1134,17 @@ ngx_stream_ssl_session_cache(ngx_conf_t for (i = 1; i < cf->args->nelts; i++) { if (ngx_strcmp(value[i].data, "off") == 0) { - scf->builtin_session_cache = NGX_SSL_NO_SCACHE; + sscf->builtin_session_cache = NGX_SSL_NO_SCACHE; continue; } if (ngx_strcmp(value[i].data, "none") == 0) { - scf->builtin_session_cache = NGX_SSL_NONE_SCACHE; + sscf->builtin_session_cache = NGX_SSL_NONE_SCACHE; continue; } if (ngx_strcmp(value[i].data, "builtin") == 0) { - scf->builtin_session_cache = NGX_SSL_DFLT_BUILTIN_SCACHE; + sscf->builtin_session_cache = NGX_SSL_DFLT_BUILTIN_SCACHE; continue; } @@ -1159,7 +1159,7 @@ ngx_stream_ssl_session_cache(ngx_conf_t goto invalid; } - scf->builtin_session_cache = n; + sscf->builtin_session_cache = n; continue; } @@ -1202,13 +1202,13 @@ ngx_stream_ssl_session_cache(ngx_conf_t return NGX_CONF_ERROR; } - scf->shm_zone = ngx_shared_memory_add(cf, &name, n, + sscf->shm_zone = ngx_shared_memory_add(cf, &name, n, &ngx_stream_ssl_module); - if (scf->shm_zone == NULL) { + if (sscf->shm_zone == NULL) { return NGX_CONF_ERROR; } - scf->shm_zone->init = ngx_ssl_session_cache_init; + sscf->shm_zone->init = ngx_ssl_session_cache_init; continue; } @@ -1216,8 +1216,8 @@ ngx_stream_ssl_session_cache(ngx_conf_t goto invalid; } - if (scf->shm_zone && scf->builtin_session_cache == NGX_CONF_UNSET) { - scf->builtin_session_cache = NGX_SSL_NO_BUILTIN_SCACHE; + if (sscf->shm_zone && sscf->builtin_session_cache == NGX_CONF_UNSET) { + sscf->builtin_session_cache = NGX_SSL_NO_BUILTIN_SCACHE; } return NGX_CONF_OK; @@ -1236,14 +1236,14 @@ ngx_stream_ssl_alpn(ngx_conf_t *cf, ngx_ { #ifdef TLSEXT_TYPE_application_layer_protocol_negotiation - ngx_stream_ssl_conf_t *scf = conf; + ngx_stream_ssl_srv_conf_t *sscf = conf; u_char *p; size_t len; ngx_str_t *value; ngx_uint_t i; - if (scf->alpn.len) { + if (sscf->alpn.len) { return "is duplicate"; } @@ -1260,19 +1260,19 @@ ngx_stream_ssl_alpn(ngx_conf_t *cf, ngx_ len += value[i].len + 1; } - scf->alpn.data = ngx_pnalloc(cf->pool, len); - if (scf->alpn.data == NULL) { + sscf->alpn.data = ngx_pnalloc(cf->pool, len); + if (sscf->alpn.data == NULL) { return NGX_CONF_ERROR; } - p = scf->alpn.data; + p = sscf->alpn.data; for (i = 1; i < cf->args->nelts; i++) { *p++ = value[i].len; p = ngx_cpymem(p, value[i].data, value[i].len); } - scf->alpn.len = len; + sscf->alpn.len = len; return NGX_CONF_OK; @@ -1301,9 +1301,9 @@ ngx_stream_ssl_init(ngx_conf_t *cf) { ngx_uint_t a, p, s; ngx_stream_handler_pt *h; - ngx_stream_ssl_conf_t *sscf; ngx_stream_conf_addr_t *addr; ngx_stream_conf_port_t *port; + ngx_stream_ssl_srv_conf_t *sscf; ngx_stream_core_srv_conf_t **cscfp, *cscf; ngx_stream_core_main_conf_t *cmcf; diff -r 913518341c20 -r fa75fccf7fa0 src/stream/ngx_stream_ssl_module.h --- a/src/stream/ngx_stream_ssl_module.h Wed Feb 21 17:36:02 2024 +0400 +++ b/src/stream/ngx_stream_ssl_module.h Fri Mar 22 14:18:51 2024 +0400 @@ -53,7 +53,7 @@ typedef struct { ngx_flag_t session_tickets; ngx_array_t *session_ticket_keys; -} ngx_stream_ssl_conf_t; +} ngx_stream_ssl_srv_conf_t; extern ngx_module_t ngx_stream_ssl_module; From pluknet at nginx.com Fri Mar 22 11:17:11 2024 From: pluknet at nginx.com (=?utf-8?q?Sergey_Kandaurov?=) Date: Fri, 22 Mar 2024 11:17:11 +0000 Subject: [nginx] Overhauled some diagnostic messages akin to 1b05b9bbcebf. Message-ID: details: https://hg.nginx.org/nginx/rev/71880e854bc7 branches: changeset: 9219:71880e854bc7 user: Sergey Kandaurov date: Fri Mar 22 14:51:14 2024 +0400 description: Overhauled some diagnostic messages akin to 1b05b9bbcebf. diffstat: src/http/modules/ngx_http_referer_module.c | 2 +- src/http/modules/ngx_http_ssi_filter_module.c | 2 +- src/mail/ngx_mail_core_module.c | 4 ++-- src/stream/ngx_stream_core_module.c | 4 ++-- 4 files changed, 6 insertions(+), 6 deletions(-) diffs (66 lines): diff -r fa75fccf7fa0 -r 71880e854bc7 src/http/modules/ngx_http_referer_module.c --- a/src/http/modules/ngx_http_referer_module.c Fri Mar 22 14:18:51 2024 +0400 +++ b/src/http/modules/ngx_http_referer_module.c Fri Mar 22 14:51:14 2024 +0400 @@ -631,7 +631,7 @@ ngx_http_add_regex_referer(ngx_conf_t *c #else ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, - "the using of the regex \"%V\" requires PCRE library", + "using regex \"%V\" requires PCRE library", name); return NGX_ERROR; diff -r fa75fccf7fa0 -r 71880e854bc7 src/http/modules/ngx_http_ssi_filter_module.c --- a/src/http/modules/ngx_http_ssi_filter_module.c Fri Mar 22 14:18:51 2024 +0400 +++ b/src/http/modules/ngx_http_ssi_filter_module.c Fri Mar 22 14:51:14 2024 +0400 @@ -2001,7 +2001,7 @@ ngx_http_ssi_regex_match(ngx_http_reques #else ngx_log_error(NGX_LOG_ALERT, r->connection->log, 0, - "the using of the regex \"%V\" in SSI requires PCRE library", + "using regex \"%V\" in SSI requires PCRE library", pattern); return NGX_HTTP_SSI_ERROR; diff -r fa75fccf7fa0 -r 71880e854bc7 src/mail/ngx_mail_core_module.c --- a/src/mail/ngx_mail_core_module.c Fri Mar 22 14:18:51 2024 +0400 +++ b/src/mail/ngx_mail_core_module.c Fri Mar 22 14:51:14 2024 +0400 @@ -441,7 +441,7 @@ ngx_mail_core_listen(ngx_conf_t *cf, ngx continue; #else ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, - "bind ipv6only is not supported " + "ipv6only is not supported " "on this platform"); return NGX_CONF_ERROR; #endif @@ -564,7 +564,7 @@ ngx_mail_core_listen(ngx_conf_t *cf, ngx } ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, - "the invalid \"%V\" parameter", &value[i]); + "invalid parameter \"%V\"", &value[i]); return NGX_CONF_ERROR; } diff -r fa75fccf7fa0 -r 71880e854bc7 src/stream/ngx_stream_core_module.c --- a/src/stream/ngx_stream_core_module.c Fri Mar 22 14:18:51 2024 +0400 +++ b/src/stream/ngx_stream_core_module.c Fri Mar 22 14:51:14 2024 +0400 @@ -1036,7 +1036,7 @@ ngx_stream_core_listen(ngx_conf_t *cf, n continue; #else ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, - "bind ipv6only is not supported " + "ipv6only is not supported " "on this platform"); return NGX_CONF_ERROR; #endif @@ -1164,7 +1164,7 @@ ngx_stream_core_listen(ngx_conf_t *cf, n } ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, - "the invalid \"%V\" parameter", &value[i]); + "invalid parameter \"%V\"", &value[i]); return NGX_CONF_ERROR; } From pluknet at nginx.com Fri Mar 22 11:17:14 2024 From: pluknet at nginx.com (=?utf-8?q?Sergey_Kandaurov?=) Date: Fri, 22 Mar 2024 11:17:14 +0000 Subject: [nginx] Stream: reshuffled ngx_stream_listen_opt_t fields. Message-ID: details: https://hg.nginx.org/nginx/rev/c8cf6b1cc82e branches: changeset: 9220:c8cf6b1cc82e user: Sergey Kandaurov date: Fri Mar 22 14:53:19 2024 +0400 description: Stream: reshuffled ngx_stream_listen_opt_t fields. In preparation for adding more parameters to the listen directive, and to be in sync with the corresponding structure in the http module. No functional changes. diffstat: src/stream/ngx_stream.h | 15 ++++++++------- 1 files changed, 8 insertions(+), 7 deletions(-) diffs (30 lines): diff -r 71880e854bc7 -r c8cf6b1cc82e src/stream/ngx_stream.h --- a/src/stream/ngx_stream.h Fri Mar 22 14:51:14 2024 +0400 +++ b/src/stream/ngx_stream.h Fri Mar 22 14:53:19 2024 +0400 @@ -56,18 +56,19 @@ typedef struct { unsigned reuseport:1; unsigned so_keepalive:2; unsigned proxy_protocol:1; + + int backlog; + int rcvbuf; + int sndbuf; + int type; +#if (NGX_HAVE_TCP_FASTOPEN) + int fastopen; +#endif #if (NGX_HAVE_KEEPALIVE_TUNABLE) int tcp_keepidle; int tcp_keepintvl; int tcp_keepcnt; #endif - int backlog; - int rcvbuf; - int sndbuf; -#if (NGX_HAVE_TCP_FASTOPEN) - int fastopen; -#endif - int type; } ngx_stream_listen_opt_t; From pluknet at nginx.com Fri Mar 22 11:17:17 2024 From: pluknet at nginx.com (=?utf-8?q?Sergey_Kandaurov?=) Date: Fri, 22 Mar 2024 11:17:17 +0000 Subject: [nginx] Stream: the "deferred" parameter of the "listen" directive. Message-ID: details: https://hg.nginx.org/nginx/rev/bed832296f5d branches: changeset: 9221:bed832296f5d user: Sergey Kandaurov date: Fri Mar 22 14:53:19 2024 +0400 description: Stream: the "deferred" parameter of the "listen" directive. The Linux TCP_DEFER_ACCEPT support. diffstat: src/stream/ngx_stream.c | 4 ++++ src/stream/ngx_stream.h | 1 + src/stream/ngx_stream_core_module.c | 19 +++++++++++++++++++ 3 files changed, 24 insertions(+), 0 deletions(-) diffs (61 lines): diff -r c8cf6b1cc82e -r bed832296f5d src/stream/ngx_stream.c --- a/src/stream/ngx_stream.c Fri Mar 22 14:53:19 2024 +0400 +++ b/src/stream/ngx_stream.c Fri Mar 22 14:53:19 2024 +0400 @@ -1021,6 +1021,10 @@ ngx_stream_add_listening(ngx_conf_t *cf, ls->keepcnt = addr->opt.tcp_keepcnt; #endif +#if (NGX_HAVE_DEFERRED_ACCEPT && defined TCP_DEFER_ACCEPT) + ls->deferred_accept = addr->opt.deferred_accept; +#endif + #if (NGX_HAVE_INET6) ls->ipv6only = addr->opt.ipv6only; #endif diff -r c8cf6b1cc82e -r bed832296f5d src/stream/ngx_stream.h --- a/src/stream/ngx_stream.h Fri Mar 22 14:53:19 2024 +0400 +++ b/src/stream/ngx_stream.h Fri Mar 22 14:53:19 2024 +0400 @@ -53,6 +53,7 @@ typedef struct { #if (NGX_HAVE_INET6) unsigned ipv6only:1; #endif + unsigned deferred_accept:1; unsigned reuseport:1; unsigned so_keepalive:2; unsigned proxy_protocol:1; diff -r c8cf6b1cc82e -r bed832296f5d src/stream/ngx_stream_core_module.c --- a/src/stream/ngx_stream_core_module.c Fri Mar 22 14:53:19 2024 +0400 +++ b/src/stream/ngx_stream_core_module.c Fri Mar 22 14:53:19 2024 +0400 @@ -1015,6 +1015,19 @@ ngx_stream_core_listen(ngx_conf_t *cf, n continue; } + if (ngx_strcmp(value[i].data, "deferred") == 0) { +#if (NGX_HAVE_DEFERRED_ACCEPT && defined TCP_DEFER_ACCEPT) + lsopt.deferred_accept = 1; + lsopt.set = 1; + lsopt.bind = 1; +#else + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, + "the deferred accept is not supported " + "on this platform, ignored"); +#endif + continue; + } + if (ngx_strncmp(value[i].data, "ipv6only=o", 10) == 0) { #if (NGX_HAVE_INET6 && defined IPV6_V6ONLY) if (ngx_strcmp(&value[i].data[10], "n") == 0) { @@ -1173,6 +1186,12 @@ ngx_stream_core_listen(ngx_conf_t *cf, n return "\"backlog\" parameter is incompatible with \"udp\""; } +#if (NGX_HAVE_DEFERRED_ACCEPT && defined TCP_DEFER_ACCEPT) + if (lsopt.deferred_accept) { + return "\"deferred\" parameter is incompatible with \"udp\""; + } +#endif + #if (NGX_STREAM_SSL) if (lsopt.ssl) { return "\"ssl\" parameter is incompatible with \"udp\""; From pluknet at nginx.com Fri Mar 22 11:17:20 2024 From: pluknet at nginx.com (=?utf-8?q?Sergey_Kandaurov?=) Date: Fri, 22 Mar 2024 11:17:20 +0000 Subject: [nginx] Stream: the "accept_filter" parameter of the "listen" directive. Message-ID: details: https://hg.nginx.org/nginx/rev/c78790d3d061 branches: changeset: 9222:c78790d3d061 user: Sergey Kandaurov date: Fri Mar 22 14:53:19 2024 +0400 description: Stream: the "accept_filter" parameter of the "listen" directive. The FreeBSD accept filters support. diffstat: src/stream/ngx_stream.c | 4 ++++ src/stream/ngx_stream.h | 4 ++++ src/stream/ngx_stream_core_module.c | 20 ++++++++++++++++++++ 3 files changed, 28 insertions(+), 0 deletions(-) diffs (65 lines): diff -r bed832296f5d -r c78790d3d061 src/stream/ngx_stream.c --- a/src/stream/ngx_stream.c Fri Mar 22 14:53:19 2024 +0400 +++ b/src/stream/ngx_stream.c Fri Mar 22 14:53:19 2024 +0400 @@ -1021,6 +1021,10 @@ ngx_stream_add_listening(ngx_conf_t *cf, ls->keepcnt = addr->opt.tcp_keepcnt; #endif +#if (NGX_HAVE_DEFERRED_ACCEPT && defined SO_ACCEPTFILTER) + ls->accept_filter = addr->opt.accept_filter; +#endif + #if (NGX_HAVE_DEFERRED_ACCEPT && defined TCP_DEFER_ACCEPT) ls->deferred_accept = addr->opt.deferred_accept; #endif diff -r bed832296f5d -r c78790d3d061 src/stream/ngx_stream.h --- a/src/stream/ngx_stream.h Fri Mar 22 14:53:19 2024 +0400 +++ b/src/stream/ngx_stream.h Fri Mar 22 14:53:19 2024 +0400 @@ -70,6 +70,10 @@ typedef struct { int tcp_keepintvl; int tcp_keepcnt; #endif + +#if (NGX_HAVE_DEFERRED_ACCEPT && defined SO_ACCEPTFILTER) + char *accept_filter; +#endif } ngx_stream_listen_opt_t; diff -r bed832296f5d -r c78790d3d061 src/stream/ngx_stream_core_module.c --- a/src/stream/ngx_stream_core_module.c Fri Mar 22 14:53:19 2024 +0400 +++ b/src/stream/ngx_stream_core_module.c Fri Mar 22 14:53:19 2024 +0400 @@ -1015,6 +1015,20 @@ ngx_stream_core_listen(ngx_conf_t *cf, n continue; } + if (ngx_strncmp(value[i].data, "accept_filter=", 14) == 0) { +#if (NGX_HAVE_DEFERRED_ACCEPT && defined SO_ACCEPTFILTER) + lsopt.accept_filter = (char *) &value[i].data[14]; + lsopt.set = 1; + lsopt.bind = 1; +#else + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, + "accept filters \"%V\" are not supported " + "on this platform, ignored", + &value[i]); +#endif + continue; + } + if (ngx_strcmp(value[i].data, "deferred") == 0) { #if (NGX_HAVE_DEFERRED_ACCEPT && defined TCP_DEFER_ACCEPT) lsopt.deferred_accept = 1; @@ -1186,6 +1200,12 @@ ngx_stream_core_listen(ngx_conf_t *cf, n return "\"backlog\" parameter is incompatible with \"udp\""; } +#if (NGX_HAVE_DEFERRED_ACCEPT && defined SO_ACCEPTFILTER) + if (lsopt.accept_filter) { + return "\"accept_filter\" parameter is incompatible with \"udp\""; + } +#endif + #if (NGX_HAVE_DEFERRED_ACCEPT && defined TCP_DEFER_ACCEPT) if (lsopt.deferred_accept) { return "\"deferred\" parameter is incompatible with \"udp\""; From pluknet at nginx.com Fri Mar 22 11:17:22 2024 From: pluknet at nginx.com (=?utf-8?q?Sergey_Kandaurov?=) Date: Fri, 22 Mar 2024 11:17:22 +0000 Subject: [nginx] Stream: the "setfib" parameter of the "listen" directive. Message-ID: details: https://hg.nginx.org/nginx/rev/dd516985310f branches: changeset: 9223:dd516985310f user: Sergey Kandaurov date: Fri Mar 22 14:53:19 2024 +0400 description: Stream: the "setfib" parameter of the "listen" directive. The FreeBSD SO_SETFIB support. diffstat: src/stream/ngx_stream.c | 4 ++++ src/stream/ngx_stream.h | 3 +++ src/stream/ngx_stream_core_module.c | 19 +++++++++++++++++++ 3 files changed, 26 insertions(+), 0 deletions(-) diffs (63 lines): diff -r c78790d3d061 -r dd516985310f src/stream/ngx_stream.c --- a/src/stream/ngx_stream.c Fri Mar 22 14:53:19 2024 +0400 +++ b/src/stream/ngx_stream.c Fri Mar 22 14:53:19 2024 +0400 @@ -1033,6 +1033,10 @@ ngx_stream_add_listening(ngx_conf_t *cf, ls->ipv6only = addr->opt.ipv6only; #endif +#if (NGX_HAVE_SETFIB) + ls->setfib = addr->opt.setfib; +#endif + #if (NGX_HAVE_TCP_FASTOPEN) ls->fastopen = addr->opt.fastopen; #endif diff -r c78790d3d061 -r dd516985310f src/stream/ngx_stream.h --- a/src/stream/ngx_stream.h Fri Mar 22 14:53:19 2024 +0400 +++ b/src/stream/ngx_stream.h Fri Mar 22 14:53:19 2024 +0400 @@ -62,6 +62,9 @@ typedef struct { int rcvbuf; int sndbuf; int type; +#if (NGX_HAVE_SETFIB) + int setfib; +#endif #if (NGX_HAVE_TCP_FASTOPEN) int fastopen; #endif diff -r c78790d3d061 -r dd516985310f src/stream/ngx_stream_core_module.c --- a/src/stream/ngx_stream_core_module.c Fri Mar 22 14:53:19 2024 +0400 +++ b/src/stream/ngx_stream_core_module.c Fri Mar 22 14:53:19 2024 +0400 @@ -920,6 +920,9 @@ ngx_stream_core_listen(ngx_conf_t *cf, n lsopt.type = SOCK_STREAM; lsopt.rcvbuf = -1; lsopt.sndbuf = -1; +#if (NGX_HAVE_SETFIB) + lsopt.setfib = -1; +#endif #if (NGX_HAVE_TCP_FASTOPEN) lsopt.fastopen = -1; #endif @@ -949,6 +952,22 @@ ngx_stream_core_listen(ngx_conf_t *cf, n continue; } +#if (NGX_HAVE_SETFIB) + if (ngx_strncmp(value[i].data, "setfib=", 7) == 0) { + lsopt.setfib = ngx_atoi(value[i].data + 7, value[i].len - 7); + lsopt.set = 1; + lsopt.bind = 1; + + if (lsopt.setfib == NGX_ERROR) { + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, + "invalid setfib \"%V\"", &value[i]); + return NGX_CONF_ERROR; + } + + continue; + } +#endif + #if (NGX_HAVE_TCP_FASTOPEN) if (ngx_strncmp(value[i].data, "fastopen=", 9) == 0) { lsopt.fastopen = ngx_atoi(value[i].data + 9, value[i].len - 9); From pluknet at nginx.com Fri Mar 22 11:17:25 2024 From: pluknet at nginx.com (=?utf-8?q?Sergey_Kandaurov?=) Date: Fri, 22 Mar 2024 11:17:25 +0000 Subject: [nginx] Stream: moved fastopen compatibility check. Message-ID: details: https://hg.nginx.org/nginx/rev/6317e21a15e0 branches: changeset: 9224:6317e21a15e0 user: Roman Arutyunyan date: Thu Jan 18 19:12:38 2024 +0400 description: Stream: moved fastopen compatibility check. The move makes the code look similar to the corresponding code in http module. diffstat: src/stream/ngx_stream_core_module.c | 12 ++++++------ 1 files changed, 6 insertions(+), 6 deletions(-) diffs (29 lines): diff -r dd516985310f -r 6317e21a15e0 src/stream/ngx_stream_core_module.c --- a/src/stream/ngx_stream_core_module.c Fri Mar 22 14:53:19 2024 +0400 +++ b/src/stream/ngx_stream_core_module.c Thu Jan 18 19:12:38 2024 +0400 @@ -1215,6 +1215,12 @@ ngx_stream_core_listen(ngx_conf_t *cf, n } if (lsopt.type == SOCK_DGRAM) { +#if (NGX_HAVE_TCP_FASTOPEN) + if (lsopt.fastopen != -1) { + return "\"fastopen\" parameter is incompatible with \"udp\""; + } +#endif + if (backlog) { return "\"backlog\" parameter is incompatible with \"udp\""; } @@ -1244,12 +1250,6 @@ ngx_stream_core_listen(ngx_conf_t *cf, n if (lsopt.proxy_protocol) { return "\"proxy_protocol\" parameter is incompatible with \"udp\""; } - -#if (NGX_HAVE_TCP_FASTOPEN) - if (lsopt.fastopen != -1) { - return "\"fastopen\" parameter is incompatible with \"udp\""; - } -#endif } for (n = 0; n < u.naddrs; n++) { From pluknet at nginx.com Fri Mar 22 14:41:12 2024 From: pluknet at nginx.com (=?iso-8859-1?q?Sergey_Kandaurov?=) Date: Fri, 22 Mar 2024 18:41:12 +0400 Subject: [PATCH] Stream: $server_name Message-ID: <96ff3f8b7e508a345c18.1711118472@enoparse.local> # HG changeset patch # User Sergey Kandaurov # Date 1711118390 -14400 # Fri Mar 22 18:39:50 2024 +0400 # Node ID 96ff3f8b7e508a345c18bb1e1071f248f1e5a75c # Parent 6317e21a15e083a215b52480b8ef473fdfe34158 Stream: $server_name. diff --git a/src/stream/ngx_stream_variables.c b/src/stream/ngx_stream_variables.c --- a/src/stream/ngx_stream_variables.c +++ b/src/stream/ngx_stream_variables.c @@ -29,6 +29,8 @@ static ngx_int_t ngx_stream_variable_ser ngx_stream_variable_value_t *v, uintptr_t data); static ngx_int_t ngx_stream_variable_server_port(ngx_stream_session_t *s, ngx_stream_variable_value_t *v, uintptr_t data); +static ngx_int_t ngx_stream_variable_server_name(ngx_stream_session_t *s, + ngx_stream_variable_value_t *v, uintptr_t data); static ngx_int_t ngx_stream_variable_bytes(ngx_stream_session_t *s, ngx_stream_variable_value_t *v, uintptr_t data); static ngx_int_t ngx_stream_variable_session_time(ngx_stream_session_t *s, @@ -91,6 +93,9 @@ static ngx_stream_variable_t ngx_stream { ngx_string("server_port"), NULL, ngx_stream_variable_server_port, 0, 0, 0 }, + { ngx_string("server_name"), NULL, ngx_stream_variable_server_name, + 0, 0, 0 }, + { ngx_string("bytes_sent"), NULL, ngx_stream_variable_bytes, 0, 0, 0 }, @@ -722,6 +727,24 @@ ngx_stream_variable_server_port(ngx_stre static ngx_int_t +ngx_stream_variable_server_name(ngx_stream_session_t *s, + ngx_stream_variable_value_t *v, uintptr_t data) +{ + ngx_stream_core_srv_conf_t *cscf; + + cscf = ngx_stream_get_module_srv_conf(s, ngx_stream_core_module); + + v->len = cscf->server_name.len; + v->valid = 1; + v->no_cacheable = 0; + v->not_found = 0; + v->data = cscf->server_name.data; + + return NGX_OK; +} + + +static ngx_int_t ngx_stream_variable_bytes(ngx_stream_session_t *s, ngx_stream_variable_value_t *v, uintptr_t data) { From arut at nginx.com Sat Mar 23 07:25:55 2024 From: arut at nginx.com (Roman Arutyunyan) Date: Sat, 23 Mar 2024 11:25:55 +0400 Subject: [PATCH] Stream: $server_name In-Reply-To: <96ff3f8b7e508a345c18.1711118472@enoparse.local> References: <96ff3f8b7e508a345c18.1711118472@enoparse.local> Message-ID: <20240323072555.o6vwcqgoj22o4log@N00W24XTQX> Hi, On Fri, Mar 22, 2024 at 06:41:12PM +0400, Sergey Kandaurov wrote: > # HG changeset patch > # User Sergey Kandaurov > # Date 1711118390 -14400 > # Fri Mar 22 18:39:50 2024 +0400 > # Node ID 96ff3f8b7e508a345c18bb1e1071f248f1e5a75c > # Parent 6317e21a15e083a215b52480b8ef473fdfe34158 > Stream: $server_name. > > diff --git a/src/stream/ngx_stream_variables.c b/src/stream/ngx_stream_variables.c > --- a/src/stream/ngx_stream_variables.c > +++ b/src/stream/ngx_stream_variables.c > @@ -29,6 +29,8 @@ static ngx_int_t ngx_stream_variable_ser > ngx_stream_variable_value_t *v, uintptr_t data); > static ngx_int_t ngx_stream_variable_server_port(ngx_stream_session_t *s, > ngx_stream_variable_value_t *v, uintptr_t data); > +static ngx_int_t ngx_stream_variable_server_name(ngx_stream_session_t *s, > + ngx_stream_variable_value_t *v, uintptr_t data); > static ngx_int_t ngx_stream_variable_bytes(ngx_stream_session_t *s, > ngx_stream_variable_value_t *v, uintptr_t data); > static ngx_int_t ngx_stream_variable_session_time(ngx_stream_session_t *s, > @@ -91,6 +93,9 @@ static ngx_stream_variable_t ngx_stream > { ngx_string("server_port"), NULL, > ngx_stream_variable_server_port, 0, 0, 0 }, > > + { ngx_string("server_name"), NULL, ngx_stream_variable_server_name, > + 0, 0, 0 }, > + > { ngx_string("bytes_sent"), NULL, ngx_stream_variable_bytes, > 0, 0, 0 }, > > @@ -722,6 +727,24 @@ ngx_stream_variable_server_port(ngx_stre > > > static ngx_int_t > +ngx_stream_variable_server_name(ngx_stream_session_t *s, > + ngx_stream_variable_value_t *v, uintptr_t data) > +{ > + ngx_stream_core_srv_conf_t *cscf; > + > + cscf = ngx_stream_get_module_srv_conf(s, ngx_stream_core_module); > + > + v->len = cscf->server_name.len; > + v->valid = 1; > + v->no_cacheable = 0; > + v->not_found = 0; > + v->data = cscf->server_name.data; > + > + return NGX_OK; > +} > + > + > +static ngx_int_t > ngx_stream_variable_bytes(ngx_stream_session_t *s, > ngx_stream_variable_value_t *v, uintptr_t data) > { > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > https://mailman.nginx.org/mailman/listinfo/nginx-devel Looks ok From leesm815 at gmail.com Mon Mar 25 07:03:42 2024 From: leesm815 at gmail.com (Sangmin Lee) Date: Mon, 25 Mar 2024 16:03:42 +0900 Subject: I would like to post a PR for fixing memory-leak-issue Message-ID: Hello all, Recently, I have encountered a memory leak issue on the Nginx when using the gRPC stream, please refer links below for details, https://trac.nginx.org/nginx/ticket/2614 https://stackoverflow.com/questions/78112806/memory-leak-like-issue-happens-as-long-as-nginx-keeps-having-long-lived-grpc-str and I think I found a way to fix it but, unfortunately, I could not understand overall Nginx's internals, so my fix might be just a workaround to this issue which could have a potential side effect. But this fix points out, at least, where the deallocation does not happen which should do obviously and I would like the person who understands internal better to go over this fix. Therefore, I was about to upload a PR to ' https://github.com/nginx/nginx ' but it seems that this repo has not accepted the PR at all. So my question is, where would be the right place to discuss this and where should I post the PR? Best regards, Sangmin -------------- next part -------------- An HTML attachment was scrubbed... URL: From pluknet at nginx.com Mon Mar 25 14:46:55 2024 From: pluknet at nginx.com (=?utf-8?q?Sergey_Kandaurov?=) Date: Mon, 25 Mar 2024 14:46:55 +0000 Subject: [nginx] Stream: $server_name. Message-ID: details: https://hg.nginx.org/nginx/rev/96ff3f8b7e50 branches: changeset: 9225:96ff3f8b7e50 user: Sergey Kandaurov date: Fri Mar 22 18:39:50 2024 +0400 description: Stream: $server_name. diffstat: src/stream/ngx_stream_variables.c | 23 +++++++++++++++++++++++ 1 files changed, 23 insertions(+), 0 deletions(-) diffs (47 lines): diff -r 6317e21a15e0 -r 96ff3f8b7e50 src/stream/ngx_stream_variables.c --- a/src/stream/ngx_stream_variables.c Thu Jan 18 19:12:38 2024 +0400 +++ b/src/stream/ngx_stream_variables.c Fri Mar 22 18:39:50 2024 +0400 @@ -29,6 +29,8 @@ static ngx_int_t ngx_stream_variable_ser ngx_stream_variable_value_t *v, uintptr_t data); static ngx_int_t ngx_stream_variable_server_port(ngx_stream_session_t *s, ngx_stream_variable_value_t *v, uintptr_t data); +static ngx_int_t ngx_stream_variable_server_name(ngx_stream_session_t *s, + ngx_stream_variable_value_t *v, uintptr_t data); static ngx_int_t ngx_stream_variable_bytes(ngx_stream_session_t *s, ngx_stream_variable_value_t *v, uintptr_t data); static ngx_int_t ngx_stream_variable_session_time(ngx_stream_session_t *s, @@ -91,6 +93,9 @@ static ngx_stream_variable_t ngx_stream { ngx_string("server_port"), NULL, ngx_stream_variable_server_port, 0, 0, 0 }, + { ngx_string("server_name"), NULL, ngx_stream_variable_server_name, + 0, 0, 0 }, + { ngx_string("bytes_sent"), NULL, ngx_stream_variable_bytes, 0, 0, 0 }, @@ -722,6 +727,24 @@ ngx_stream_variable_server_port(ngx_stre static ngx_int_t +ngx_stream_variable_server_name(ngx_stream_session_t *s, + ngx_stream_variable_value_t *v, uintptr_t data) +{ + ngx_stream_core_srv_conf_t *cscf; + + cscf = ngx_stream_get_module_srv_conf(s, ngx_stream_core_module); + + v->len = cscf->server_name.len; + v->valid = 1; + v->no_cacheable = 0; + v->not_found = 0; + v->data = cscf->server_name.data; + + return NGX_OK; +} + + +static ngx_int_t ngx_stream_variable_bytes(ngx_stream_session_t *s, ngx_stream_variable_value_t *v, uintptr_t data) { From yar at nginx.com Mon Mar 25 15:02:44 2024 From: yar at nginx.com (=?utf-8?q?Yaroslav_Zhuravlev?=) Date: Mon, 25 Mar 2024 15:02:44 +0000 Subject: [PATCH] Mentioned fastopen in stream listen bind Message-ID: xml/en/docs/stream/ngx_stream_core_module.xml | 6 ++++-- xml/ru/docs/stream/ngx_stream_core_module.xml | 6 ++++-- 2 files changed, 8 insertions(+), 4 deletions(-) -------------- next part -------------- A non-text attachment was scrubbed... Name: nginx.org.patch Type: text/x-patch Size: 2334 bytes Desc: not available URL: From yar at nginx.com Mon Mar 25 15:06:09 2024 From: yar at nginx.com (=?utf-8?q?Yaroslav_Zhuravlev?=) Date: Mon, 25 Mar 2024 15:06:09 +0000 Subject: [PATCH] Documented accept_filter, deferred, setfib in stream listen Message-ID: <24664fe8ba2cc09e285d.1711379169@ORK-ML-00007151> xml/en/docs/stream/ngx_stream_core_module.xml | 44 ++++++++++++++++++++++++-- xml/ru/docs/stream/ngx_stream_core_module.xml | 44 ++++++++++++++++++++++++-- 2 files changed, 80 insertions(+), 8 deletions(-) -------------- next part -------------- A non-text attachment was scrubbed... Name: nginx.org.patch Type: text/x-patch Size: 6611 bytes Desc: not available URL: From pluknet at nginx.com Mon Mar 25 15:45:05 2024 From: pluknet at nginx.com (Sergey Kandaurov) Date: Mon, 25 Mar 2024 19:45:05 +0400 Subject: [PATCH 2 of 2] SSL: add $ssl_curve when using AWS-LC In-Reply-To: References: <5e923992006199748e79.1709083334@ip-172-31-36-66.ec2.internal> Message-ID: On Wed, Feb 28, 2024 at 01:22:15AM +0000, Piotr Sikora via nginx-devel wrote: > # HG changeset patch > # User Piotr Sikora > # Date 1708977632 0 > # Mon Feb 26 20:00:32 2024 +0000 > # Branch patch009 > # Node ID dfffc67d286b788204f60701ef4179566d933a1b > # Parent 5e923992006199748e79b08b1e65c4ef41f07495 > SSL: add $ssl_curve when using AWS-LC. > > Signed-off-by: Piotr Sikora Not sure nginx needs such complexity. Instead, I'd expect the missing API to be added to the library, especially that both parent and grandparent implement it. BoringSSL didn't have such API for quite some time until recently. Since AWS-LC is a fork of BoringSSL, it is welcome to sync up. As far as I can see, they used to sync quite often, so it is ought to be resolved automagically after the next sync. To ease to task, here I provide certain git hashes: 6cf98208371e5c2c8b9d34ce3b8c452ea90e2963 - SSL_get_negotiated_group 28c24092e39bfd70852afa2923a3d12d2e9be2f5 - TLSEXT_nid_unknown Minimalistic version of the patch for AWS-LC would be: diff --git a/include/openssl/ssl.h b/include/openssl/ssl.h index 7bcef522c..40075bc0a 100644 --- a/include/openssl/ssl.h +++ b/include/openssl/ssl.h @@ -2622,6 +2622,10 @@ OPENSSL_EXPORT int SSL_CTX_set1_groups_list(SSL_CTX *ctx, const char *groups); // failure. OPENSSL_EXPORT int SSL_set1_groups_list(SSL *ssl, const char *groups); +// SSL_get_negotiated_group returns the NID of the group used by |ssl|'s most +// recently completed handshake, or |NID_undef| if not applicable. +OPENSSL_EXPORT int SSL_get_negotiated_group(const SSL *ssl); + // SSL_GROUP_* define TLS group IDs. #define SSL_GROUP_SECP224R1 21 #define SSL_GROUP_SECP256R1 23 @@ -5627,6 +5631,11 @@ OPENSSL_EXPORT int SSL_CTX_set_tlsext_status_arg(SSL_CTX *ctx, void *arg); #define SSL_CURVE_SECP256R1_KYBER768_DRAFT00 SSL_GROUP_SECP256R1_KYBER768_DRAFT00 #define SSL_CURVE_X25519_KYBER768_DRAFT00 SSL_GROUP_X25519_KYBER768_DRAFT00 +// TLSEXT_nid_unknown is a constant used in OpenSSL for +// |SSL_get_negotiated_group| to return an unrecognized group. BoringSSL never +// returns this value, but we define this constant for compatibility. +#define TLSEXT_nid_unknown 0x1000000 + // SSL_get_curve_id calls |SSL_get_group_id|. OPENSSL_EXPORT uint16_t SSL_get_curve_id(const SSL *ssl); @@ -5784,6 +5793,7 @@ OPENSSL_EXPORT int SSL_set1_curves_list(SSL *ssl, const char *curves); #define SSL_get0_chain_certs SSL_get0_chain_certs #define SSL_get_max_cert_list SSL_get_max_cert_list #define SSL_get_mode SSL_get_mode +#define SSL_get_negotiated_group SSL_get_negotiated_group #define SSL_get_options SSL_get_options #define SSL_get_secure_renegotiation_support \ SSL_get_secure_renegotiation_support diff --git a/ssl/internal.h b/ssl/internal.h index 573613f72..3ce6387b5 100644 --- a/ssl/internal.h +++ b/ssl/internal.h @@ -1237,6 +1237,10 @@ bool ssl_nid_to_group_id(uint16_t *out_group_id, int nid); // true. Otherwise, it returns false. bool ssl_name_to_group_id(uint16_t *out_group_id, const char *name, size_t len); +// ssl_group_id_to_nid returns the NID corresponding to |group_id| or +// |NID_undef| if unknown. +int ssl_group_id_to_nid(uint16_t group_id); + // Handshake messages. diff --git a/ssl/ssl_key_share.cc b/ssl/ssl_key_share.cc index 7a5b2c7bb..68f8a67c7 100644 --- a/ssl/ssl_key_share.cc +++ b/ssl/ssl_key_share.cc @@ -778,6 +778,15 @@ bool ssl_name_to_group_id(uint16_t *out_group_id, const char *name, size_t len) return false; } +int ssl_group_id_to_nid(uint16_t group_id) { + for (const auto &group : kNamedGroups) { + if (group.group_id == group_id) { + return group.nid; + } + } + return NID_undef; +} + BSSL_NAMESPACE_END using namespace bssl; diff --git a/ssl/ssl_lib.cc b/ssl/ssl_lib.cc index 1069db201..17ed5224a 100644 --- a/ssl/ssl_lib.cc +++ b/ssl/ssl_lib.cc @@ -2082,6 +2082,14 @@ uint16_t SSL_get_group_id(const SSL *ssl) { return session->group_id; } +int SSL_get_negotiated_group(const SSL *ssl) { + uint16_t group_id = SSL_get_group_id(ssl); + if (group_id == 0) { + return NID_undef; + } + return ssl_group_id_to_nid(group_id); +} + int SSL_CTX_set_tmp_dh(SSL_CTX *ctx, const DH *dh) { return 1; } int SSL_set_tmp_dh(SSL *ssl, const DH *dh) { return 1; } > > diff -r 5e9239920061 -r dfffc67d286b src/event/ngx_event_openssl.c > --- a/src/event/ngx_event_openssl.c Mon Feb 26 20:00:30 2024 +0000 > +++ b/src/event/ngx_event_openssl.c Mon Feb 26 20:00:32 2024 +0000 > @@ -5163,6 +5163,72 @@ > return NGX_OK; > } > > +#elif defined(OPENSSL_IS_AWSLC) > + > + uint16_t curve_id; > + > + curve_id = SSL_get_curve_id(c->ssl->connection); > + > + /* > + * Hardcoded table with ANSI / SECG curve names (e.g. "prime256v1"), > + * which is the same format that OpenSSL returns for $ssl_curve. > + * > + * Without this table, we'd need to make 3 additional library calls > + * to convert from curve_id to ANSI / SECG curve name: > + * > + * nist_name = SSL_get_curve_name(curve_id); > + * nid = EC_curve_nist2nid(nist_name); > + * ansi_name = OBJ_nid2sn(nid); > + */ > + > + switch (curve_id) { > + > +#ifdef SSL_CURVE_SECP224R1 > + case SSL_CURVE_SECP224R1: > + ngx_str_set(s, "secp224r1"); > + return NGX_OK; > +#endif > + > +#ifdef SSL_CURVE_SECP256R1 > + case SSL_CURVE_SECP256R1: > + ngx_str_set(s, "prime256v1"); > + return NGX_OK; > +#endif > + > +#ifdef SSL_CURVE_SECP384R1 > + case SSL_CURVE_SECP384R1: > + ngx_str_set(s, "secp384r1"); > + return NGX_OK; > +#endif > + > +#ifdef SSL_CURVE_SECP521R1 > + case SSL_CURVE_SECP521R1: > + ngx_str_set(s, "secp521r1"); > + return NGX_OK; > +#endif > + > +#ifdef SSL_CURVE_X25519 > + case SSL_CURVE_X25519: > + ngx_str_set(s, "x25519"); > + return NGX_OK; > +#endif > + > + case 0: > + break; > + > + default: > + s->len = sizeof("0x0000") - 1; > + > + s->data = ngx_pnalloc(pool, s->len); > + if (s->data == NULL) { > + return NGX_ERROR; > + } > + > + ngx_sprintf(s->data, "0x%04xd", curve_id); > + > + return NGX_OK; > + } > + > #endif > > ngx_str_null(s); > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > https://mailman.nginx.org/mailman/listinfo/nginx-devel From pluknet at nginx.com Mon Mar 25 16:02:00 2024 From: pluknet at nginx.com (Sergey Kandaurov) Date: Mon, 25 Mar 2024 20:02:00 +0400 Subject: [PATCH] Documented accept_filter, deferred, setfib in stream listen In-Reply-To: <24664fe8ba2cc09e285d.1711379169@ORK-ML-00007151> References: <24664fe8ba2cc09e285d.1711379169@ORK-ML-00007151> Message-ID: <9C6A2C82-932E-400C-816F-58A79C1CC927@nginx.com> > On 25 Mar 2024, at 19:06, Yaroslav Zhuravlev wrote: > > xml/en/docs/stream/ngx_stream_core_module.xml | 44 ++++++++++++++++++++++++-- > xml/ru/docs/stream/ngx_stream_core_module.xml | 44 ++++++++++++++++++++++++-- > 2 files changed, 80 insertions(+), 8 deletions(-) > > > # HG changeset patch > # User Yaroslav Zhuravlev > # Date 1706699963 0 > # Wed Jan 31 11:19:23 2024 +0000 > # Node ID 24664fe8ba2cc09e285d6ff8c761e0d074e5af20 > # Parent b35edde818698738de2ba059de4917439e7ab754 > Documented accept_filter, deferred, setfib in stream listen. > > diff --git a/xml/en/docs/stream/ngx_stream_core_module.xml b/xml/en/docs/stream/ngx_stream_core_module.xml > --- a/xml/en/docs/stream/ngx_stream_core_module.xml > +++ b/xml/en/docs/stream/ngx_stream_core_module.xml > @@ -82,10 +82,13 @@ > [ssl] > [udp] > [proxy_protocol] > + [setfib=number] > [fastopen=number] > [backlog=number] > [rcvbuf=size] > [sndbuf=size] > + [accept_filter=filter] > + [deferred] > [bind] > [ipv6only=on|off] > [reuseport] > @@ -154,6 +157,15 @@ > > > > +setfib=number > + > + > +this parameter (1.25.4) sets the associated routing table, FIB Wrong version here and throughout the patch, otherwise looks good (together with the "fastopen bind" patch). > +(the SO_SETFIB option) for the listening socket. > +This currently works only on FreeBSD. > + > + > + > fastopen=number > > > @@ -200,6 +212,29 @@ > > > > +accept_filter=filter > + > + > +sets the name of accept filter > +(the SO_ACCEPTFILTER option) for the listening socket > +that filters incoming connections before passing them to > +accept (1.25.4). > +This works only on FreeBSD and NetBSD 5.0+. > +Possible values are > +dataready > +and > +httpready. > + > + > + > +deferred > + > + > +instructs to use a deferred accept > +(the TCP_DEFER_ACCEPT socket option) on Linux (1.25.4). > + > + > + > bind > > > @@ -212,11 +247,12 @@ > bind only to *:port. > It should be noted that the getsockname system call will be > made in this case to determine the address that accepted the connection. > -If the > +If the setfib, > fastopen, > -backlog, > -rcvbuf, sndbuf, > -ipv6only, reuseport, > +backlog, rcvbuf, > +sndbuf, accept_filter, > +deferred, ipv6only, > +reuseport, > or so_keepalive parameters > are used then for a given > address:port pair > diff --git a/xml/ru/docs/stream/ngx_stream_core_module.xml b/xml/ru/docs/stream/ngx_stream_core_module.xml > --- a/xml/ru/docs/stream/ngx_stream_core_module.xml > +++ b/xml/ru/docs/stream/ngx_stream_core_module.xml > @@ -82,10 +82,13 @@ > [ssl] > [udp] > [proxy_protocol] > + [setfib=число] > [fastopen=число] > [backlog=число] > [rcvbuf=размер] > [sndbuf=размер] > + [accept_filter=фильтр] > + [deferred] > [bind] > [ipv6only=on|off] > [reuseport] > @@ -155,6 +158,15 @@ > > > > +setfib=число > + > + > +этот параметр (1.25.4) задаёт таблицу маршрутизации, FIB > +(параметр SO_SETFIB) для слушающего сокета. > +В настоящий момент это работает только на FreeBSD. > + > + > + > fastopen=число > > > @@ -201,6 +213,29 @@ > > > > +accept_filter=фильтр > + > + > +задаёт название accept-фильтра > +(параметр SO_ACCEPTFILTER) для слушающего сокета, > +который включается для фильтрации входящих соединений > +перед передачей их в accept (1.25.4). > +Работает только на FreeBSD и NetBSD 5.0+. > +Можно использовать два фильтра: > +dataready > +и > +httpready. > + > + > + > +deferred > + > + > +указывает использовать отложенный accept > +(параметр TCP_DEFER_ACCEPT сокета) на Linux (1.25.4). > + > + > + > bind > > > @@ -214,11 +249,12 @@ > bind только на *:порт. > Необходимо заметить, что в этом случае для определения адреса, на которой > пришло соединение, делается системный вызов getsockname. > -Если же используются параметры > +Если же используются параметры setfib, > fastopen, > -backlog, > -rcvbuf, sndbuf, > -ipv6only, reuseport > +backlog, rcvbuf, > +sndbuf, accept_filter, > +deferred, ipv6only, > +reuseport > или so_keepalive, > то для данной пары > адрес:порт всегда делается > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > https://mailman.nginx.org/mailman/listinfo/nginx-devel -- Sergey Kandaurov From piotr at aviatrix.com Tue Mar 26 01:13:39 2024 From: piotr at aviatrix.com (Piotr Sikora) Date: Tue, 26 Mar 2024 01:13:39 +0000 Subject: [PATCH 2 of 2] SSL: add $ssl_curve when using AWS-LC In-Reply-To: References: <5e923992006199748e79.1709083334@ip-172-31-36-66.ec2.internal> Message-ID: Hi Sergey, > Not sure nginx needs such complexity. > Instead, I'd expect the missing API to be added to the library, > especially that both parent and grandparent implement it. > > BoringSSL didn't have such API for quite some time until recently. Right, this patch predates addition of this API to BoringSSL, and I only removed BoringSSL's #ifdef when submitting it last month. Feel free to skip it. > Since AWS-LC is a fork of BoringSSL, it is welcome to sync up. > As far as I can see, they used to sync quite often, so it is ought > to be resolved automagically after the next sync. To ease to task, > here I provide certain git hashes: > > 6cf98208371e5c2c8b9d34ce3b8c452ea90e2963 - SSL_get_negotiated_group > 28c24092e39bfd70852afa2923a3d12d2e9be2f5 - TLSEXT_nid_unknown You'd assume that's the case, but they've merged a lot of commits more recent than those two, so they've skipped them (intentionally or not). Best regards, Piotr Sikora From pluknet at nginx.com Wed Mar 27 17:46:26 2024 From: pluknet at nginx.com (=?utf-8?q?Sergey_Kandaurov?=) Date: Wed, 27 Mar 2024 17:46:26 +0000 Subject: [nginx] Geo: fixed uninitialized memory access. Message-ID: details: https://hg.nginx.org/nginx/rev/687c8306746f branches: changeset: 9226:687c8306746f user: Piotr Sikora date: Thu Mar 14 18:37:20 2024 +0400 description: Geo: fixed uninitialized memory access. While copying ngx_http_variable_value_t structures to geo binary base in ngx_http_geo_copy_values(), and similarly in the stream module, uninitialized parts of these structures are copied as well. These include the "escape" field and possible holes. Calculating crc32 of this data triggers uninitialized memory access. Found with MemorySanitizer. Signed-off-by: Piotr Sikora diffstat: src/http/modules/ngx_http_geo_module.c | 4 +--- src/stream/ngx_stream_geo_module.c | 4 +--- 2 files changed, 2 insertions(+), 6 deletions(-) diffs (42 lines): diff -r 96ff3f8b7e50 -r 687c8306746f src/http/modules/ngx_http_geo_module.c --- a/src/http/modules/ngx_http_geo_module.c Fri Mar 22 18:39:50 2024 +0400 +++ b/src/http/modules/ngx_http_geo_module.c Thu Mar 14 18:37:20 2024 +0400 @@ -1259,7 +1259,7 @@ ngx_http_geo_value(ngx_conf_t *cf, ngx_h return gvvn->value; } - val = ngx_palloc(ctx->pool, sizeof(ngx_http_variable_value_t)); + val = ngx_pcalloc(ctx->pool, sizeof(ngx_http_variable_value_t)); if (val == NULL) { return NULL; } @@ -1271,8 +1271,6 @@ ngx_http_geo_value(ngx_conf_t *cf, ngx_h } val->valid = 1; - val->no_cacheable = 0; - val->not_found = 0; gvvn = ngx_palloc(ctx->temp_pool, sizeof(ngx_http_geo_variable_value_node_t)); diff -r 96ff3f8b7e50 -r 687c8306746f src/stream/ngx_stream_geo_module.c --- a/src/stream/ngx_stream_geo_module.c Fri Mar 22 18:39:50 2024 +0400 +++ b/src/stream/ngx_stream_geo_module.c Thu Mar 14 18:37:20 2024 +0400 @@ -1209,7 +1209,7 @@ ngx_stream_geo_value(ngx_conf_t *cf, ngx return gvvn->value; } - val = ngx_palloc(ctx->pool, sizeof(ngx_stream_variable_value_t)); + val = ngx_pcalloc(ctx->pool, sizeof(ngx_stream_variable_value_t)); if (val == NULL) { return NULL; } @@ -1221,8 +1221,6 @@ ngx_stream_geo_value(ngx_conf_t *cf, ngx } val->valid = 1; - val->no_cacheable = 0; - val->not_found = 0; gvvn = ngx_palloc(ctx->temp_pool, sizeof(ngx_stream_geo_variable_value_node_t)); From pluknet at nginx.com Wed Mar 27 17:46:29 2024 From: pluknet at nginx.com (=?utf-8?q?Sergey_Kandaurov?=) Date: Wed, 27 Mar 2024 17:46:29 +0000 Subject: [nginx] Fixed undefined behaviour with IPv4-mapped IPv6 addresses. Message-ID: details: https://hg.nginx.org/nginx/rev/f208413f307f branches: changeset: 9227:f208413f307f user: Sergey Kandaurov date: Mon Mar 18 17:14:30 2024 +0400 description: Fixed undefined behaviour with IPv4-mapped IPv6 addresses. Previously, it could result when left-shifting signed integer due to implicit integer promotion, such that the most significant bit appeared on the sign bit. In practice, though, this results in the same left value as with an explicit cast, at least on known compilers, such as GCC and Clang. The reason is that in_addr_t, which is equivalent to uint32_t and same as "unsigned int" in ILP32 and LP64 data type models, has the same type width as the intermediate after integer promotion, so there's no side effects such as sign-extension. This explains why adding an explicit cast does not change object files in practice. Found with UndefinedBehaviorSanitizer (shift). Based on a patch by Piotr Sikora. diffstat: src/core/ngx_inet.c | 2 +- src/http/modules/ngx_http_access_module.c | 2 +- src/http/modules/ngx_http_geo_module.c | 4 ++-- src/http/modules/ngx_http_geoip_module.c | 2 +- src/stream/ngx_stream_access_module.c | 2 +- src/stream/ngx_stream_geo_module.c | 4 ++-- src/stream/ngx_stream_geoip_module.c | 2 +- 7 files changed, 9 insertions(+), 9 deletions(-) diffs (102 lines): diff -r 687c8306746f -r f208413f307f src/core/ngx_inet.c --- a/src/core/ngx_inet.c Thu Mar 14 18:37:20 2024 +0400 +++ b/src/core/ngx_inet.c Mon Mar 18 17:14:30 2024 +0400 @@ -507,7 +507,7 @@ ngx_cidr_match(struct sockaddr *sa, ngx_ p = inaddr6->s6_addr; - inaddr = p[12] << 24; + inaddr = (in_addr_t) p[12] << 24; inaddr += p[13] << 16; inaddr += p[14] << 8; inaddr += p[15]; diff -r 687c8306746f -r f208413f307f src/http/modules/ngx_http_access_module.c --- a/src/http/modules/ngx_http_access_module.c Thu Mar 14 18:37:20 2024 +0400 +++ b/src/http/modules/ngx_http_access_module.c Mon Mar 18 17:14:30 2024 +0400 @@ -148,7 +148,7 @@ ngx_http_access_handler(ngx_http_request p = sin6->sin6_addr.s6_addr; if (alcf->rules && IN6_IS_ADDR_V4MAPPED(&sin6->sin6_addr)) { - addr = p[12] << 24; + addr = (in_addr_t) p[12] << 24; addr += p[13] << 16; addr += p[14] << 8; addr += p[15]; diff -r 687c8306746f -r f208413f307f src/http/modules/ngx_http_geo_module.c --- a/src/http/modules/ngx_http_geo_module.c Thu Mar 14 18:37:20 2024 +0400 +++ b/src/http/modules/ngx_http_geo_module.c Mon Mar 18 17:14:30 2024 +0400 @@ -199,7 +199,7 @@ ngx_http_geo_cidr_variable(ngx_http_requ p = inaddr6->s6_addr; if (IN6_IS_ADDR_V4MAPPED(inaddr6)) { - inaddr = p[12] << 24; + inaddr = (in_addr_t) p[12] << 24; inaddr += p[13] << 16; inaddr += p[14] << 8; inaddr += p[15]; @@ -272,7 +272,7 @@ ngx_http_geo_range_variable(ngx_http_req if (IN6_IS_ADDR_V4MAPPED(inaddr6)) { p = inaddr6->s6_addr; - inaddr = p[12] << 24; + inaddr = (in_addr_t) p[12] << 24; inaddr += p[13] << 16; inaddr += p[14] << 8; inaddr += p[15]; diff -r 687c8306746f -r f208413f307f src/http/modules/ngx_http_geoip_module.c --- a/src/http/modules/ngx_http_geoip_module.c Thu Mar 14 18:37:20 2024 +0400 +++ b/src/http/modules/ngx_http_geoip_module.c Mon Mar 18 17:14:30 2024 +0400 @@ -266,7 +266,7 @@ ngx_http_geoip_addr(ngx_http_request_t * if (IN6_IS_ADDR_V4MAPPED(inaddr6)) { p = inaddr6->s6_addr; - inaddr = p[12] << 24; + inaddr = (in_addr_t) p[12] << 24; inaddr += p[13] << 16; inaddr += p[14] << 8; inaddr += p[15]; diff -r 687c8306746f -r f208413f307f src/stream/ngx_stream_access_module.c --- a/src/stream/ngx_stream_access_module.c Thu Mar 14 18:37:20 2024 +0400 +++ b/src/stream/ngx_stream_access_module.c Mon Mar 18 17:14:30 2024 +0400 @@ -144,7 +144,7 @@ ngx_stream_access_handler(ngx_stream_ses p = sin6->sin6_addr.s6_addr; if (ascf->rules && IN6_IS_ADDR_V4MAPPED(&sin6->sin6_addr)) { - addr = p[12] << 24; + addr = (in_addr_t) p[12] << 24; addr += p[13] << 16; addr += p[14] << 8; addr += p[15]; diff -r 687c8306746f -r f208413f307f src/stream/ngx_stream_geo_module.c --- a/src/stream/ngx_stream_geo_module.c Thu Mar 14 18:37:20 2024 +0400 +++ b/src/stream/ngx_stream_geo_module.c Mon Mar 18 17:14:30 2024 +0400 @@ -190,7 +190,7 @@ ngx_stream_geo_cidr_variable(ngx_stream_ p = inaddr6->s6_addr; if (IN6_IS_ADDR_V4MAPPED(inaddr6)) { - inaddr = p[12] << 24; + inaddr = (in_addr_t) p[12] << 24; inaddr += p[13] << 16; inaddr += p[14] << 8; inaddr += p[15]; @@ -263,7 +263,7 @@ ngx_stream_geo_range_variable(ngx_stream if (IN6_IS_ADDR_V4MAPPED(inaddr6)) { p = inaddr6->s6_addr; - inaddr = p[12] << 24; + inaddr = (in_addr_t) p[12] << 24; inaddr += p[13] << 16; inaddr += p[14] << 8; inaddr += p[15]; diff -r 687c8306746f -r f208413f307f src/stream/ngx_stream_geoip_module.c --- a/src/stream/ngx_stream_geoip_module.c Thu Mar 14 18:37:20 2024 +0400 +++ b/src/stream/ngx_stream_geoip_module.c Mon Mar 18 17:14:30 2024 +0400 @@ -236,7 +236,7 @@ ngx_stream_geoip_addr(ngx_stream_session if (IN6_IS_ADDR_V4MAPPED(inaddr6)) { p = inaddr6->s6_addr; - inaddr = p[12] << 24; + inaddr = (in_addr_t) p[12] << 24; inaddr += p[13] << 16; inaddr += p[14] << 8; inaddr += p[15]; From pluknet at nginx.com Wed Mar 27 17:46:32 2024 From: pluknet at nginx.com (=?utf-8?q?Sergey_Kandaurov?=) Date: Wed, 27 Mar 2024 17:46:32 +0000 Subject: [nginx] Rewrite: fixed "return" directive without response text. Message-ID: details: https://hg.nginx.org/nginx/rev/dd4a570ff4ac branches: changeset: 9228:dd4a570ff4ac user: Piotr Sikora date: Mon Feb 26 20:00:28 2024 +0000 description: Rewrite: fixed "return" directive without response text. Previously, the response text wasn't initialized and the rewrite module was sending response body set to NULL. Found with UndefinedBehaviorSanitizer (pointer-overflow). Signed-off-by: Piotr Sikora diffstat: src/http/modules/ngx_http_rewrite_module.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diffs (11 lines): diff -r f208413f307f -r dd4a570ff4ac src/http/modules/ngx_http_rewrite_module.c --- a/src/http/modules/ngx_http_rewrite_module.c Mon Mar 18 17:14:30 2024 +0400 +++ b/src/http/modules/ngx_http_rewrite_module.c Mon Feb 26 20:00:28 2024 +0000 @@ -489,6 +489,7 @@ ngx_http_rewrite_return(ngx_conf_t *cf, } if (cf->args->nelts == 2) { + ngx_str_set(&ret->text.value, ""); return NGX_CONF_OK; } From pluknet at nginx.com Wed Mar 27 17:46:35 2024 From: pluknet at nginx.com (=?utf-8?q?Sergey_Kandaurov?=) Date: Wed, 27 Mar 2024 17:46:35 +0000 Subject: [nginx] Win32: fixed unique file index calculations. Message-ID: details: https://hg.nginx.org/nginx/rev/000e2ded0a51 branches: changeset: 9229:000e2ded0a51 user: Piotr Sikora date: Mon Feb 26 20:00:35 2024 +0000 description: Win32: fixed unique file index calculations. The old code was breaking strict aliasing rules. Signed-off-by: Piotr Sikora diffstat: src/os/win32/ngx_files.h | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diffs (13 lines): diff -r dd4a570ff4ac -r 000e2ded0a51 src/os/win32/ngx_files.h --- a/src/os/win32/ngx_files.h Mon Feb 26 20:00:28 2024 +0000 +++ b/src/os/win32/ngx_files.h Mon Feb 26 20:00:35 2024 +0000 @@ -154,7 +154,8 @@ ngx_int_t ngx_file_info(u_char *filename (((off_t) (fi)->nFileSizeHigh << 32) | (fi)->nFileSizeLow) #define ngx_file_fs_size(fi) ngx_file_size(fi) -#define ngx_file_uniq(fi) (*(ngx_file_uniq_t *) &(fi)->nFileIndexHigh) +#define ngx_file_uniq(fi) \ + (((ngx_file_uniq_t) (fi)->nFileIndexHigh << 32) | (fi)->nFileIndexLow) /* 116444736000000000 is commented in src/os/win32/ngx_time.c */ From pluknet at nginx.com Wed Mar 27 17:46:38 2024 From: pluknet at nginx.com (=?utf-8?q?Sergey_Kandaurov?=) Date: Wed, 27 Mar 2024 17:46:38 +0000 Subject: [nginx] Configure: fixed Linux crypt_r() test to add libcrypt. Message-ID: details: https://hg.nginx.org/nginx/rev/fb989e24c60a branches: changeset: 9230:fb989e24c60a user: Sergey Kandaurov date: Mon Feb 26 20:00:38 2024 +0000 description: Configure: fixed Linux crypt_r() test to add libcrypt. Previously, the resulting binary was successfully linked because libcrypt was added in a separate test for crypt(). Patch by Piotr Sikora. diffstat: auto/os/linux | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diffs (14 lines): diff -r 000e2ded0a51 -r fb989e24c60a auto/os/linux --- a/auto/os/linux Mon Feb 26 20:00:35 2024 +0000 +++ b/auto/os/linux Mon Feb 26 20:00:38 2024 +0000 @@ -228,6 +228,10 @@ ngx_feature_test="struct crypt_data cd; crypt_r(\"key\", \"salt\", &cd);" . auto/feature +if [ $ngx_found = yes ]; then + CRYPT_LIB="-lcrypt" +fi + ngx_include="sys/vfs.h"; . auto/include From pluknet at nginx.com Wed Mar 27 17:46:40 2024 From: pluknet at nginx.com (=?utf-8?q?Sergey_Kandaurov?=) Date: Wed, 27 Mar 2024 17:46:40 +0000 Subject: [nginx] Detect cache line size at runtime on macOS. Message-ID: details: https://hg.nginx.org/nginx/rev/61cd12c25878 branches: changeset: 9231:61cd12c25878 user: Piotr Sikora date: Mon Feb 26 20:00:40 2024 +0000 description: Detect cache line size at runtime on macOS. Notably, Apple Silicon CPUs have 128 byte cache line size, which is twice the default configured for generic aarch64. Signed-off-by: Piotr Sikora diffstat: src/os/unix/ngx_darwin_init.c | 16 +++++++++++----- src/os/unix/ngx_posix_init.c | 5 ++++- 2 files changed, 15 insertions(+), 6 deletions(-) diffs (55 lines): diff -r fb989e24c60a -r 61cd12c25878 src/os/unix/ngx_darwin_init.c --- a/src/os/unix/ngx_darwin_init.c Mon Feb 26 20:00:38 2024 +0000 +++ b/src/os/unix/ngx_darwin_init.c Mon Feb 26 20:00:40 2024 +0000 @@ -9,11 +9,12 @@ #include -char ngx_darwin_kern_ostype[16]; -char ngx_darwin_kern_osrelease[128]; -int ngx_darwin_hw_ncpu; -int ngx_darwin_kern_ipc_somaxconn; -u_long ngx_darwin_net_inet_tcp_sendspace; +char ngx_darwin_kern_ostype[16]; +char ngx_darwin_kern_osrelease[128]; +int ngx_darwin_hw_ncpu; +int ngx_darwin_kern_ipc_somaxconn; +u_long ngx_darwin_net_inet_tcp_sendspace; +int64_t ngx_darwin_hw_cachelinesize; ngx_uint_t ngx_debug_malloc; @@ -56,6 +57,10 @@ sysctl_t sysctls[] = { &ngx_darwin_kern_ipc_somaxconn, sizeof(ngx_darwin_kern_ipc_somaxconn), 0 }, + { "hw.cachelinesize", + &ngx_darwin_hw_cachelinesize, + sizeof(ngx_darwin_hw_cachelinesize), 0 }, + { NULL, NULL, 0, 0 } }; @@ -155,6 +160,7 @@ ngx_os_specific_init(ngx_log_t *log) return NGX_ERROR; } + ngx_cacheline_size = ngx_darwin_hw_cachelinesize; ngx_ncpu = ngx_darwin_hw_ncpu; if (ngx_darwin_kern_ipc_somaxconn > 32767) { diff -r fb989e24c60a -r 61cd12c25878 src/os/unix/ngx_posix_init.c --- a/src/os/unix/ngx_posix_init.c Mon Feb 26 20:00:38 2024 +0000 +++ b/src/os/unix/ngx_posix_init.c Mon Feb 26 20:00:40 2024 +0000 @@ -51,7 +51,10 @@ ngx_os_init(ngx_log_t *log) } ngx_pagesize = getpagesize(); - ngx_cacheline_size = NGX_CPU_CACHE_LINE; + + if (ngx_cacheline_size == 0) { + ngx_cacheline_size = NGX_CPU_CACHE_LINE; + } for (n = ngx_pagesize; n >>= 1; ngx_pagesize_shift++) { /* void */ } From pluknet at nginx.com Wed Mar 27 17:46:43 2024 From: pluknet at nginx.com (=?utf-8?q?Sergey_Kandaurov?=) Date: Wed, 27 Mar 2024 17:46:43 +0000 Subject: [nginx] Configure: set cache line size for more architectures. Message-ID: details: https://hg.nginx.org/nginx/rev/427aa785edf8 branches: changeset: 9232:427aa785edf8 user: Sergey Kandaurov date: Wed Mar 27 19:36:51 2024 +0400 description: Configure: set cache line size for more architectures. Based on a patch by Piotr Sikora. diffstat: auto/os/conf | 15 +++++++++++++++ 1 files changed, 15 insertions(+), 0 deletions(-) diffs (25 lines): diff -r 61cd12c25878 -r 427aa785edf8 auto/os/conf --- a/auto/os/conf Mon Feb 26 20:00:40 2024 +0000 +++ b/auto/os/conf Wed Mar 27 19:36:51 2024 +0400 @@ -115,6 +115,21 @@ case "$NGX_MACHINE" in NGX_MACH_CACHE_LINE=64 ;; + ppc64* | powerpc64*) + have=NGX_ALIGNMENT value=16 . auto/define + NGX_MACH_CACHE_LINE=128 + ;; + + riscv64) + have=NGX_ALIGNMENT value=16 . auto/define + NGX_MACH_CACHE_LINE=64 + ;; + + s390x) + have=NGX_ALIGNMENT value=16 . auto/define + NGX_MACH_CACHE_LINE=256 + ;; + *) have=NGX_ALIGNMENT value=16 . auto/define NGX_MACH_CACHE_LINE=32 From pluknet at nginx.com Wed Mar 27 17:46:46 2024 From: pluknet at nginx.com (=?utf-8?q?Sergey_Kandaurov?=) Date: Wed, 27 Mar 2024 17:46:46 +0000 Subject: [nginx] Configure: added support for Homebrew on Apple Silicon. Message-ID: details: https://hg.nginx.org/nginx/rev/398495d816f0 branches: changeset: 9233:398495d816f0 user: Piotr Sikora date: Mon Feb 26 20:00:43 2024 +0000 description: Configure: added support for Homebrew on Apple Silicon. Signed-off-by: Piotr Sikora diffstat: auto/lib/geoip/conf | 17 +++++++++++++++++ auto/lib/google-perftools/conf | 16 ++++++++++++++++ auto/lib/libgd/conf | 17 +++++++++++++++++ auto/lib/openssl/conf | 18 ++++++++++++++++++ auto/lib/pcre/conf | 16 ++++++++++++++++ 5 files changed, 84 insertions(+), 0 deletions(-) diffs (134 lines): diff -r 427aa785edf8 -r 398495d816f0 auto/lib/geoip/conf --- a/auto/lib/geoip/conf Wed Mar 27 19:36:51 2024 +0400 +++ b/auto/lib/geoip/conf Mon Feb 26 20:00:43 2024 +0000 @@ -64,6 +64,23 @@ if [ $ngx_found = no ]; then fi +if [ $ngx_found = no ]; then + + # Homebrew on Apple Silicon + + ngx_feature="GeoIP library in /opt/homebrew/" + ngx_feature_path="/opt/homebrew/include" + + if [ $NGX_RPATH = YES ]; then + ngx_feature_libs="-R/opt/homebrew/lib -L/opt/homebrew/lib -lGeoIP" + else + ngx_feature_libs="-L/opt/homebrew/lib -lGeoIP" + fi + + . auto/feature +fi + + if [ $ngx_found = yes ]; then CORE_INCS="$CORE_INCS $ngx_feature_path" diff -r 427aa785edf8 -r 398495d816f0 auto/lib/google-perftools/conf --- a/auto/lib/google-perftools/conf Wed Mar 27 19:36:51 2024 +0400 +++ b/auto/lib/google-perftools/conf Mon Feb 26 20:00:43 2024 +0000 @@ -46,6 +46,22 @@ if [ $ngx_found = no ]; then fi +if [ $ngx_found = no ]; then + + # Homebrew on Apple Silicon + + ngx_feature="Google perftools in /opt/homebrew/" + + if [ $NGX_RPATH = YES ]; then + ngx_feature_libs="-R/opt/homebrew/lib -L/opt/homebrew/lib -lprofiler" + else + ngx_feature_libs="-L/opt/homebrew/lib -lprofiler" + fi + + . auto/feature +fi + + if [ $ngx_found = yes ]; then CORE_LIBS="$CORE_LIBS $ngx_feature_libs" diff -r 427aa785edf8 -r 398495d816f0 auto/lib/libgd/conf --- a/auto/lib/libgd/conf Wed Mar 27 19:36:51 2024 +0400 +++ b/auto/lib/libgd/conf Mon Feb 26 20:00:43 2024 +0000 @@ -65,6 +65,23 @@ if [ $ngx_found = no ]; then fi +if [ $ngx_found = no ]; then + + # Homebrew on Apple Silicon + + ngx_feature="GD library in /opt/homebrew/" + ngx_feature_path="/opt/homebrew/include" + + if [ $NGX_RPATH = YES ]; then + ngx_feature_libs="-R/opt/homebrew/lib -L/opt/homebrew/lib -lgd" + else + ngx_feature_libs="-L/opt/homebrew/lib -lgd" + fi + + . auto/feature +fi + + if [ $ngx_found = yes ]; then CORE_INCS="$CORE_INCS $ngx_feature_path" diff -r 427aa785edf8 -r 398495d816f0 auto/lib/openssl/conf --- a/auto/lib/openssl/conf Wed Mar 27 19:36:51 2024 +0400 +++ b/auto/lib/openssl/conf Mon Feb 26 20:00:43 2024 +0000 @@ -122,6 +122,24 @@ else . auto/feature fi + if [ $ngx_found = no ]; then + + # Homebrew on Apple Silicon + + ngx_feature="OpenSSL library in /opt/homebrew/" + ngx_feature_path="/opt/homebrew/include" + + if [ $NGX_RPATH = YES ]; then + ngx_feature_libs="-R/opt/homebrew/lib -L/opt/homebrew/lib -lssl -lcrypto" + else + ngx_feature_libs="-L/opt/homebrew/lib -lssl -lcrypto" + fi + + ngx_feature_libs="$ngx_feature_libs $NGX_LIBDL $NGX_LIBPTHREAD" + + . auto/feature + fi + if [ $ngx_found = yes ]; then have=NGX_SSL . auto/have CORE_INCS="$CORE_INCS $ngx_feature_path" diff -r 427aa785edf8 -r 398495d816f0 auto/lib/pcre/conf --- a/auto/lib/pcre/conf Wed Mar 27 19:36:51 2024 +0400 +++ b/auto/lib/pcre/conf Mon Feb 26 20:00:43 2024 +0000 @@ -182,6 +182,22 @@ else . auto/feature fi + if [ $ngx_found = no ]; then + + # Homebrew on Apple Silicon + + ngx_feature="PCRE library in /opt/homebrew/" + ngx_feature_path="/opt/homebrew/include" + + if [ $NGX_RPATH = YES ]; then + ngx_feature_libs="-R/opt/homebrew/lib -L/opt/homebrew/lib -lpcre" + else + ngx_feature_libs="-L/opt/homebrew/lib -lpcre" + fi + + . auto/feature + fi + if [ $ngx_found = yes ]; then CORE_INCS="$CORE_INCS $ngx_feature_path" CORE_LIBS="$CORE_LIBS $ngx_feature_libs" From pluknet at nginx.com Wed Mar 27 17:46:49 2024 From: pluknet at nginx.com (=?utf-8?q?Sergey_Kandaurov?=) Date: Wed, 27 Mar 2024 17:46:49 +0000 Subject: [nginx] Configure: fixed "make install" when cross-compiling to Windows. Message-ID: details: https://hg.nginx.org/nginx/rev/c2e753d214b0 branches: changeset: 9234:c2e753d214b0 user: Piotr Sikora date: Mon Feb 26 20:00:46 2024 +0000 description: Configure: fixed "make install" when cross-compiling to Windows. Signed-off-by: Piotr Sikora diffstat: auto/install | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff -r 398495d816f0 -r c2e753d214b0 auto/install --- a/auto/install Mon Feb 26 20:00:43 2024 +0000 +++ b/auto/install Mon Feb 26 20:00:46 2024 +0000 @@ -112,7 +112,7 @@ install: build $NGX_INSTALL_PERL_MODULES test ! -f '\$(DESTDIR)$NGX_SBIN_PATH' \\ || mv '\$(DESTDIR)$NGX_SBIN_PATH' \\ '\$(DESTDIR)$NGX_SBIN_PATH.old' - cp $NGX_OBJS/nginx '\$(DESTDIR)$NGX_SBIN_PATH' + cp $NGX_OBJS/nginx$ngx_binext '\$(DESTDIR)$NGX_SBIN_PATH' test -d '\$(DESTDIR)$NGX_CONF_PREFIX' \\ || mkdir -p '\$(DESTDIR)$NGX_CONF_PREFIX' From pluknet at nginx.com Wed Mar 27 17:46:52 2024 From: pluknet at nginx.com (=?utf-8?q?Sergey_Kandaurov?=) Date: Wed, 27 Mar 2024 17:46:52 +0000 Subject: [nginx] Configure: allow cross-compiling to Windows using Clang. Message-ID: details: https://hg.nginx.org/nginx/rev/99e7050ac886 branches: changeset: 9235:99e7050ac886 user: Piotr Sikora date: Mon Feb 26 20:00:48 2024 +0000 description: Configure: allow cross-compiling to Windows using Clang. Signed-off-by: Piotr Sikora diffstat: auto/os/win32 | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff -r c2e753d214b0 -r 99e7050ac886 auto/os/win32 --- a/auto/os/win32 Mon Feb 26 20:00:46 2024 +0000 +++ b/auto/os/win32 Mon Feb 26 20:00:48 2024 +0000 @@ -18,7 +18,7 @@ ngx_binext=".exe" case "$NGX_CC_NAME" in - gcc) + clang | gcc) CORE_LIBS="$CORE_LIBS -ladvapi32 -lws2_32" MAIN_LINK="$MAIN_LINK -Wl,--export-all-symbols" MAIN_LINK="$MAIN_LINK -Wl,--out-implib=$NGX_OBJS/libnginx.a" From pluknet at nginx.com Wed Mar 27 17:48:21 2024 From: pluknet at nginx.com (Sergey Kandaurov) Date: Wed, 27 Mar 2024 21:48:21 +0400 Subject: [PATCH] Geo: fix uninitialized memory access In-Reply-To: References: <20240314155515.f3c6bttkxzoo2unz@N00W24XTQX> Message-ID: > On 21 Mar 2024, at 10:44, Piotr Sikora via nginx-devel wrote: > > Hi Roman, > >> Thanks for the patch, looks valid, except we no longer need to explicitly >> initialize fields to zero. > > Right, I was going back-and-forth between which version I should send. > >> Also, I think we need more details about the >> uninitialized memory access. See updated patch. > > LGTM, thanks! > Pushed, thanks. -- Sergey Kandaurov From pluknet at nginx.com Wed Mar 27 17:50:15 2024 From: pluknet at nginx.com (Sergey Kandaurov) Date: Wed, 27 Mar 2024 21:50:15 +0400 Subject: [PATCH] Core: fix conversion of IPv4-mapped IPv6 addresses In-Reply-To: References: <5584232259d28489efba.1709083309@ip-172-31-36-66.ec2.internal> Message-ID: <989EE24B-8EAC-45F0-979F-C18B5B2257F8@nginx.com> > On 21 Mar 2024, at 10:53, Piotr Sikora via nginx-devel wrote: > > Hi Sergey, > >> The "shift" remark doesn't describe a problem in details. > > It's not a remark, it's the name of the UndefinedBehaviorSanitizer > check that caught the issue [1]. > Thanks for clarification, restored. >> @@ -507,7 +507,7 @@ ngx_cidr_match(struct sockaddr *sa, ngx_ >> >> p = inaddr6->s6_addr; >> >> - inaddr = p[12] << 24; >> + inaddr = (in_addr_t) p[12] << 24; >> inaddr += p[13] << 16; >> inaddr += p[14] << 8; >> inaddr += p[15]; > > While this minimizes the diff and silences the error at hand, > I find my version more readable. > > But you're obviously welcome to commit either version. > > [1] https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html > Pushed with commit log refinements, thanks. -- Sergey Kandaurov From pluknet at nginx.com Wed Mar 27 17:51:02 2024 From: pluknet at nginx.com (Sergey Kandaurov) Date: Wed, 27 Mar 2024 21:51:02 +0400 Subject: [PATCH] Rewrite: fix "return" directive without response text In-Reply-To: <3cde11b747c08c69889e.1709083317@ip-172-31-36-66.ec2.internal> References: <3cde11b747c08c69889e.1709083317@ip-172-31-36-66.ec2.internal> Message-ID: <15517DE4-99C9-4EEF-B83A-D75F9F448F55@nginx.com> > On 28 Feb 2024, at 05:21, Piotr Sikora via nginx-devel wrote: > > # HG changeset patch > # User Piotr Sikora > # Date 1708977628 0 > # Mon Feb 26 20:00:28 2024 +0000 > # Branch patch008 > # Node ID 3cde11b747c08c69889edc014a700317fe4d1d88 > # Parent 5584232259d28489efba149f2f5ae730691ff0d4 > Rewrite: fix "return" directive without response text. > > Previously, the response text wasn't initialized and the rewrite module > was sending response body set to NULL. > > Found with UndefinedBehaviorSanitizer (pointer-overflow). > > Signed-off-by: Piotr Sikora > > diff -r 5584232259d2 -r 3cde11b747c0 src/http/modules/ngx_http_rewrite_module.c > --- a/src/http/modules/ngx_http_rewrite_module.c Mon Feb 26 20:00:26 2024 +0000 > +++ b/src/http/modules/ngx_http_rewrite_module.c Mon Feb 26 20:00:28 2024 +0000 > @@ -489,6 +489,7 @@ > } > > if (cf->args->nelts == 2) { > + ngx_str_set(&ret->text.value, ""); > return NGX_CONF_OK; > } > Pushed, thanks. -- Sergey Kandaurov From pluknet at nginx.com Wed Mar 27 17:51:26 2024 From: pluknet at nginx.com (Sergey Kandaurov) Date: Wed, 27 Mar 2024 21:51:26 +0400 Subject: [PATCH] Win32: fix unique file index calculations In-Reply-To: <04e3155b3b9651fee708.1709083515@ip-172-31-36-66.ec2.internal> References: <04e3155b3b9651fee708.1709083515@ip-172-31-36-66.ec2.internal> Message-ID: > On 28 Feb 2024, at 05:25, Piotr Sikora via nginx-devel wrote: > > # HG changeset patch > # User Piotr Sikora > # Date 1708977635 0 > # Mon Feb 26 20:00:35 2024 +0000 > # Branch patch012 > # Node ID 04e3155b3b9651fee708898aaf82ac35532806ee > # Parent 9b57470dc49f8d8d10abe30a5df628732d7618dc > Win32: fix unique file index calculations. > > The old code was breaking strict aliasing rules. > > Signed-off-by: Piotr Sikora > > diff -r 9b57470dc49f -r 04e3155b3b96 src/os/win32/ngx_files.h > --- a/src/os/win32/ngx_files.h Mon Feb 26 20:00:33 2024 +0000 > +++ b/src/os/win32/ngx_files.h Mon Feb 26 20:00:35 2024 +0000 > @@ -154,7 +154,8 @@ > (((off_t) (fi)->nFileSizeHigh << 32) | (fi)->nFileSizeLow) > #define ngx_file_fs_size(fi) ngx_file_size(fi) > > -#define ngx_file_uniq(fi) (*(ngx_file_uniq_t *) &(fi)->nFileIndexHigh) > +#define ngx_file_uniq(fi) \ > + (((ngx_file_uniq_t) (fi)->nFileIndexHigh << 32) | (fi)->nFileIndexLow) > > > /* 116444736000000000 is commented in src/os/win32/ngx_time.c */ Pushed, thanks. -- Sergey Kandaurov From pluknet at nginx.com Wed Mar 27 17:52:28 2024 From: pluknet at nginx.com (Sergey Kandaurov) Date: Wed, 27 Mar 2024 21:52:28 +0400 Subject: [PATCH] Configure: link libcrypt when a feature using it is detected In-Reply-To: References: <570e97dddeeddb79c715.1709083399@ip-172-31-36-66.ec2.internal> <67E7F428-19EB-4370-B117-78FF9D6BEC18@nginx.com> Message-ID: > On 21 Mar 2024, at 11:23, Piotr Sikora via nginx-devel wrote: > > Hi Sergey, > >> I'd rewrote the description to be more specific: >> >> : Configure: fixed Linux crypt_r() test to add libcrypt. >> : >> : Previously, the resulting binary was successfully linked >> : because libcrypt was added in a separate test for crypt(). > > That's fine with me. > > Best regards, > Piotr Sikora Pushed with the updated description, thanks. -- Sergey Kandaurov From pluknet at nginx.com Wed Mar 27 17:53:18 2024 From: pluknet at nginx.com (Sergey Kandaurov) Date: Wed, 27 Mar 2024 21:53:18 +0400 Subject: [PATCH] macOS: detect cache line size at runtime In-Reply-To: References: Message-ID: <8386E2CF-F72D-4134-B501-4EFA9F88CEEF@nginx.com> > On 21 Mar 2024, at 11:19, Piotr Sikora via nginx-devel wrote: > > Hi Sergey, > >> I prefer not to introduce more ad-hoc prefixes in the log summary. >> Something like moving the "macOS" part to the end should be fine. > > That's fine with me. > >> style: this breaks a perfect indentation of two spaces after type; >> further, it appears to be unsorted by type; I'd put it after u_long > > Good catch, thanks! > >> This makes the following slight update to the patch. >> If you're okey with it, I will commit it then. > > LGTM. > Pushed the updated patch, thanks. -- Sergey Kandaurov From pluknet at nginx.com Wed Mar 27 17:57:32 2024 From: pluknet at nginx.com (Sergey Kandaurov) Date: Wed, 27 Mar 2024 21:57:32 +0400 Subject: [PATCH] Configure: set cache line sizes for more architectures In-Reply-To: References: Message-ID: <9BC4011C-7F14-409E-8239-086407EA5749@nginx.com> > On 28 Feb 2024, at 05:24, Piotr Sikora via nginx-devel wrote: > > # HG changeset patch > # User Piotr Sikora > # Date 1708977642 0 > # Mon Feb 26 20:00:42 2024 +0000 > # Branch patch016 > # Node ID bb99cbe3a343ae581d2369b990aee66e69679ca2 > # Parent f58bc1041ebca635517b919d58b49923bf24f76d > Configure: set cache line sizes for more architectures. > > Signed-off-by: Piotr Sikora Thanks for the patch. > > diff -r f58bc1041ebc -r bb99cbe3a343 auto/os/conf > --- a/auto/os/conf Mon Feb 26 20:00:40 2024 +0000 > +++ b/auto/os/conf Mon Feb 26 20:00:42 2024 +0000 > @@ -115,6 +115,21 @@ > NGX_MACH_CACHE_LINE=64 > ;; > > + ppc64 | ppc64le) This can be replaced with a wildcard. Also, other systems may report a different value, e.g.: $ grep -r MACHINE_ARCH sys/powerpc/include/param.h sys/powerpc/include/param.h:#ifndef MACHINE_ARCH sys/powerpc/include/param.h:#define MACHINE_ARCH "powerpc64le" sys/powerpc/include/param.h:#define MACHINE_ARCH "powerpc64" sys/powerpc/include/param.h:#define MACHINE_ARCH "powerpcspe" sys/powerpc/include/param.h:#define MACHINE_ARCH "powerpc" sys/powerpc/include/param.h:#ifndef MACHINE_ARCH32 sys/powerpc/include/param.h:#define MACHINE_ARCH32 "powerpc" As such, I pushed an updated version: https://hg.nginx.org/nginx/rev/427aa785edf8 > + have=NGX_ALIGNMENT value=16 . auto/define > + NGX_MACH_CACHE_LINE=128 > + ;; > + > + riscv64) > + have=NGX_ALIGNMENT value=16 . auto/define > + NGX_MACH_CACHE_LINE=64 > + ;; > + > + s390x) > + have=NGX_ALIGNMENT value=16 . auto/define > + NGX_MACH_CACHE_LINE=256 > + ;; > + > *) > have=NGX_ALIGNMENT value=16 . auto/define > NGX_MACH_CACHE_LINE=32 -- Sergey Kandaurov From pluknet at nginx.com Wed Mar 27 17:58:18 2024 From: pluknet at nginx.com (Sergey Kandaurov) Date: Wed, 27 Mar 2024 21:58:18 +0400 Subject: [PATCH] Configure: add support for Homebrew on Apple Silicon In-Reply-To: References: <2E747794-8E5F-4B41-B0AF-422DC7A10629@nginx.com> Message-ID: <41FE9EDA-B68A-415E-81B3-0B2AED770D07@nginx.com> > On 11 Mar 2024, at 19:14, Sergey Kandaurov wrote: > > >> On 8 Mar 2024, at 19:31, Piotr Sikora via nginx-devel wrote: >> >> Hi Sergey, >> >>> An obvious question is why do you need this change. Homebrew seems >>> to be quite niche to pay attention. >> >> Homebrew [1] is orders of magnitude more popular than MacPorts [2], >> which is already supported by the configure script. > > Thanks for clarifying this. > >> >>> Using appropriate paths in >>> --with-cc-opt / --with-ld-opt should work (not tested). >> >> Everything under auto/lib can be replaced with --with-{cc,ld}-opt, >> so I don't really understand this reasoning. > > Given the high popularity, it may be indeed not an option. > >> >>> A quick grep for MacPorts search paths suggests that some libraries >>> are missing in the change. If this is on purpose, please reflect >>> this in the description. >> >> libxml2, libxslt, and libexslt are all installed as part of Xcode, >> which is required to use Homebrew and compile anything on macOS. > > This raises a question whether we need to test it for MacPorts as well, > which also requires SDK. Obviously, it is out of scope of this patch. > >> >> I'll ship update patch in a moment. >> >>> Apple Silicon is something from the marketing language, >>> using Apple ARM instead should be fine. >>> >>> Notably, Homebrew uses Hardware::CPU.arm Ruby language boolean >>> to make the distinction. >> >> There is no such thing as "Apple ARM". >> >> The official documentation uses the term "Apple silicon" [3], >> Homebrew refers to the supported platform as "Apple Silicon" [4], >> and Wikipedia has an article about "Apple silicon" [5]. > > Thank you for clarification. > >> >>> Further, given the smooth decay on Intel-based hardware, >>> I'd reduce this just to "Homebrew". >> >> But that would be misleading, seeing that the new code path doesn't do >> anything for Homebrew on Intel. >> >> And then, there is Homebrew on Linux [6]. > > Given all the above argumentation, it makes sense > to push the patch just as is. Pushed, thanks. -- Sergey Kandaurov From pluknet at nginx.com Wed Mar 27 17:59:14 2024 From: pluknet at nginx.com (Sergey Kandaurov) Date: Wed, 27 Mar 2024 21:59:14 +0400 Subject: [PATCH] Configure: fix "make install" when cross-compiling to Windows In-Reply-To: References: Message-ID: <801A1E51-CACF-451B-8B55-B2E4AF9E599C@nginx.com> > On 28 Feb 2024, at 05:25, Piotr Sikora via nginx-devel wrote: > > # HG changeset patch > # User Piotr Sikora > # Date 1708977646 0 > # Mon Feb 26 20:00:46 2024 +0000 > # Branch patch018 > # Node ID ea1ab31c166c52372b40429a1cccece9ec9e003b > # Parent dd95daa55cf6131a7e845edd6ad3b429bcef6f98 > Configure: fix "make install" when cross-compiling to Windows. > > Signed-off-by: Piotr Sikora Both patches for crossbuilding on win32 pushed, thanks. -- Sergey Kandaurov