From wkpark at gmail.com  Tue Nov  1 11:15:48 2022
From: wkpark at gmail.com (Won-Kyu Park)
Date: Tue, 1 Nov 2022 20:15:48 +0900
Subject: [PATCH] fix ngx_realpath() for win32
Message-ID: <CAJ3S5=ciooYdcM7sMqURKU32B1XzYe_E8hgPfUbQxjU-9eYJkg@mail.gmail.com>

# HG changeset patch
# User Won-Kyu Park <wkpark at gmail.com>
# Date 1667298839 -32400
#      Tue Nov 01 19:33:59 2022 +0900
# Node ID c964b33d60679f5cd8d43a8f859d16c2ac6f89e1
# Parent  1ae25660c0c76edef14121ca64362f28b9d57a70
fix ngx_realpath() for win32

diff -r 1ae25660c0c7 -r c964b33d6067 src/os/win32/ngx_files.c
--- a/src/os/win32/ngx_files.c Wed Oct 19 10:56:21 2022 +0300
+++ b/src/os/win32/ngx_files.c Tue Nov 01 19:33:59 2022 +0900
@@ -416,14 +416,6 @@
 }


-u_char *
-ngx_realpath(u_char *path, u_char *resolved)
-{
-    /* STUB */
-    return path;
-}
-
-
 ngx_int_t
 ngx_open_dir(ngx_str_t *name, ngx_dir_t *dir)
 {
diff -r 1ae25660c0c7 -r c964b33d6067 src/os/win32/ngx_files.h
--- a/src/os/win32/ngx_files.h Wed Oct 19 10:56:21 2022 +0300
+++ b/src/os/win32/ngx_files.h Tue Nov 01 19:33:59 2022 +0900
@@ -172,7 +172,7 @@
 void ngx_close_file_mapping(ngx_file_mapping_t *fm);


-u_char *ngx_realpath(u_char *path, u_char *resolved);
+#define ngx_realpath(path, resolved) (u_char *)_fullpath((char
*)(resolved), (char *)(path), MAX_PATH)
 #define ngx_realpath_n              ""
 #define ngx_getcwd(buf, size)       GetCurrentDirectory(size, (char *) buf)
 #define ngx_getcwd_n                "GetCurrentDirectory()"

---- END --------
This small fix would set $realpath_root as expected for win32.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20221101/a8a64efe/attachment.htm>

From mdounin at mdounin.ru  Tue Nov  1 14:14:02 2022
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 1 Nov 2022 17:14:02 +0300
Subject: [PATCH] Core: support for reading PROXY protocol v2 TLVs
In-Reply-To: <20221031120700.z5o4el64rhhpfn3y@N00W24XTQX>
References: <YxYcrmqdL6HA4jQD@mdounin.ru>
 <20220909154658.fpnpndo2opnnzywx@N00W24XTQX>
 <Yx+k6bMb52CndRJy@mdounin.ru>
 <20220913150304.k2fjjdxgesgzbilu@N00W24XTQX>
 <YykpmhVRzb3PXbT3@mdounin.ru>
 <20220927094125.w7oo4g2quw3yyqfh@N00W24XTQX>
 <Y0TE9H4TlGJE1aTE@mdounin.ru>
 <20221011130111.oiljq55eydpp3dh6@N00W24XTQX>
 <Y0XX5Hu9dFNmyS67@mdounin.ru>
 <20221031120700.z5o4el64rhhpfn3y@N00W24XTQX>
Message-ID: <Y2Epqv3Aw4X3/zPM@mdounin.ru>

Hello!

On Mon, Oct 31, 2022 at 04:07:00PM +0400, Roman Arutyunyan wrote:

> While testing the feature, it became clear that 107 bytes as maximum PROXY
> protocol header size may not be enough.  This limit came from v1 and stayed
> unchanged when v2 was added.  With this limit, there are only 79 bytes left for
> TLVs in case of IPv4 and 55 bytes in case of IPv6.
> 
> Attached is a patch that increases buffer size up to 65K while reading PROXY
> protocl header.  Writing is not changed since only v1 is supported.

[...]

> # HG changeset patch
> # User Roman Arutyunyan <arut at nginx.com>
> # Date 1667216033 -14400
> #      Mon Oct 31 15:33:53 2022 +0400
> # Node ID 8c99314f90eccc2ad5aaf4b3de5368e964c4ffe0
> # Parent  81b4326daac70d6de70abbc3fe36d4f6e3da54a2
> Increased maximum read PROXY protocol header size.
> 
> Maximum size for reading the PROXY protocol header is increased to 65536 to
> accommodate a bigger number of TLVs, which are supported since cca4c8a715de.
> 
> Maximum size for writing the PROXY protocol header is not changed since only
> version 1 is currently supported.
> 
> diff --git a/src/core/ngx_proxy_protocol.c b/src/core/ngx_proxy_protocol.c
> --- a/src/core/ngx_proxy_protocol.c
> +++ b/src/core/ngx_proxy_protocol.c
> @@ -281,7 +281,7 @@ ngx_proxy_protocol_write(ngx_connection_
>  {
>      ngx_uint_t  port, lport;
>  
> -    if (last - buf < NGX_PROXY_PROTOCOL_MAX_HEADER) {
> +    if (last - buf < NGX_PROXY_PROTOCOL_V1_MAX_HEADER) {
>          return NULL;
>      }
>  
> diff --git a/src/core/ngx_proxy_protocol.h b/src/core/ngx_proxy_protocol.h
> --- a/src/core/ngx_proxy_protocol.h
> +++ b/src/core/ngx_proxy_protocol.h
> @@ -13,7 +13,8 @@
>  #include <ngx_core.h>
>  
>  
> -#define NGX_PROXY_PROTOCOL_MAX_HEADER  107
> +#define NGX_PROXY_PROTOCOL_V1_MAX_HEADER  107
> +#define NGX_PROXY_PROTOCOL_MAX_HEADER     65536
>  
>  
>  struct ngx_proxy_protocol_s {
> diff --git a/src/http/ngx_http_request.c b/src/http/ngx_http_request.c
> --- a/src/http/ngx_http_request.c
> +++ b/src/http/ngx_http_request.c
> @@ -641,7 +641,7 @@ ngx_http_alloc_request(ngx_connection_t 
>  static void
>  ngx_http_ssl_handshake(ngx_event_t *rev)
>  {
> -    u_char                    *p, buf[NGX_PROXY_PROTOCOL_MAX_HEADER + 1];
> +    u_char                    *p;
>      size_t                     size;
>      ssize_t                    n;
>      ngx_err_t                  err;
> @@ -651,6 +651,7 @@ ngx_http_ssl_handshake(ngx_event_t *rev)
>      ngx_http_ssl_srv_conf_t   *sscf;
>      ngx_http_core_loc_conf_t  *clcf;
>      ngx_http_core_srv_conf_t  *cscf;
> +    static u_char              buf[NGX_PROXY_PROTOCOL_MAX_HEADER + 1];
>  

I'm somewhat sceptical about the idea of allocation of up to 3 
global 64k buffers, especially given that the protocol requires 
atomic sending of the header, which means it cannot reliably 
exceed 1500 bytes in most configurations.

Further, the maximum size of the proxy protocol header in non-SSL 
case is still limited by client_header_buffer_size, which is 1024 
bytes by default.  Assuming there will be headers which does not 
fit into 1k buffer, it is not clear why these will magically work 
in case of SSL, but will require client_header_buffer_size tuning 
otherwise.

Might be some dynamically allocated buffer, sized after 
client_header_buffer_size, like in non-SSL case, would be a better 
idea?

(It also might make sense to avoid assuming atomicity, which 
clearly cannot be guaranteed if the header is larger than MTU.  
This will also require dynamically allocated per-connection 
buffers.)

Alternatively, we can consider using some larger-than-now on-stack 
buffers, something like 4k should be acceptable given we use such 
buffers in other places anyway.  It should be enough for most 
proxy protocol headers.

[...]

-- 
Maxim Dounin
http://mdounin.ru/


From mdounin at mdounin.ru  Tue Nov  1 15:29:28 2022
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 1 Nov 2022 18:29:28 +0300
Subject: [PATCH] fix ngx_realpath() for win32
In-Reply-To: <CAJ3S5=ciooYdcM7sMqURKU32B1XzYe_E8hgPfUbQxjU-9eYJkg@mail.gmail.com>
References: <CAJ3S5=ciooYdcM7sMqURKU32B1XzYe_E8hgPfUbQxjU-9eYJkg@mail.gmail.com>
Message-ID: <Y2E7WDilO+sDGzKS@mdounin.ru>

Hello!

On Tue, Nov 01, 2022 at 08:15:48PM +0900, Won-Kyu Park wrote:

> # HG changeset patch
> # User Won-Kyu Park <wkpark at gmail.com>
> # Date 1667298839 -32400
> #      Tue Nov 01 19:33:59 2022 +0900
> # Node ID c964b33d60679f5cd8d43a8f859d16c2ac6f89e1
> # Parent  1ae25660c0c76edef14121ca64362f28b9d57a70
> fix ngx_realpath() for win32
> 
> diff -r 1ae25660c0c7 -r c964b33d6067 src/os/win32/ngx_files.c
> --- a/src/os/win32/ngx_files.c Wed Oct 19 10:56:21 2022 +0300
> +++ b/src/os/win32/ngx_files.c Tue Nov 01 19:33:59 2022 +0900
> @@ -416,14 +416,6 @@
>  }
> 
> 
> -u_char *
> -ngx_realpath(u_char *path, u_char *resolved)
> -{
> -    /* STUB */
> -    return path;
> -}
> -
> -
>  ngx_int_t
>  ngx_open_dir(ngx_str_t *name, ngx_dir_t *dir)
>  {
> diff -r 1ae25660c0c7 -r c964b33d6067 src/os/win32/ngx_files.h
> --- a/src/os/win32/ngx_files.h Wed Oct 19 10:56:21 2022 +0300
> +++ b/src/os/win32/ngx_files.h Tue Nov 01 19:33:59 2022 +0900
> @@ -172,7 +172,7 @@
>  void ngx_close_file_mapping(ngx_file_mapping_t *fm);
> 
> 
> -u_char *ngx_realpath(u_char *path, u_char *resolved);
> +#define ngx_realpath(path, resolved) (u_char *)_fullpath((char
> *)(resolved), (char *)(path), MAX_PATH)
>  #define ngx_realpath_n              ""
>  #define ngx_getcwd(buf, size)       GetCurrentDirectory(size, (char *) buf)
>  #define ngx_getcwd_n                "GetCurrentDirectory()"
> 
> ---- END --------
> This small fix would set $realpath_root as expected for win32.

Could you please clarify "as expected"?

It doesn't look like _fullpath() resolves symbolic links, which is 
expected behaviour for $realpath_root.

Rather, it looks like GetFinalPathNameByHandle() should be used if 
we want to actually support $realpath_root as it is expected to 
work, that is, providing path with symlinks resolved.  It might be 
tricky to use it though, as it is only available in Windows Vista 
or later.

-- 
Maxim Dounin
http://mdounin.ru/


From wkpark at gmail.com  Wed Nov  2 06:43:50 2022
From: wkpark at gmail.com (Won-Kyu Park)
Date: Wed, 2 Nov 2022 15:43:50 +0900
Subject: [PATCH] fix ngx_realpath() for win32
In-Reply-To: <CAJ3S5=ciooYdcM7sMqURKU32B1XzYe_E8hgPfUbQxjU-9eYJkg@mail.gmail.com>
References: <CAJ3S5=ciooYdcM7sMqURKU32B1XzYe_E8hgPfUbQxjU-9eYJkg@mail.gmail.com>
Message-ID: <CAJ3S5=crj7pFw6sCWLudWoGAUSRkPtV7yYcLgqH2w9WHXNnKow@mail.gmail.com>

> Hello!
>
> On Tue, Nov 01, 2022 at 08:15:48PM +0900, Won-Kyu Park wrote:
>
> ...
>
> Could you please clarify "as expected"?
>
> It doesn't look like _fullpath() resolves symbolic links, which is expected behaviour for $realpath_root.
>

You are right but for most situations, translating a relative path to
fullpath is enough, at least for me. (please see use case)

> Rather, it looks like GetFinalPathNameByHandle() should be used if we want to actually support $realpath_root as it is expected to work, that is, providing path with symlinks resolved. It might
> be tricky to use it though, as it is only available in Windows Vista or later.
>
> -- Maxim Dounin http://mdounin.ru/

yes. GetFinalPathNameByHandle() is not available in pre-Vista.
full implementation might be preferred but this simple oneliner solve
most use cases I guess.

- win32/mingw32 only support _fullpath()
- PHP7 use GetFinalPathNameByHandleW()
-------------

use case:

server {
        location / {
            root   ../to/html; # works nicely without problem
            index  index.php index.html index.htm;
        }

        location ~ \.php$ {
            root           ../to/html; # not work. $realpath_root
gives wrong value (c:\foobar\nginxroot\..\wiki), $document_root
returns relative path
            # for this case, $realpath_root gives expected value with
this _fullpath fix.
            #root c:/to/my/absolute/path/html; # works fine
            fastcgi_pass   127.0.0.1:9000;
            fastcgi_index  index.php;

            # fix to use realpath_root
            #fastcgi_param SCRIPT_FILENAME
$document_root$fastcgi_script_name; # not work.
            fastcgi_param SCRIPT_FILENAME
$realpath_root$fastcgi_script_name; # works fine with _fullpath fix.
            ...
        }
}


From arut at nginx.com  Wed Nov  2 13:06:25 2022
From: arut at nginx.com (Roman Arutyunyan)
Date: Wed, 2 Nov 2022 17:06:25 +0400
Subject: [PATCH] Core: support for reading PROXY protocol v2 TLVs
In-Reply-To: <Y2Epqv3Aw4X3/zPM@mdounin.ru>
References: <20220909154658.fpnpndo2opnnzywx@N00W24XTQX>
 <Yx+k6bMb52CndRJy@mdounin.ru>
 <20220913150304.k2fjjdxgesgzbilu@N00W24XTQX>
 <YykpmhVRzb3PXbT3@mdounin.ru>
 <20220927094125.w7oo4g2quw3yyqfh@N00W24XTQX>
 <Y0TE9H4TlGJE1aTE@mdounin.ru>
 <20221011130111.oiljq55eydpp3dh6@N00W24XTQX>
 <Y0XX5Hu9dFNmyS67@mdounin.ru>
 <20221031120700.z5o4el64rhhpfn3y@N00W24XTQX>
 <Y2Epqv3Aw4X3/zPM@mdounin.ru>
Message-ID: <20221102130625.3tmxup3rythk6wkx@N00W24XTQX>

Hi,

On Tue, Nov 01, 2022 at 05:14:02PM +0300, Maxim Dounin wrote:
> Hello!
> 
> On Mon, Oct 31, 2022 at 04:07:00PM +0400, Roman Arutyunyan wrote:
> 
> > While testing the feature, it became clear that 107 bytes as maximum PROXY
> > protocol header size may not be enough.  This limit came from v1 and stayed
> > unchanged when v2 was added.  With this limit, there are only 79 bytes left for
> > TLVs in case of IPv4 and 55 bytes in case of IPv6.
> > 
> > Attached is a patch that increases buffer size up to 65K while reading PROXY
> > protocl header.  Writing is not changed since only v1 is supported.
> 
> [...]
> 
> > # HG changeset patch
> > # User Roman Arutyunyan <arut at nginx.com>
> > # Date 1667216033 -14400
> > #      Mon Oct 31 15:33:53 2022 +0400
> > # Node ID 8c99314f90eccc2ad5aaf4b3de5368e964c4ffe0
> > # Parent  81b4326daac70d6de70abbc3fe36d4f6e3da54a2
> > Increased maximum read PROXY protocol header size.
> > 
> > Maximum size for reading the PROXY protocol header is increased to 65536 to
> > accommodate a bigger number of TLVs, which are supported since cca4c8a715de.
> > 
> > Maximum size for writing the PROXY protocol header is not changed since only
> > version 1 is currently supported.
> > 
> > diff --git a/src/core/ngx_proxy_protocol.c b/src/core/ngx_proxy_protocol.c
> > --- a/src/core/ngx_proxy_protocol.c
> > +++ b/src/core/ngx_proxy_protocol.c
> > @@ -281,7 +281,7 @@ ngx_proxy_protocol_write(ngx_connection_
> >  {
> >      ngx_uint_t  port, lport;
> >  
> > -    if (last - buf < NGX_PROXY_PROTOCOL_MAX_HEADER) {
> > +    if (last - buf < NGX_PROXY_PROTOCOL_V1_MAX_HEADER) {
> >          return NULL;
> >      }
> >  
> > diff --git a/src/core/ngx_proxy_protocol.h b/src/core/ngx_proxy_protocol.h
> > --- a/src/core/ngx_proxy_protocol.h
> > +++ b/src/core/ngx_proxy_protocol.h
> > @@ -13,7 +13,8 @@
> >  #include <ngx_core.h>
> >  
> >  
> > -#define NGX_PROXY_PROTOCOL_MAX_HEADER  107
> > +#define NGX_PROXY_PROTOCOL_V1_MAX_HEADER  107
> > +#define NGX_PROXY_PROTOCOL_MAX_HEADER     65536
> >  
> >  
> >  struct ngx_proxy_protocol_s {
> > diff --git a/src/http/ngx_http_request.c b/src/http/ngx_http_request.c
> > --- a/src/http/ngx_http_request.c
> > +++ b/src/http/ngx_http_request.c
> > @@ -641,7 +641,7 @@ ngx_http_alloc_request(ngx_connection_t 
> >  static void
> >  ngx_http_ssl_handshake(ngx_event_t *rev)
> >  {
> > -    u_char                    *p, buf[NGX_PROXY_PROTOCOL_MAX_HEADER + 1];
> > +    u_char                    *p;
> >      size_t                     size;
> >      ssize_t                    n;
> >      ngx_err_t                  err;
> > @@ -651,6 +651,7 @@ ngx_http_ssl_handshake(ngx_event_t *rev)
> >      ngx_http_ssl_srv_conf_t   *sscf;
> >      ngx_http_core_loc_conf_t  *clcf;
> >      ngx_http_core_srv_conf_t  *cscf;
> > +    static u_char              buf[NGX_PROXY_PROTOCOL_MAX_HEADER + 1];
> >  
> 
> I'm somewhat sceptical about the idea of allocation of up to 3 
> global 64k buffers, especially given that the protocol requires 
> atomic sending of the header, which means it cannot reliably 
> exceed 1500 bytes in most configurations.
> 
> Further, the maximum size of the proxy protocol header in non-SSL 
> case is still limited by client_header_buffer_size, which is 1024 
> bytes by default.  Assuming there will be headers which does not 
> fit into 1k buffer, it is not clear why these will magically work 
> in case of SSL, but will require client_header_buffer_size tuning 
> otherwise.

It will require tuning in some cases and work automatically in others.
Currently it only works where tuning is available.

> Might be some dynamically allocated buffer, sized after 
> client_header_buffer_size, like in non-SSL case, would be a better 
> idea?

We read PROXY protocol in mail and stream as well, which would require
similar settings there.

> (It also might make sense to avoid assuming atomicity, which 
> clearly cannot be guaranteed if the header is larger than MTU.  
> This will also require dynamically allocated per-connection 
> buffers.)

Yes, dynamic per-connection buffers is what I've been trying to avoid.

> Alternatively, we can consider using some larger-than-now on-stack 
> buffers, something like 4k should be acceptable given we use such 
> buffers in other places anyway.  It should be enough for most 
> proxy protocol headers.

I like this alternative.  This was in fact my first thought, but the actual
number was (and still is) hard to pick.  I suggest that we increase the
on-stack buffer for reading and leave the writing buffer size as is for now.
The writing part will need refactoring anyway when we add ppv2 support.

--
Roman Arutyunyan
-------------- next part --------------
# HG changeset patch
# User Roman Arutyunyan <arut at nginx.com>
# Date 1667382376 -14400
#      Wed Nov 02 13:46:16 2022 +0400
# Node ID dc5f16e6a243c15f58e2c6a62f7a83f536729174
# Parent  81b4326daac70d6de70abbc3fe36d4f6e3da54a2
Increased maximum read PROXY protocol header size.

Maximum size for reading the PROXY protocol header is increased to 4096 to
accommodate a bigger number of TLVs, which are supported since cca4c8a715de.

Maximum size for writing the PROXY protocol header is not changed since only
version 1 is currently supported.

diff --git a/src/core/ngx_proxy_protocol.c b/src/core/ngx_proxy_protocol.c
--- a/src/core/ngx_proxy_protocol.c
+++ b/src/core/ngx_proxy_protocol.c
@@ -281,7 +281,7 @@ ngx_proxy_protocol_write(ngx_connection_
 {
     ngx_uint_t  port, lport;
 
-    if (last - buf < NGX_PROXY_PROTOCOL_MAX_HEADER) {
+    if (last - buf < NGX_PROXY_PROTOCOL_V1_MAX_HEADER) {
         return NULL;
     }
 
diff --git a/src/core/ngx_proxy_protocol.h b/src/core/ngx_proxy_protocol.h
--- a/src/core/ngx_proxy_protocol.h
+++ b/src/core/ngx_proxy_protocol.h
@@ -13,7 +13,8 @@
 #include <ngx_core.h>
 
 
-#define NGX_PROXY_PROTOCOL_MAX_HEADER  107
+#define NGX_PROXY_PROTOCOL_V1_MAX_HEADER  107
+#define NGX_PROXY_PROTOCOL_MAX_HEADER     4096
 
 
 struct ngx_proxy_protocol_s {
diff --git a/src/mail/ngx_mail_proxy_module.c b/src/mail/ngx_mail_proxy_module.c
--- a/src/mail/ngx_mail_proxy_module.c
+++ b/src/mail/ngx_mail_proxy_module.c
@@ -890,7 +890,7 @@ ngx_mail_proxy_send_proxy_protocol(ngx_m
     u_char            *p;
     ssize_t            n, size;
     ngx_connection_t  *c;
-    u_char             buf[NGX_PROXY_PROTOCOL_MAX_HEADER];
+    u_char             buf[NGX_PROXY_PROTOCOL_V1_MAX_HEADER];
 
     s->connection->log->action = "sending PROXY protocol header to upstream";
 
@@ -898,7 +898,7 @@ ngx_mail_proxy_send_proxy_protocol(ngx_m
                    "mail proxy send PROXY protocol header");
 
     p = ngx_proxy_protocol_write(s->connection, buf,
-                                 buf + NGX_PROXY_PROTOCOL_MAX_HEADER);
+                                 buf + NGX_PROXY_PROTOCOL_V1_MAX_HEADER);
     if (p == NULL) {
         ngx_mail_proxy_internal_server_error(s);
         return NGX_ERROR;
diff --git a/src/stream/ngx_stream_proxy_module.c b/src/stream/ngx_stream_proxy_module.c
--- a/src/stream/ngx_stream_proxy_module.c
+++ b/src/stream/ngx_stream_proxy_module.c
@@ -894,7 +894,7 @@ ngx_stream_proxy_init_upstream(ngx_strea
             return;
         }
 
-        p = ngx_pnalloc(c->pool, NGX_PROXY_PROTOCOL_MAX_HEADER);
+        p = ngx_pnalloc(c->pool, NGX_PROXY_PROTOCOL_V1_MAX_HEADER);
         if (p == NULL) {
             ngx_stream_proxy_finalize(s, NGX_STREAM_INTERNAL_SERVER_ERROR);
             return;
@@ -902,7 +902,8 @@ ngx_stream_proxy_init_upstream(ngx_strea
 
         cl->buf->pos = p;
 
-        p = ngx_proxy_protocol_write(c, p, p + NGX_PROXY_PROTOCOL_MAX_HEADER);
+        p = ngx_proxy_protocol_write(c, p,
+                                     p + NGX_PROXY_PROTOCOL_V1_MAX_HEADER);
         if (p == NULL) {
             ngx_stream_proxy_finalize(s, NGX_STREAM_INTERNAL_SERVER_ERROR);
             return;
@@ -946,14 +947,15 @@ ngx_stream_proxy_send_proxy_protocol(ngx
     ngx_connection_t             *c, *pc;
     ngx_stream_upstream_t        *u;
     ngx_stream_proxy_srv_conf_t  *pscf;
-    u_char                        buf[NGX_PROXY_PROTOCOL_MAX_HEADER];
+    u_char                        buf[NGX_PROXY_PROTOCOL_V1_MAX_HEADER];
 
     c = s->connection;
 
     ngx_log_debug0(NGX_LOG_DEBUG_STREAM, c->log, 0,
                    "stream proxy send PROXY protocol header");
 
-    p = ngx_proxy_protocol_write(c, buf, buf + NGX_PROXY_PROTOCOL_MAX_HEADER);
+    p = ngx_proxy_protocol_write(c, buf,
+                                 buf + NGX_PROXY_PROTOCOL_V1_MAX_HEADER);
     if (p == NULL) {
         ngx_stream_proxy_finalize(s, NGX_STREAM_INTERNAL_SERVER_ERROR);
         return NGX_ERROR;

From mdounin at mdounin.ru  Thu Nov  3 00:14:11 2022
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Thu, 3 Nov 2022 03:14:11 +0300
Subject: [PATCH] fix ngx_realpath() for win32
In-Reply-To: <CAJ3S5=crj7pFw6sCWLudWoGAUSRkPtV7yYcLgqH2w9WHXNnKow@mail.gmail.com>
References: <CAJ3S5=ciooYdcM7sMqURKU32B1XzYe_E8hgPfUbQxjU-9eYJkg@mail.gmail.com>
 <CAJ3S5=crj7pFw6sCWLudWoGAUSRkPtV7yYcLgqH2w9WHXNnKow@mail.gmail.com>
Message-ID: <Y2MH07ChE/RA4UO9@mdounin.ru>

Hello!

On Wed, Nov 02, 2022 at 03:43:50PM +0900, Won-Kyu Park wrote:

> > Hello!
> >
> > On Tue, Nov 01, 2022 at 08:15:48PM +0900, Won-Kyu Park wrote:
> >
> > ...
> >
> > Could you please clarify "as expected"?
> >
> > It doesn't look like _fullpath() resolves symbolic links, which is expected behaviour for $realpath_root.
> >
> 
> You are right but for most situations, translating a relative path to
> fullpath is enough, at least for me. (please see use case)

First of all, $document_root is already absolute, not a relative.  
It can contain some relative references, such as "/../", if you've 
wrote them in the configuration, but it's still absolute.

Resolving relative references is not what the variable is expected 
to do.  If you need an absolute path without any relative elements 
it is always possible to write one in the configuration.

What the $realpath_root variable is expected to do is symlink 
resolution, providing a way to switch to a different site version 
atomically by changing a symlink to a directory with the 
particular site version.  For example, this makes it possible to 
arbitrary change included files in php code between different site 
versions: since symlinks are resolved, all included files are 
loaded from the particular site version, and changing the symlink 
does not create a race condition.  Further, it is also resolves 
opcode cache issues in such deployment schemes, as opcode cache 
essentially makes the race window longer.  See, for example, 
https://serverfault.com/questions/848503/nginx-caching-symlinks 
and https://deployer.org/docs/7.x/avoid-php-fpm-reloading for some 
practical details.

[...]

> > Rather, it looks like GetFinalPathNameByHandle() should be 
> > used if we want to actually support $realpath_root as it is 
> > expected to work, that is, providing path with symlinks 
> > resolved. It might
> > be tricky to use it though, as it is only available in Windows 
> > Vista or later.
> 
> yes. GetFinalPathNameByHandle() is not available in pre-Vista.
> full implementation might be preferred but this simple oneliner solve
> most use cases I guess.

Well, the problem is that the $realpath_root variable with your 
patch applied does not become better regarding its main property, 
that is, symbolic links resolution.  Rather, the relevant code is 
no longer marked to be a stub and therefore becomes silently 
broken rather than not implemented.

Arguably, _fullpath() can be seen as a better stub than the one 
which is currently used.  But I don't think that $realpath_root 
actually guarantees resolution of relative references in the path, 
it is only documented to resolve symlinks.

> 
> - win32/mingw32 only support _fullpath()
> - PHP7 use GetFinalPathNameByHandleW()
> -------------
> 
> use case:
> 
> server {
>         location / {
>             root   ../to/html; # works nicely without problem
>             index  index.php index.html index.htm;
>         }
> 
>         location ~ \.php$ {
>             root           ../to/html; # not work. $realpath_root
> gives wrong value (c:\foobar\nginxroot\..\wiki), $document_root
> returns relative path
>             # for this case, $realpath_root gives expected value with
> this _fullpath fix.
>             #root c:/to/my/absolute/path/html; # works fine
>             fastcgi_index  index.php;
> 
>             # fix to use realpath_root
>             #fastcgi_param SCRIPT_FILENAME
> $document_root$fastcgi_script_name; # not work.
>             fastcgi_param SCRIPT_FILENAME
> $realpath_root$fastcgi_script_name; # works fine with _fullpath fix.
>             ...
>             fastcgi_pass   127.0.0.1:9000;
>         }
> }

The "c:\foobar\nginxroot\..\wiki" and "c:\foobar\wiki" paths are 
equivalent as long as nginxroot exists (and both are absolute).  
The problem with your configuration is that PHP rejects paths with 
".." for security reasons 
(https://github.com/php/php-src/blob/master/sapi/cgi/cgi_main.c#L1048).

While using $realpath_root on Unix systems resolves this, this is 
more like a side effect, and not a guaranteed behaviour.

A better solution might be to avoid using ".." in paths sent to 
PHP.  This might be done either by providing appropriate absolute 
paths in nginx configuration or by changing things to keep the 
php code under nginx prefix (by changing nginx prefix in effect, 
by actually moving directories, or by creating a symlink).

-- 
Maxim Dounin
http://mdounin.ru/


From mdounin at mdounin.ru  Thu Nov  3 01:51:55 2022
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Thu, 3 Nov 2022 04:51:55 +0300
Subject: [PATCH] Core: support for reading PROXY protocol v2 TLVs
In-Reply-To: <20221102130625.3tmxup3rythk6wkx@N00W24XTQX>
References: <Yx+k6bMb52CndRJy@mdounin.ru>
 <20220913150304.k2fjjdxgesgzbilu@N00W24XTQX>
 <YykpmhVRzb3PXbT3@mdounin.ru>
 <20220927094125.w7oo4g2quw3yyqfh@N00W24XTQX>
 <Y0TE9H4TlGJE1aTE@mdounin.ru>
 <20221011130111.oiljq55eydpp3dh6@N00W24XTQX>
 <Y0XX5Hu9dFNmyS67@mdounin.ru>
 <20221031120700.z5o4el64rhhpfn3y@N00W24XTQX>
 <Y2Epqv3Aw4X3/zPM@mdounin.ru>
 <20221102130625.3tmxup3rythk6wkx@N00W24XTQX>
Message-ID: <Y2Meuzmj+/fl6UWd@mdounin.ru>

Hello!

On Wed, Nov 02, 2022 at 05:06:25PM +0400, Roman Arutyunyan wrote:

[...]

> # HG changeset patch
> # User Roman Arutyunyan <arut at nginx.com>
> # Date 1667382376 -14400
> #      Wed Nov 02 13:46:16 2022 +0400
> # Node ID dc5f16e6a243c15f58e2c6a62f7a83f536729174
> # Parent  81b4326daac70d6de70abbc3fe36d4f6e3da54a2
> Increased maximum read PROXY protocol header size.
> 
> Maximum size for reading the PROXY protocol header is increased to 4096 to
> accommodate a bigger number of TLVs, which are supported since cca4c8a715de.
> 
> Maximum size for writing the PROXY protocol header is not changed since only
> version 1 is currently supported.
> 
> diff --git a/src/core/ngx_proxy_protocol.c b/src/core/ngx_proxy_protocol.c
> --- a/src/core/ngx_proxy_protocol.c
> +++ b/src/core/ngx_proxy_protocol.c
> @@ -281,7 +281,7 @@ ngx_proxy_protocol_write(ngx_connection_
>  {
>      ngx_uint_t  port, lport;
>  
> -    if (last - buf < NGX_PROXY_PROTOCOL_MAX_HEADER) {
> +    if (last - buf < NGX_PROXY_PROTOCOL_V1_MAX_HEADER) {
>          return NULL;
>      }

A side note: here an error is detected and returned, but no 
logging of the error happens neither in ngx_proxy_protocol_write() 
nor in its callers.  This needs to be fixed.

(Given that ngx_proxy_protocol_write() can also fail due to 
ngx_connection_local_sockaddr() failure, the logging should be 
added to ngx_proxy_protocol_write() itself.  Alternatively, the 
error detection can be completely removed, given that the error 
can never happen.)

>  
> diff --git a/src/core/ngx_proxy_protocol.h b/src/core/ngx_proxy_protocol.h
> --- a/src/core/ngx_proxy_protocol.h
> +++ b/src/core/ngx_proxy_protocol.h
> @@ -13,7 +13,8 @@
>  #include <ngx_core.h>
>  
>  
> -#define NGX_PROXY_PROTOCOL_MAX_HEADER  107
> +#define NGX_PROXY_PROTOCOL_V1_MAX_HEADER  107
> +#define NGX_PROXY_PROTOCOL_MAX_HEADER     4096
>  
>  
>  struct ngx_proxy_protocol_s {
> diff --git a/src/mail/ngx_mail_proxy_module.c b/src/mail/ngx_mail_proxy_module.c
> --- a/src/mail/ngx_mail_proxy_module.c
> +++ b/src/mail/ngx_mail_proxy_module.c
> @@ -890,7 +890,7 @@ ngx_mail_proxy_send_proxy_protocol(ngx_m
>      u_char            *p;
>      ssize_t            n, size;
>      ngx_connection_t  *c;
> -    u_char             buf[NGX_PROXY_PROTOCOL_MAX_HEADER];
> +    u_char             buf[NGX_PROXY_PROTOCOL_V1_MAX_HEADER];
>  
>      s->connection->log->action = "sending PROXY protocol header to upstream";
>  
> @@ -898,7 +898,7 @@ ngx_mail_proxy_send_proxy_protocol(ngx_m
>                     "mail proxy send PROXY protocol header");
>  
>      p = ngx_proxy_protocol_write(s->connection, buf,
> -                                 buf + NGX_PROXY_PROTOCOL_MAX_HEADER);
> +                                 buf + NGX_PROXY_PROTOCOL_V1_MAX_HEADER);
>      if (p == NULL) {
>          ngx_mail_proxy_internal_server_error(s);
>          return NGX_ERROR;
> diff --git a/src/stream/ngx_stream_proxy_module.c b/src/stream/ngx_stream_proxy_module.c
> --- a/src/stream/ngx_stream_proxy_module.c
> +++ b/src/stream/ngx_stream_proxy_module.c
> @@ -894,7 +894,7 @@ ngx_stream_proxy_init_upstream(ngx_strea
>              return;
>          }
>  
> -        p = ngx_pnalloc(c->pool, NGX_PROXY_PROTOCOL_MAX_HEADER);
> +        p = ngx_pnalloc(c->pool, NGX_PROXY_PROTOCOL_V1_MAX_HEADER);
>          if (p == NULL) {
>              ngx_stream_proxy_finalize(s, NGX_STREAM_INTERNAL_SERVER_ERROR);
>              return;
> @@ -902,7 +902,8 @@ ngx_stream_proxy_init_upstream(ngx_strea
>  
>          cl->buf->pos = p;
>  
> -        p = ngx_proxy_protocol_write(c, p, p + NGX_PROXY_PROTOCOL_MAX_HEADER);
> +        p = ngx_proxy_protocol_write(c, p,
> +                                     p + NGX_PROXY_PROTOCOL_V1_MAX_HEADER);
>          if (p == NULL) {
>              ngx_stream_proxy_finalize(s, NGX_STREAM_INTERNAL_SERVER_ERROR);
>              return;
> @@ -946,14 +947,15 @@ ngx_stream_proxy_send_proxy_protocol(ngx
>      ngx_connection_t             *c, *pc;
>      ngx_stream_upstream_t        *u;
>      ngx_stream_proxy_srv_conf_t  *pscf;
> -    u_char                        buf[NGX_PROXY_PROTOCOL_MAX_HEADER];
> +    u_char                        buf[NGX_PROXY_PROTOCOL_V1_MAX_HEADER];
>  
>      c = s->connection;
>  
>      ngx_log_debug0(NGX_LOG_DEBUG_STREAM, c->log, 0,
>                     "stream proxy send PROXY protocol header");
>  
> -    p = ngx_proxy_protocol_write(c, buf, buf + NGX_PROXY_PROTOCOL_MAX_HEADER);
> +    p = ngx_proxy_protocol_write(c, buf,
> +                                 buf + NGX_PROXY_PROTOCOL_V1_MAX_HEADER);
>      if (p == NULL) {
>          ngx_stream_proxy_finalize(s, NGX_STREAM_INTERNAL_SERVER_ERROR);
>          return NGX_ERROR;

Looks good.

-- 
Maxim Dounin
http://mdounin.ru/


From i at ciel.dev  Thu Nov  3 04:25:52 2022
From: i at ciel.dev (Ciel)
Date: Thu, 03 Nov 2022 04:25:52 +0000
Subject: [PATCH] SSI: ensure context of main request exists for subrequest using SSI
Message-ID: <-SpypjU_KG4ByLBWOaBV_3rln7SC_v-4sAF5Z9nLtOr3EL7bvilrVTfbPXmVCw1XetJSevc6xGYxdGuqLfTfvnAYR2G1ZKcyZqTurHo0vRc=@ciel.dev>

# HG changeset patch
# User Ciel Zhao <i at ciel.dev>
# Date 1667411069 -28800
#      Thu Nov 03 01:44:29 2022 +0800
# Node ID 85141e004b5af89a9d443bc0084a34291193567a
# Parent  1ae25660c0c76edef14121ca64362f28b9d57a70
SSI: ensure context of main request exists for subrequest using SSI

As the SSI parser always uses the context from the main request for storing
variables and blocks, that context should always exist for subrequests using
SSI, even though the main request does not necessarily have SSI enabled.

However, `ngx_http_get_module_ctx(r->main, ...)` is getting NULL in such cases,
resulting in the worker crashing SIGSEGV when accessing its attributes.

This patch checks the context of the main request after initializing the context
for subrequests, and creates one if not exists.

diff -r 1ae25660c0c7 -r 85141e004b5a src/http/modules/ngx_http_ssi_filter_module.c
--- a/src/http/modules/ngx_http_ssi_filter_module.c  Wed Oct 19 10:56:21 2022 +0300
+++ b/src/http/modules/ngx_http_ssi_filter_module.c  Thu Nov 03 01:44:29 2022 +0800
@@ -329,7 +329,7 @@
 static ngx_int_t
 ngx_http_ssi_header_filter(ngx_http_request_t *r)
 {
-    ngx_http_ssi_ctx_t       *ctx;
+    ngx_http_ssi_ctx_t       *ctx, *mctx;
     ngx_http_ssi_loc_conf_t  *slcf;
 
     slcf = ngx_http_get_module_loc_conf(r, ngx_http_ssi_filter_module);
@@ -348,6 +348,16 @@
 
     ngx_http_set_ctx(r, ctx, ngx_http_ssi_filter_module);
 
+    mctx = ngx_http_get_module_ctx(r->main, ngx_http_ssi_filter_module);
+    if (mctx == NULL && r != r->main) {
+        mctx = ngx_pcalloc(r->main->pool, sizeof(ngx_http_ssi_ctx_t));
+        if (mctx == NULL) {
+            return NGX_ERROR;
+        }
+
+        ngx_http_set_ctx(r->main, mctx, ngx_http_ssi_filter_module);
+    }
+
 
     ctx->value_len = slcf->value_len;
     ctx->last_out = &ctx->out;
@@ -403,8 +413,12 @@
     ngx_str_t                 *params[NGX_HTTP_SSI_MAX_PARAMS + 1];
 
     ctx = ngx_http_get_module_ctx(r, ngx_http_ssi_filter_module);
+    slcf = ngx_http_get_module_loc_conf(r, ngx_http_ssi_filter_module);
 
     if (ctx == NULL
+        || !slcf->enable
+        || r->headers_out.content_length_n == 0
+        || ngx_http_test_content_type(r, &slcf->types) == NULL
         || (in == NULL
             && ctx->buf == NULL
             && ctx->in == NULL
@@ -450,8 +464,6 @@
         }
     }
 
-    slcf = ngx_http_get_module_loc_conf(r, ngx_http_ssi_filter_module);
-
     while (ctx->in || ctx->buf) {
 
         if (ctx->buf == NULL) {



From arut at nginx.com  Thu Nov  3 12:08:37 2022
From: arut at nginx.com (Roman Arutyunyan)
Date: Thu, 03 Nov 2022 12:08:37 +0000
Subject: [nginx] Version bump.
Message-ID: <hg.a2924855f453.1667477317.6026610855610030274@dev.nginx>

details:   https://hg.nginx.org/nginx/rev/a2924855f453
branches:  
changeset: 8098:a2924855f453
user:      Roman Arutyunyan <arut at nginx.com>
date:      Thu Nov 03 15:52:55 2022 +0400
description:
Version bump.

diffstat:

 src/core/nginx.h |  4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diffs (14 lines):

diff -r 1ae25660c0c7 -r a2924855f453 src/core/nginx.h
--- a/src/core/nginx.h	Wed Oct 19 10:56:21 2022 +0300
+++ b/src/core/nginx.h	Thu Nov 03 15:52:55 2022 +0400
@@ -9,8 +9,8 @@
 #define _NGINX_H_INCLUDED_
 
 
-#define nginx_version      1023002
-#define NGINX_VERSION      "1.23.2"
+#define nginx_version      1023003
+#define NGINX_VERSION      "1.23.3"
 #define NGINX_VER          "nginx/" NGINX_VERSION
 
 #ifdef NGX_BUILD


From arut at nginx.com  Thu Nov  3 12:08:40 2022
From: arut at nginx.com (Roman Arutyunyan)
Date: Thu, 03 Nov 2022 12:08:40 +0000
Subject: [nginx] Increased maximum read PROXY protocol header size.
Message-ID: <hg.17d6a537fb1b.1667477320.6026610855610030274@dev.nginx>

details:   https://hg.nginx.org/nginx/rev/17d6a537fb1b
branches:  
changeset: 8099:17d6a537fb1b
user:      Roman Arutyunyan <arut at nginx.com>
date:      Wed Nov 02 13:46:16 2022 +0400
description:
Increased maximum read PROXY protocol header size.

Maximum size for reading the PROXY protocol header is increased to 4096 to
accommodate a bigger number of TLVs, which are supported since cca4c8a715de.

Maximum size for writing the PROXY protocol header is not changed since only
version 1 is currently supported.

diffstat:

 src/core/ngx_proxy_protocol.c        |   2 +-
 src/core/ngx_proxy_protocol.h        |   3 ++-
 src/mail/ngx_mail_proxy_module.c     |   4 ++--
 src/stream/ngx_stream_proxy_module.c |  10 ++++++----
 4 files changed, 11 insertions(+), 8 deletions(-)

diffs (86 lines):

diff -r a2924855f453 -r 17d6a537fb1b src/core/ngx_proxy_protocol.c
--- a/src/core/ngx_proxy_protocol.c	Thu Nov 03 15:52:55 2022 +0400
+++ b/src/core/ngx_proxy_protocol.c	Wed Nov 02 13:46:16 2022 +0400
@@ -281,7 +281,7 @@ ngx_proxy_protocol_write(ngx_connection_
 {
     ngx_uint_t  port, lport;
 
-    if (last - buf < NGX_PROXY_PROTOCOL_MAX_HEADER) {
+    if (last - buf < NGX_PROXY_PROTOCOL_V1_MAX_HEADER) {
         return NULL;
     }
 
diff -r a2924855f453 -r 17d6a537fb1b src/core/ngx_proxy_protocol.h
--- a/src/core/ngx_proxy_protocol.h	Thu Nov 03 15:52:55 2022 +0400
+++ b/src/core/ngx_proxy_protocol.h	Wed Nov 02 13:46:16 2022 +0400
@@ -13,7 +13,8 @@
 #include <ngx_core.h>
 
 
-#define NGX_PROXY_PROTOCOL_MAX_HEADER  107
+#define NGX_PROXY_PROTOCOL_V1_MAX_HEADER  107
+#define NGX_PROXY_PROTOCOL_MAX_HEADER     4096
 
 
 struct ngx_proxy_protocol_s {
diff -r a2924855f453 -r 17d6a537fb1b src/mail/ngx_mail_proxy_module.c
--- a/src/mail/ngx_mail_proxy_module.c	Thu Nov 03 15:52:55 2022 +0400
+++ b/src/mail/ngx_mail_proxy_module.c	Wed Nov 02 13:46:16 2022 +0400
@@ -890,7 +890,7 @@ ngx_mail_proxy_send_proxy_protocol(ngx_m
     u_char            *p;
     ssize_t            n, size;
     ngx_connection_t  *c;
-    u_char             buf[NGX_PROXY_PROTOCOL_MAX_HEADER];
+    u_char             buf[NGX_PROXY_PROTOCOL_V1_MAX_HEADER];
 
     s->connection->log->action = "sending PROXY protocol header to upstream";
 
@@ -898,7 +898,7 @@ ngx_mail_proxy_send_proxy_protocol(ngx_m
                    "mail proxy send PROXY protocol header");
 
     p = ngx_proxy_protocol_write(s->connection, buf,
-                                 buf + NGX_PROXY_PROTOCOL_MAX_HEADER);
+                                 buf + NGX_PROXY_PROTOCOL_V1_MAX_HEADER);
     if (p == NULL) {
         ngx_mail_proxy_internal_server_error(s);
         return NGX_ERROR;
diff -r a2924855f453 -r 17d6a537fb1b src/stream/ngx_stream_proxy_module.c
--- a/src/stream/ngx_stream_proxy_module.c	Thu Nov 03 15:52:55 2022 +0400
+++ b/src/stream/ngx_stream_proxy_module.c	Wed Nov 02 13:46:16 2022 +0400
@@ -894,7 +894,7 @@ ngx_stream_proxy_init_upstream(ngx_strea
             return;
         }
 
-        p = ngx_pnalloc(c->pool, NGX_PROXY_PROTOCOL_MAX_HEADER);
+        p = ngx_pnalloc(c->pool, NGX_PROXY_PROTOCOL_V1_MAX_HEADER);
         if (p == NULL) {
             ngx_stream_proxy_finalize(s, NGX_STREAM_INTERNAL_SERVER_ERROR);
             return;
@@ -902,7 +902,8 @@ ngx_stream_proxy_init_upstream(ngx_strea
 
         cl->buf->pos = p;
 
-        p = ngx_proxy_protocol_write(c, p, p + NGX_PROXY_PROTOCOL_MAX_HEADER);
+        p = ngx_proxy_protocol_write(c, p,
+                                     p + NGX_PROXY_PROTOCOL_V1_MAX_HEADER);
         if (p == NULL) {
             ngx_stream_proxy_finalize(s, NGX_STREAM_INTERNAL_SERVER_ERROR);
             return;
@@ -946,14 +947,15 @@ ngx_stream_proxy_send_proxy_protocol(ngx
     ngx_connection_t             *c, *pc;
     ngx_stream_upstream_t        *u;
     ngx_stream_proxy_srv_conf_t  *pscf;
-    u_char                        buf[NGX_PROXY_PROTOCOL_MAX_HEADER];
+    u_char                        buf[NGX_PROXY_PROTOCOL_V1_MAX_HEADER];
 
     c = s->connection;
 
     ngx_log_debug0(NGX_LOG_DEBUG_STREAM, c->log, 0,
                    "stream proxy send PROXY protocol header");
 
-    p = ngx_proxy_protocol_write(c, buf, buf + NGX_PROXY_PROTOCOL_MAX_HEADER);
+    p = ngx_proxy_protocol_write(c, buf,
+                                 buf + NGX_PROXY_PROTOCOL_V1_MAX_HEADER);
     if (p == NULL) {
         ngx_stream_proxy_finalize(s, NGX_STREAM_INTERNAL_SERVER_ERROR);
         return NGX_ERROR;


From mdounin at mdounin.ru  Thu Nov  3 17:06:44 2022
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Thu, 3 Nov 2022 20:06:44 +0300
Subject: [PATCH] SSI: ensure context of main request exists for
 subrequest using SSI
In-Reply-To: <-SpypjU_KG4ByLBWOaBV_3rln7SC_v-4sAF5Z9nLtOr3EL7bvilrVTfbPXmVCw1XetJSevc6xGYxdGuqLfTfvnAYR2G1ZKcyZqTurHo0vRc=@ciel.dev>
References: <-SpypjU_KG4ByLBWOaBV_3rln7SC_v-4sAF5Z9nLtOr3EL7bvilrVTfbPXmVCw1XetJSevc6xGYxdGuqLfTfvnAYR2G1ZKcyZqTurHo0vRc=@ciel.dev>
Message-ID: <Y2P1JK9cD+nMyRfh@mdounin.ru>

Hello!

On Thu, Nov 03, 2022 at 04:25:52AM +0000, Ciel via nginx-devel wrote:

> # HG changeset patch
> # User Ciel Zhao <i at ciel.dev>
> # Date 1667411069 -28800
> #      Thu Nov 03 01:44:29 2022 +0800
> # Node ID 85141e004b5af89a9d443bc0084a34291193567a
> # Parent  1ae25660c0c76edef14121ca64362f28b9d57a70
> SSI: ensure context of main request exists for subrequest using SSI
> 
> As the SSI parser always uses the context from the main request for storing
> variables and blocks, that context should always exist for subrequests using
> SSI, even though the main request does not necessarily have SSI enabled.
> 
> However, `ngx_http_get_module_ctx(r->main, ...)` is getting NULL in such cases,
> resulting in the worker crashing SIGSEGV when accessing its attributes.
> 
> This patch checks the context of the main request after initializing the context
> for subrequests, and creates one if not exists.

Thanks for the patch.

It looks like an attempt to fix ticket #1263 
(https://trac.nginx.org/nginx/ticket/1263).  I've linked this 
thread to the ticket.  It might be a good idea to add a reference 
into commit log.

See below for some comments about the code.

> 
> diff -r 1ae25660c0c7 -r 85141e004b5a src/http/modules/ngx_http_ssi_filter_module.c
> --- a/src/http/modules/ngx_http_ssi_filter_module.c  Wed Oct 19 10:56:21 2022 +0300
> +++ b/src/http/modules/ngx_http_ssi_filter_module.c  Thu Nov 03 01:44:29 2022 +0800
> @@ -329,7 +329,7 @@
>  static ngx_int_t
>  ngx_http_ssi_header_filter(ngx_http_request_t *r)
>  {
> -    ngx_http_ssi_ctx_t       *ctx;
> +    ngx_http_ssi_ctx_t       *ctx, *mctx;
>      ngx_http_ssi_loc_conf_t  *slcf;
>  
>      slcf = ngx_http_get_module_loc_conf(r, ngx_http_ssi_filter_module);
> @@ -348,6 +348,16 @@
>  
>      ngx_http_set_ctx(r, ctx, ngx_http_ssi_filter_module);
>  
> +    mctx = ngx_http_get_module_ctx(r->main, ngx_http_ssi_filter_module);
> +    if (mctx == NULL && r != r->main) {
> +        mctx = ngx_pcalloc(r->main->pool, sizeof(ngx_http_ssi_ctx_t));
> +        if (mctx == NULL) {
> +            return NGX_ERROR;
> +        }
> +
> +        ngx_http_set_ctx(r->main, mctx, ngx_http_ssi_filter_module);
> +    }
> +
>  
>      ctx->value_len = slcf->value_len;
>      ctx->last_out = &ctx->out;

In many cases the main request context is not needed.  It might be 
a good idea to create it only when needed, and avoid doing so if 
it isn't.

Further, in theory, SSI processing of a (in-memory/background) 
subrequest might happen even before main request is actually seen 
by the SSI module, so the actual context will be created later.  
This probably needs to be taken into account.

> @@ -403,8 +413,12 @@
>      ngx_str_t                 *params[NGX_HTTP_SSI_MAX_PARAMS + 1];
>  
>      ctx = ngx_http_get_module_ctx(r, ngx_http_ssi_filter_module);
> +    slcf = ngx_http_get_module_loc_conf(r, ngx_http_ssi_filter_module);
>  
>      if (ctx == NULL
> +        || !slcf->enable
> +        || r->headers_out.content_length_n == 0
> +        || ngx_http_test_content_type(r, &slcf->types) == NULL
>          || (in == NULL
>              && ctx->buf == NULL
>              && ctx->in == NULL

This does not look like a good solution to filter out processing 
of a main request if SSI is not enabled for it.  Rather, there 
should be an explicit flag, so it would be possible to avoid 
evaluation of complex constructs on each body filter call - there 
can be a lot of such calls.

> @@ -450,8 +464,6 @@
>          }
>      }
>  
> -    slcf = ngx_http_get_module_loc_conf(r, ngx_http_ssi_filter_module);
> -
>      while (ctx->in || ctx->buf) {
>  
>          if (ctx->buf == NULL) {
> 

-- 
Maxim Dounin
http://mdounin.ru/


From i at ciel.dev  Fri Nov  4 09:00:33 2022
From: i at ciel.dev (Ciel)
Date: Fri, 04 Nov 2022 09:00:33 +0000
Subject: [PATCH] SSI: ensure context of main request exists for subrequest using SSI
In-Reply-To: <Y2P1JK9cD+nMyRfh@mdounin.ru>
References: <-SpypjU_KG4ByLBWOaBV_3rln7SC_v-4sAF5Z9nLtOr3EL7bvilrVTfbPXmVCw1XetJSevc6xGYxdGuqLfTfvnAYR2G1ZKcyZqTurHo0vRc=@ciel.dev> <Y2P1JK9cD+nMyRfh@mdounin.ru>
Message-ID: <-gUKUM8ZX15I_C0Ar8PG2FqZyzZg4agx-52A6kAx8_zLiAc-eXbg5903hwRPZxT_6wrpIvVyxTUqNlx9-2JmiTtjVlCoQwp2KM6igGnoddo=@ciel.dev>

Hi Maxim,

Thanks for the quick reply.

> It looks like an attempt to fix ticket #1263
> (https://trac.nginx.org/nginx/ticket/1263). I've linked this
> thread to the ticket. It might be a good idea to add a reference
> into commit log.

I encountered this problem building my own website, and spent some hours to
figure out the problem and patch. I didn't guess this bug have such history.
Reference will be in the updated patch.

> In many cases the main request context is not needed. It might be
> a good idea to create it only when needed, and avoid doing so if
> it isn't.
> 
> Further, in theory, SSI processing of a (in-memory/background)
> subrequest might happen even before main request is actually seen
> by the SSI module, so the actual context will be created later.
> This probably needs to be taken into account.

As we only use the main context for variables and stub blocks, it seems appreciable
to have the context created on demand. However, we can't say if the SSI templates
use these commands until we processed them. So I think this left us merely two 
options:

a. create main context for every requests having SSI enabled (previous patch)
b. check main context existence everywhere we use it

>From my purview, most SSI templates have some variables involved, so just go ahead
and create context can save many checks, while not introducing much waste of memory.

> This does not look like a good solution to filter out processing
> of a main request if SSI is not enabled for it. Rather, there
> should be an explicit flag, so it would be possible to avoid
> evaluation of complex constructs on each body filter call - there
> can be a lot of such calls.

Totally agree, and this can be checked by `ngx_http_ssi_header_filter` in main request,
to initialize the dummy context if needed. But I wonder shall I add a bit field into 
`ngx_http_ssi_ctx_t` (as there're already some so no memory cost) or add an
`ngx_flag_t` (8B on amd64) instead.

In the issue tracker 6 years ago, you mentioned,

> It was written when there were no subrequests except subrequests created by the SSI
> module itself, and assumes in many places that its context always exists in the main
> request.

Though I'm not 100% familiar with the process of subrequests, this arise some more
questions in my mind: If subrequests are processed by the SSI filter (before postpone)
asynchronously and concurrently, could this introduce some out-of-order execution of
SSI commands among subrequests, then lead to non-determined result? If this really
happens, should we move the SSI filter past the postpone filter? Looking forward to
some enlightenment.

I'll post my updated patch after these discussions settle. And thanks again for attention.

Ciel Zhao


From sb at nginx.com  Fri Nov  4 12:30:35 2022
From: sb at nginx.com (Sergey Budnevich)
Date: Fri, 4 Nov 2022 12:30:35 +0000
Subject: [PATCH] Linux packages: added Ubuntu 22.10 "kinetic"
In-Reply-To: <ba6c27b903c7cd1b7277.1666697209@QGCD7XG9R9>
References: <ba6c27b903c7cd1b7277.1666697209@QGCD7XG9R9>
Message-ID: <57419E93-C6B3-4222-8828-6DD9C96E5268@nginx.com>

Looks good to me.

> On 25 Oct 2022, at 12:26, Konstantin Pavlov <thresh at nginx.com> wrote:
> 
> # HG changeset patch
> # User Konstantin Pavlov <thresh at nginx.com>
> # Date 1666697160 -14400
> #      Tue Oct 25 15:26:00 2022 +0400
> # Node ID ba6c27b903c7cd1b7277e6fcebf2308e863e6c64
> # Parent  e4a87f3a05d851f874bcbe8750280929eb5f9894
> Linux packages: added Ubuntu 22.10 "kinetic".
> 
> diff -r e4a87f3a05d8 -r ba6c27b903c7 xml/en/linux_packages.xml
> --- a/xml/en/linux_packages.xml	Fri Oct 21 16:33:37 2022 -0700
> +++ b/xml/en/linux_packages.xml	Tue Oct 25 15:26:00 2022 +0400
> @@ -7,7 +7,7 @@
> <article name="nginx: Linux packages"
>         link="/en/linux_packages.html"
>         lang="en"
> -         rev="80">
> +         rev="81">
> 
> <section name="Supported distributions and versions" id="distributions">
> 
> @@ -87,6 +87,11 @@ versions:
> <td>x86_64, aarch64/arm64, s390x</td>
> </tr>
> 
> +<tr>
> +<td width="30%">22.10 “kinetic”</td>
> +<td>x86_64, aarch64/arm64</td>
> +</tr>
> +
> </table>
> </para>
> 
> diff -r e4a87f3a05d8 -r ba6c27b903c7 xml/ru/linux_packages.xml
> --- a/xml/ru/linux_packages.xml	Fri Oct 21 16:33:37 2022 -0700
> +++ b/xml/ru/linux_packages.xml	Tue Oct 25 15:26:00 2022 +0400
> @@ -7,7 +7,7 @@
> <article name="nginx: пакеты для Linux"
>         link="/ru/linux_packages.html"
>         lang="ru"
> -         rev="80">
> +         rev="81">
> 
> <section name="Поддерживаемые дистрибутивы и версии" id="distributions">
> 
> @@ -87,6 +87,11 @@
> <td>x86_64, aarch64/arm64, s390x</td>
> </tr>
> 
> +<tr>
> +<td width="30%">22.10 “kinetic”</td>
> +<td>x86_64, aarch64/arm64</td>
> +</tr>
> +
> </table>
> </para>
> 
> _______________________________________________
> nginx-devel mailing list -- nginx-devel at nginx.org
> To unsubscribe send an email to nginx-devel-leave at nginx.org


From alx.manpages at gmail.com  Fri Nov  4 15:24:14 2022
From: alx.manpages at gmail.com (Alejandro Colomar)
Date: Fri,  4 Nov 2022 16:24:14 +0100
Subject: [PATCH] Removed the unsafe ngx_memcmp() wrapper for memcmp(3)
Message-ID: <20221104152414.14208-1-alx@kernel.org>

From: Alejandro Colomar <alx at nginx.com>

The casts are unnecessary, since memcmp(3)'s arguments are
'const void *', which allows implicit conversion from any pointer type.
It might have been necessary in the times of K&R C, where 'void *'
didn't exist yet, up until the early 2000s, because some old systems
still had limited or no support for ISO C89.  Those systems passed away
a long time ago, and current systems, even the oldest living ones, have
support for ISO C89.

The changes, apart from the removal of the macro itself, were scripted:

$ find src/ -type f \
  | grep '\.[ch]$' \
  | xargs sed -i 's/ngx_memcmp/memcmp/';

The cast in this case is (almost) innocuous, because it only hides
warnings for conversions from integer to pointer such as:

    nxt_memcmp(n, "foo", 3);  // no warnings
    memcmp(n, "foo", 3);      // warning: integer to pointer conversion

which is a difficult bug to write, since it's too obvious.  Such code
will probably be caught in a code review, but there's always a small
risk.  Since there's no reason to keep the small risk around, when we
can just avoid it by removing the cast.

In general, it's better to avoid a cast if possible, since casts will
disable many compiler warnings regarding type safety.

Cc: Andrei Zeliankou <a.zelenkov at f5.com>
Cc: Andrew Clayton <a.clayton at nginx.com>
Cc: Artem Konev <a.konev at f5.com>
Cc: Liam Crilly <liam at nginx.com>
Cc: Timo Stark <t.stark at nginx.com>
Cc: Zhidao Hong <z.hong at f5.com>
Signed-off-by: Alejandro Colomar <alx at nginx.com>
---
 src/core/ngx_inet.c                           |  4 ++--
 src/core/ngx_resolver.c                       |  4 ++--
 src/core/ngx_string.c                         |  6 ++---
 src/core/ngx_string.h                         |  4 ----
 src/event/ngx_event_openssl.c                 |  2 +-
 src/http/modules/ngx_http_geo_module.c        |  2 +-
 .../modules/ngx_http_secure_link_module.c     |  2 +-
 src/http/ngx_http_core_module.c               |  4 ++--
 src/http/ngx_http_file_cache.c                | 10 ++++-----
 src/http/ngx_http_request.c                   |  2 +-
 src/http/v2/ngx_http_v2.c                     | 22 ++++++-------------
 src/mail/ngx_mail_handler.c                   |  2 +-
 src/stream/ngx_stream_geo_module.c            |  2 +-
 src/stream/ngx_stream_handler.c               |  2 +-
 14 files changed, 28 insertions(+), 40 deletions(-)

diff --git a/src/core/ngx_inet.c b/src/core/ngx_inet.c
index 4228504a..e8ffc288 100644
--- a/src/core/ngx_inet.c
+++ b/src/core/ngx_inet.c
@@ -1349,7 +1349,7 @@ ngx_cmp_sockaddr(struct sockaddr *sa1, socklen_t slen1,
             return NGX_DECLINED;
         }
 
-        if (ngx_memcmp(&sin61->sin6_addr, &sin62->sin6_addr, 16) != 0) {
+        if (memcmp(&sin61->sin6_addr, &sin62->sin6_addr, 16) != 0) {
             return NGX_DECLINED;
         }
 
@@ -1373,7 +1373,7 @@ ngx_cmp_sockaddr(struct sockaddr *sa1, socklen_t slen1,
             len = sizeof(saun1->sun_path);
         }
 
-        if (ngx_memcmp(&saun1->sun_path, &saun2->sun_path, len) != 0) {
+        if (memcmp(&saun1->sun_path, &saun2->sun_path, len) != 0) {
             return NGX_DECLINED;
         }
 
diff --git a/src/core/ngx_resolver.c b/src/core/ngx_resolver.c
index c76c1785..64a42a49 100644
--- a/src/core/ngx_resolver.c
+++ b/src/core/ngx_resolver.c
@@ -3557,7 +3557,7 @@ ngx_resolver_lookup_addr6(ngx_resolver_t *r, struct in6_addr *addr,
 
         rn = ngx_resolver_node(node);
 
-        rc = ngx_memcmp(addr, &rn->addr6, 16);
+        rc = memcmp(addr, &rn->addr6, 16);
 
         if (rc == 0) {
             return rn;
@@ -3639,7 +3639,7 @@ ngx_resolver_rbtree_insert_addr6_value(ngx_rbtree_node_t *temp,
             rn = ngx_resolver_node(node);
             rn_temp = ngx_resolver_node(temp);
 
-            p = (ngx_memcmp(&rn->addr6, &rn_temp->addr6, 16)
+            p = (memcmp(&rn->addr6, &rn_temp->addr6, 16)
                  < 0) ? &temp->left : &temp->right;
         }
 
diff --git a/src/core/ngx_string.c b/src/core/ngx_string.c
index 98f270ac..93d5a5c6 100644
--- a/src/core/ngx_string.c
+++ b/src/core/ngx_string.c
@@ -882,7 +882,7 @@ ngx_memn2cmp(u_char *s1, u_char *s2, size_t n1, size_t n2)
         z = 1;
     }
 
-    m = ngx_memcmp(s1, s2, n);
+    m = memcmp(s1, s2, n);
 
     if (m || n1 == n2) {
         return m;
@@ -1980,7 +1980,7 @@ ngx_str_rbtree_insert_value(ngx_rbtree_node_t *temp,
             p = (n->str.len < t->str.len) ? &temp->left : &temp->right;
 
         } else {
-            p = (ngx_memcmp(n->str.data, t->str.data, n->str.len) < 0)
+            p = (memcmp(n->str.data, t->str.data, n->str.len) < 0)
                  ? &temp->left : &temp->right;
         }
 
@@ -2023,7 +2023,7 @@ ngx_str_rbtree_lookup(ngx_rbtree_t *rbtree, ngx_str_t *val, uint32_t hash)
             continue;
         }
 
-        rc = ngx_memcmp(val->data, n->str.data, val->len);
+        rc = memcmp(val->data, n->str.data, val->len);
 
         if (rc < 0) {
             node = node->left;
diff --git a/src/core/ngx_string.h b/src/core/ngx_string.h
index 0fb9be72..cfed4ab9 100644
--- a/src/core/ngx_string.h
+++ b/src/core/ngx_string.h
@@ -144,10 +144,6 @@ ngx_copy(u_char *dst, u_char *src, size_t len)
 #define ngx_movemem(dst, src, n)   (((u_char *) memmove(dst, src, n)) + (n))
 
 
-/* msvc and icc7 compile memcmp() to the inline loop */
-#define ngx_memcmp(s1, s2, n)  memcmp((const char *) s1, (const char *) s2, n)
-
-
 u_char *ngx_cpystrn(u_char *dst, u_char *src, size_t n);
 u_char *ngx_pstrdup(ngx_pool_t *pool, ngx_str_t *src);
 u_char * ngx_cdecl ngx_sprintf(u_char *buf, const char *fmt, ...);
diff --git a/src/event/ngx_event_openssl.c b/src/event/ngx_event_openssl.c
index 3be64b6c..d92dae48 100644
--- a/src/event/ngx_event_openssl.c
+++ b/src/event/ngx_event_openssl.c
@@ -4516,7 +4516,7 @@ ngx_ssl_ticket_key_callback(ngx_ssl_conn_t *ssl_conn,
         /* decrypt session ticket */
 
         for (i = 0; i < keys->nelts; i++) {
-            if (ngx_memcmp(name, key[i].name, 16) == 0) {
+            if (memcmp(name, key[i].name, 16) == 0) {
                 goto found;
             }
         }
diff --git a/src/http/modules/ngx_http_geo_module.c b/src/http/modules/ngx_http_geo_module.c
index ef4e9b84..1f243132 100644
--- a/src/http/modules/ngx_http_geo_module.c
+++ b/src/http/modules/ngx_http_geo_module.c
@@ -1497,7 +1497,7 @@ ngx_http_geo_include_binary_base(ngx_conf_t *cf, ngx_http_geo_conf_ctx_t *ctx,
 
     header = (ngx_http_geo_header_t *) base;
 
-    if (size < 16 || ngx_memcmp(&ngx_http_geo_header, header, 12) != 0) {
+    if (size < 16 || memcmp(&ngx_http_geo_header, header, 12) != 0) {
         ngx_conf_log_error(NGX_LOG_WARN, cf, 0,
              "incompatible binary geo range base \"%s\"", name->data);
         goto failed;
diff --git a/src/http/modules/ngx_http_secure_link_module.c b/src/http/modules/ngx_http_secure_link_module.c
index 4d4ce6af..23f3b77c 100644
--- a/src/http/modules/ngx_http_secure_link_module.c
+++ b/src/http/modules/ngx_http_secure_link_module.c
@@ -175,7 +175,7 @@ ngx_http_secure_link_variable(ngx_http_request_t *r,
     ngx_md5_update(&md5, val.data, val.len);
     ngx_md5_final(md5_buf, &md5);
 
-    if (ngx_memcmp(hash_buf, md5_buf, 16) != 0) {
+    if (memcmp(hash_buf, md5_buf, 16) != 0) {
         goto not_found;
     }
 
diff --git a/src/http/ngx_http_core_module.c b/src/http/ngx_http_core_module.c
index 28f7d99b..1a1bc75b 100644
--- a/src/http/ngx_http_core_module.c
+++ b/src/http/ngx_http_core_module.c
@@ -2055,7 +2055,7 @@ ngx_http_gzip_ok(ngx_http_request_t *r)
      *   Opera:   "gzip, deflate"
      */
 
-    if (ngx_memcmp(ae->value.data, "gzip,", 5) != 0
+    if (memcmp(ae->value.data, "gzip,", 5) != 0
         && ngx_http_gzip_accept_encoding(&ae->value) != NGX_OK)
     {
         return NGX_DECLINED;
@@ -2685,7 +2685,7 @@ ngx_http_set_disable_symlinks(ngx_http_request_t *r,
 
     if (from.len == 0
         || from.len > path->len
-        || ngx_memcmp(path->data, from.data, from.len) != 0)
+        || memcmp(path->data, from.data, from.len) != 0)
     {
         return NGX_OK;
     }
diff --git a/src/http/ngx_http_file_cache.c b/src/http/ngx_http_file_cache.c
index 4d2f6c42..0503d9aa 100644
--- a/src/http/ngx_http_file_cache.c
+++ b/src/http/ngx_http_file_cache.c
@@ -567,7 +567,7 @@ ngx_http_file_cache_read(ngx_http_request_t *r, ngx_http_cache_t *c)
 
     key = c->keys.elts;
     for (i = 0; i < c->keys.nelts; i++) {
-        if (ngx_memcmp(p, key[i].data, key[i].len) != 0) {
+        if (memcmp(p, key[i].data, key[i].len) != 0) {
             ngx_log_error(NGX_LOG_CRIT, r->connection->log, 0,
                           "cache file \"%s\" has md5 collision",
                           c->file.name.data);
@@ -594,7 +594,7 @@ ngx_http_file_cache_read(ngx_http_request_t *r, ngx_http_cache_t *c)
     if (h->vary_len) {
         ngx_http_file_cache_vary(r, h->vary, h->vary_len, c->variant);
 
-        if (ngx_memcmp(c->variant, h->variant, NGX_HTTP_CACHE_KEY_LEN) != 0) {
+        if (memcmp(c->variant, h->variant, NGX_HTTP_CACHE_KEY_LEN) != 0) {
             ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                            "http file cache vary mismatch");
             return ngx_http_file_cache_reopen(r, c);
@@ -995,7 +995,7 @@ ngx_http_file_cache_lookup(ngx_http_file_cache_t *cache, u_char *key)
 
         fcn = (ngx_http_file_cache_node_t *) node;
 
-        rc = ngx_memcmp(&key[sizeof(ngx_rbtree_key_t)], fcn->key,
+        rc = memcmp(&key[sizeof(ngx_rbtree_key_t)], fcn->key,
                         NGX_HTTP_CACHE_KEY_LEN - sizeof(ngx_rbtree_key_t));
 
         if (rc == 0) {
@@ -1033,7 +1033,7 @@ ngx_http_file_cache_rbtree_insert_value(ngx_rbtree_node_t *temp,
             cn = (ngx_http_file_cache_node_t *) node;
             cnt = (ngx_http_file_cache_node_t *) temp;
 
-            p = (ngx_memcmp(cn->key, cnt->key,
+            p = (memcmp(cn->key, cnt->key,
                             NGX_HTTP_CACHE_KEY_LEN - sizeof(ngx_rbtree_key_t))
                  < 0)
                     ? &temp->left : &temp->right;
@@ -1315,7 +1315,7 @@ ngx_http_file_cache_update_variant(ngx_http_request_t *r, ngx_http_cache_t *c)
     }
 
     if (c->vary.len
-        && ngx_memcmp(c->variant, c->key, NGX_HTTP_CACHE_KEY_LEN) == 0)
+        && memcmp(c->variant, c->key, NGX_HTTP_CACHE_KEY_LEN) == 0)
     {
         return NGX_OK;
     }
diff --git a/src/http/ngx_http_request.c b/src/http/ngx_http_request.c
index 131a2c83..dce37a26 100644
--- a/src/http/ngx_http_request.c
+++ b/src/http/ngx_http_request.c
@@ -253,7 +253,7 @@ ngx_http_init_connection(ngx_connection_t *c)
             /* the last address is "*" */
 
             for (i = 0; i < port->naddrs - 1; i++) {
-                if (ngx_memcmp(&addr6[i].addr6, &sin6->sin6_addr, 16) == 0) {
+                if (memcmp(&addr6[i].addr6, &sin6->sin6_addr, 16) == 0) {
                     break;
                 }
             }
diff --git a/src/http/v2/ngx_http_v2.c b/src/http/v2/ngx_http_v2.c
index 0e45a7b2..2d54b5ec 100644
--- a/src/http/v2/ngx_http_v2.c
+++ b/src/http/v2/ngx_http_v2.c
@@ -877,7 +877,7 @@ ngx_http_v2_state_preface(ngx_http_v2_connection_t *h2c, u_char *pos,
         return ngx_http_v2_state_save(h2c, pos, end, ngx_http_v2_state_preface);
     }
 
-    if (ngx_memcmp(pos, preface, sizeof(preface) - 1) != 0) {
+    if (memcmp(pos, preface, sizeof(preface) - 1) != 0) {
         ngx_log_error(NGX_LOG_INFO, h2c->connection->log, 0,
                       "invalid connection preface");
 
@@ -899,7 +899,7 @@ ngx_http_v2_state_preface_end(ngx_http_v2_connection_t *h2c, u_char *pos,
                                       ngx_http_v2_state_preface_end);
     }
 
-    if (ngx_memcmp(pos, preface, sizeof(preface) - 1) != 0) {
+    if (memcmp(pos, preface, sizeof(preface) - 1) != 0) {
         ngx_log_error(NGX_LOG_INFO, h2c->connection->log, 0,
                       "invalid connection preface");
 
@@ -1838,7 +1838,7 @@ ngx_http_v2_state_process_header(ngx_http_v2_connection_t *h2c, u_char *pos,
     }
 
     if (header->name.len == cookie.len
-        && ngx_memcmp(header->name.data, cookie.data, cookie.len) == 0)
+        && memcmp(header->name.data, cookie.data, cookie.len) == 0)
     {
         if (ngx_http_v2_cookie(r, header) != NGX_OK) {
             return ngx_http_v2_connection_error(h2c,
@@ -3496,33 +3496,25 @@ ngx_http_v2_pseudo_header(ngx_http_request_t *r, ngx_http_v2_header_t *header)
 
     switch (header->name.len) {
     case 4:
-        if (ngx_memcmp(header->name.data, "path", sizeof("path") - 1)
-            == 0)
-        {
+        if (memcmp(header->name.data, "path", sizeof("path") - 1) == 0) {
             return ngx_http_v2_parse_path(r, &header->value);
         }
 
         break;
 
     case 6:
-        if (ngx_memcmp(header->name.data, "method", sizeof("method") - 1)
-            == 0)
-        {
+        if (memcmp(header->name.data, "method", sizeof("method") - 1) == 0) {
             return ngx_http_v2_parse_method(r, &header->value);
         }
 
-        if (ngx_memcmp(header->name.data, "scheme", sizeof("scheme") - 1)
-            == 0)
-        {
+        if (memcmp(header->name.data, "scheme", sizeof("scheme") - 1) == 0) {
             return ngx_http_v2_parse_scheme(r, &header->value);
         }
 
         break;
 
     case 9:
-        if (ngx_memcmp(header->name.data, "authority", sizeof("authority") - 1)
-            == 0)
-        {
+        if (memcmp(header->name.data, "authority", sizeof("authority") - 1) == 0) {
             return ngx_http_v2_parse_authority(r, &header->value);
         }
 
diff --git a/src/mail/ngx_mail_handler.c b/src/mail/ngx_mail_handler.c
index 246ba97c..0d408025 100644
--- a/src/mail/ngx_mail_handler.c
+++ b/src/mail/ngx_mail_handler.c
@@ -76,7 +76,7 @@ ngx_mail_init_connection(ngx_connection_t *c)
             /* the last address is "*" */
 
             for (i = 0; i < port->naddrs - 1; i++) {
-                if (ngx_memcmp(&addr6[i].addr6, &sin6->sin6_addr, 16) == 0) {
+                if (memcmp(&addr6[i].addr6, &sin6->sin6_addr, 16) == 0) {
                     break;
                 }
             }
diff --git a/src/stream/ngx_stream_geo_module.c b/src/stream/ngx_stream_geo_module.c
index 4b4cad8f..677f0086 100644
--- a/src/stream/ngx_stream_geo_module.c
+++ b/src/stream/ngx_stream_geo_module.c
@@ -1423,7 +1423,7 @@ ngx_stream_geo_include_binary_base(ngx_conf_t *cf,
 
     header = (ngx_stream_geo_header_t *) base;
 
-    if (size < 16 || ngx_memcmp(&ngx_stream_geo_header, header, 12) != 0) {
+    if (size < 16 || memcmp(&ngx_stream_geo_header, header, 12) != 0) {
         ngx_conf_log_error(NGX_LOG_WARN, cf, 0,
              "incompatible binary geo range base \"%s\"", name->data);
         goto failed;
diff --git a/src/stream/ngx_stream_handler.c b/src/stream/ngx_stream_handler.c
index 669b6a18..7b6f5f38 100644
--- a/src/stream/ngx_stream_handler.c
+++ b/src/stream/ngx_stream_handler.c
@@ -70,7 +70,7 @@ ngx_stream_init_connection(ngx_connection_t *c)
             /* the last address is "*" */
 
             for (i = 0; i < port->naddrs - 1; i++) {
-                if (ngx_memcmp(&addr6[i].addr6, &sin6->sin6_addr, 16) == 0) {
+                if (memcmp(&addr6[i].addr6, &sin6->sin6_addr, 16) == 0) {
                     break;
                 }
             }
-- 
2.37.2


From alx.manpages at gmail.com  Fri Nov  4 15:28:25 2022
From: alx.manpages at gmail.com (Alejandro Colomar)
Date: Fri, 4 Nov 2022 16:28:25 +0100
Subject: [PATCH] Removed the unsafe ngx_memcmp() wrapper for memcmp(3)
In-Reply-To: <20221104152414.14208-1-alx@kernel.org>
References: <20221104152414.14208-1-alx@kernel.org>
Message-ID: <fa538537-3859-c520-b07f-f857167f9b22@gmail.com>

On 11/4/22 16:24, Alejandro Colomar wrote:
> The changes, apart from the removal of the macro itself, were scripted:
> 
> $ find src/ -type f \
>    | grep '\.[ch]$' \
>    | xargs sed -i 's/ngx_memcmp/memcmp/';

And after running the script, I compressed manually some lines that could be 
compressed, such as:


-        if (ngx_memcmp(header->name.data, "path", sizeof("path") - 1)
-            == 0)
-        {
+        if (memcmp(header->name.data, "path", sizeof("path") - 1) == 0) {

-- 
<http://www.alejandro-colomar.es/>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20221104/10df94be/attachment.bin>

From mdounin at mdounin.ru  Sat Nov  5 02:39:26 2022
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Sat, 5 Nov 2022 05:39:26 +0300
Subject: [PATCH] Removed the unsafe ngx_memcmp() wrapper for memcmp(3)
In-Reply-To: <20221104152414.14208-1-alx@kernel.org>
References: <20221104152414.14208-1-alx@kernel.org>
Message-ID: <Y2XM3nf7RZj5IMEb@mdounin.ru>

Hello!

On Fri, Nov 04, 2022 at 04:24:14PM +0100, Alejandro Colomar wrote:

> The casts are unnecessary, since memcmp(3)'s arguments are
> 'const void *', which allows implicit conversion from any pointer type.
> It might have been necessary in the times of K&R C, where 'void *'
> didn't exist yet, up until the early 2000s, because some old systems
> still had limited or no support for ISO C89.  Those systems passed away
> a long time ago, and current systems, even the oldest living ones, have
> support for ISO C89.
> 
> The changes, apart from the removal of the macro itself, were scripted:
> 
> $ find src/ -type f \
>   | grep '\.[ch]$' \
>   | xargs sed -i 's/ngx_memcmp/memcmp/';
> 
> The cast in this case is (almost) innocuous, because it only hides
> warnings for conversions from integer to pointer such as:
> 
>     nxt_memcmp(n, "foo", 3);  // no warnings
>     memcmp(n, "foo", 3);      // warning: integer to pointer conversion
> 
> which is a difficult bug to write, since it's too obvious.  Such code
> will probably be caught in a code review, but there's always a small
> risk.  Since there's no reason to keep the small risk around, when we
> can just avoid it by removing the cast.
> 
> In general, it's better to avoid a cast if possible, since casts will
> disable many compiler warnings regarding type safety.

Thanks for the patch.

The ngx_memcpy() macro was introduced in 107:b5be4b0448d3 
(nginx-0.0.1-2003-07-01-19:00:03 import) as an alias to memcmp(), 
with the following explanation why it's just an alias:

+/* msvc and icc compile memcmp() to inline loop */
+#define ngx_memcmp                memcmp

This was along with the first use of memcmp in nginx, so clearly 
the "ngx_" prefix is not an accident.

Indeed, the prefix is used in multiple places to provide optimized 
or instrumented functions, even if normally they map to generally 
available functions such as memcmp().  For example, ngx_memcpy() 
usually maps to memcpy(), but instrumented to detect large copies 
if NGX_MEMCPY_LIMIT is defined.  Even if the particular function 
is not instrumented, the prefix is important to make it possible 
to change things without huge modifications of the code, like the 
one in your patch.  Further, the prefix is also important for code 
consistency.  And, last but not least, it makes it possible to add 
casts if needed on some platforms (or remove them if they are 
considered wrong).

In particular, casts to (const char *) for ngx_memcmp() were added 
in 2007 by this commit (nginx 0.6.18):

changeset:   1648:89a47f19b9ec
user:        Igor Sysoev <igor at sysoev.ru>
date:        Fri Nov 23 17:00:11 2007 +0000
summary:     update ngx_memcmp()

Unfortunately, this commit contains no details about the 
particular system which needed the cast, though it looks like 
there were some at least at that time.

Similar casts are used by almost all nginx string macro 
definitions, since most C library functions accept "char *", while 
nginx uses "u_char *" for strings, and this results in useless 
warnings.  While it might be seen as something better to avoid, 
the only alternative would be to provide wrapper functions, which 
might not be a good idea for performance and readability reasons.  
On the other hand, macro definitions with casts are easily 
readable, have no impact on performance, and provide limited to 
no effect on the code quality as long as proper coding, testing 
and review process is implemented.

Summing the above, certainly removing the "ngx_" prefix is a bad 
idea.  Removing casts from the particular macro might be 
considered, though in general it looks like a quest to prove these 
are not needed (or no longer needed) given the wide range of 
platforms nginx supports.

Hope this helps.

[...]

-- 
Maxim Dounin
http://mdounin.ru/


From nikhilsinghal993 at gmail.com  Sat Nov  5 20:32:30 2022
From: nikhilsinghal993 at gmail.com (Nikhil Singhal)
Date: Sun, 6 Nov 2022 02:02:30 +0530
Subject: ngx_log_error, ngx_slprintf signal safe
Message-ID: <CAAdXa_drNGGGAgPWyh+8YCc4MhjWLNE_OK5tEW=KS6SM6Sto-g@mail.gmail.com>

Hi All,

Wanted to know if ngx_log_error and ngx_slprintf functions are async signal
safe or not. Any help would be appreciated.

Regards,
Nikhil Singhal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20221106/fae2eb03/attachment.htm>

From alx.manpages at gmail.com  Sun Nov  6 22:50:38 2022
From: alx.manpages at gmail.com (Alejandro Colomar)
Date: Sun, 6 Nov 2022 23:50:38 +0100
Subject: [PATCH] Removed the unsafe ngx_memcmp() wrapper for memcmp(3)
In-Reply-To: <Y2XM3nf7RZj5IMEb@mdounin.ru>
References: <20221104152414.14208-1-alx@kernel.org>
 <Y2XM3nf7RZj5IMEb@mdounin.ru>
Message-ID: <de47de8f-1d68-6c03-afa2-59d467446823@gmail.com>

Hello Maxim!

On 11/5/22 03:39, Maxim Dounin wrote:
> The ngx_memcpy() macro was introduced in 107:b5be4b0448d3
> (nginx-0.0.1-2003-07-01-19:00:03 import) as an alias to memcmp(),
> with the following explanation why it's just an alias:
> 
> +/* msvc and icc compile memcmp() to inline loop */
> +#define ngx_memcmp                memcmp
> 
> This was along with the first use of memcmp in nginx, so clearly
> the "ngx_" prefix is not an accident.

Hmm, interesting point.

> 
> Indeed, the prefix is used in multiple places to provide optimized
> or instrumented functions, even if normally they map to generally
> available functions such as memcmp().  For example, ngx_memcpy()
> usually maps to memcpy(), but instrumented to detect large copies
> if NGX_MEMCPY_LIMIT is defined.

Yeah, I can understand a wrapper over memcpy() and some others for those reasons.

These days of heavily optimized assembly specializations in libc, it's not so 
easy to outperform the libc implementation, but it might still be interesting to 
check when there are large bottlenecks.

memcmp(3) is likely to be less of a candidate for instrumenting and even less 
for optimization, but might be useful to have it around if its already there.  I 
would not write a wrapper if it didn't exist, but probably I wouldn't remove it 
now that it exists.  That makes some sense to me.


>  Even if the particular function
> is not instrumented, the prefix is important to make it possible
> to change things without huge modifications of the code, like the
> one in your patch.  Further, the prefix is also important for code
> consistency.

You may be interested in replacing these few that I just found, for added 
consistency:

src/core/ngx_proxy_protocol.c:397: 
memcpy(&src_sockaddr.sockaddr_in.sin_addr, in->src_addr, 4);
src/core/ngx_proxy_protocol.c:401: 
memcpy(&dst_sockaddr.sockaddr_in.sin_addr, in->dst_addr, 4);
src/core/ngx_proxy_protocol.c:424: 
memcpy(&src_sockaddr.sockaddr_in6.sin6_addr, in6->src_addr, 16);
src/core/ngx_proxy_protocol.c:428: 
memcpy(&dst_sockaddr.sockaddr_in6.sin6_addr, in6->dst_addr, 16);

>  And, last but not least, it makes it possible to add
> casts if needed on some platforms
> (or remove them if they are
> considered wrong).
> 
> In particular, casts to (const char *) for ngx_memcmp() were added
> in 2007 by this commit (nginx 0.6.18):
> 
> changeset:   1648:89a47f19b9ec
> user:        Igor Sysoev <igor at sysoev.ru>
> date:        Fri Nov 23 17:00:11 2007 +0000
> summary:     update ngx_memcmp()
> 
> Unfortunately, this commit contains no details about the
> particular system which needed the cast, though it looks like
> there were some at least at that time.

That date can make sense.  'void *' implementations were added after ANSI/ISO 
C89.  Systems designed before that year, would mostly have 'char *'.  Since some 
systems live up to 20+ years, by 2007 there would still be some pre-C89 systems 
around, which nginx had to support with the casts.

Now in 2022 it's been 33 years after ISO C89, so all systems that lack 'void *' 
are officially dead.  Nginx might still support some dead systems, though, so it 
merits a bit more investigation to confirm.

Since memcmp(3) comes implemented together with the other mem...(3) functions, 
checking those should help:

$ grepc ngx_memset
./src/core/ngx_string.h:89:
#define ngx_memset(buf, c, n)     (void) memset(buf, c, n)
$ grepc ngx_cpymem
./src/core/ngx_string.h:97:
#define ngx_cpymem(dst, src, n)   (((u_char *) ngx_memcpy(dst, src, n)) + (n))


./src/core/ngx_string.h:107:
#define ngx_cpymem(dst, src, n)   (((u_char *) memcpy(dst, src, n)) + (n))
$ grepc ngx_memmove
./src/core/ngx_string.h:143:
#define ngx_memmove(dst, src, n)   (void) memmove(dst, src, n)


These have no casts, so we can bet that if those work without a cast, it's 
because they are implemented in the system with 'void *'.  However, let's 
confirm that; after all, it could be that we only call them with types 
compatible with 'char *'.

`grep -rn ngx_memcpy.*data.*len` should find cases where ngx_str_t is being used 
with ngx_str_t.  It finds, among others, the following line:

src/stream/ngx_stream_upstream_zone_module.c:300: 
ngx_memcpy(dst->server.data, src->server.data, src->server.len);

ngx_str_t uses the incompatible 'u_char *', so a 'char *' memcpy(3) wouldn't 
work without a cast, so memcpy(3) definitely must be implemented with 'void *', 
which implies that all mem...(3) functions, including memcmp(3), are, with a 
very strong certainty, implemented following ISO C prototypes (i.e., they use 
'void *').

Just to confirm, both 'dst' and 'src' are of type 
'ngx_stream_upstream_rr_peer_t', which is implemented as:

$ grepc ngx_stream_upstream_rr_peer_t
./src/stream/ngx_stream_upstream_round_robin.h:17:
typedef struct ngx_stream_upstream_rr_peer_s   ngx_stream_upstream_rr_peer_t;

$ grepc ngx_stream_upstream_rr_peer_s
./src/stream/ngx_stream_upstream_round_robin.h:19:
struct ngx_stream_upstream_rr_peer_s {
     [...]
     ngx_str_t                        server;

     [...]
};

So that 'server' is definitely a ngx_str_t, which confirms that there's u_char* 
involved.

> 
> Similar casts are used by almost all nginx string macro
> definitions, since most C library functions accept "char *", while
> nginx uses "u_char *" for strings, and this results in useless
> warnings.

That's true of str...(3) functions from <string.h>, but the mem...(3) functions 
from <string.h> use 'void *', so conversions are automatic, and casts are 
unnecessary (after ANSI/ISO C89; previously, K&R C did use 'char *').

>  While it might be seen as something better to avoid,

Even if it's a bit off-topic, I'm very curious about the reason for using 
u_char.  It definitely requires a lot of extra work compared to 'char *': casts, 
type-safety, reviewing that code just works when workarounding/disabling the 
compiler warnings.  I'm guessing it was also some workaround for broken old 
implementations and it has just continued like that for consistency, but am 
curious if there are other better reasons.  Certainly, ASCII characters behave 
well (at least nowadays) independently of the signedness of char, and usually 
one doesn't do arithmetic with characters in strings.

> the only alternative would be to provide wrapper functions, which
> might not be a good idea for performance and readability reasons.

Yeah, GNU always_inline or C99 inline functions could be better than macros for 
performance, but those are not always available for systems supported by nginx, 
so macros are the way to go here.

> On the other hand, macro definitions with casts are easily
> readable, have no impact on performance, and provide limited to
> no effect on the code quality as long as proper coding, testing
> and review process is implemented.

Yes, they are the least evil, or even the only way, so that makes sense.

> 
> Summing the above, certainly removing the "ngx_" prefix is a bad
> idea.

It can make sense, especially considering that nginx has always had it.

>  Removing casts from the particular macro might be
> considered, though in general it looks like a quest to prove these
> are not needed (or no longer needed) given the wide range of
> platforms nginx supports.

I don't know how far that support goes.  Maybe Windows XP?  Is that range even 
documented?

However, as far as the quest, I think the rationale above might be enough to 
prove it even without investigating specific systems.  It could certainly be 
possible that some system had a very brain-damaged <string.h> where memcmp(3) 
used a 'char *' parameter and all other mem...(3) functions used 'void *', but I 
don't think that's likely at all.

> 
> Hope this helps.

For sure!  Thank you very much :)

Cheers,

Alex

-- 
<http://www.alejandro-colomar.es/>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20221106/64ca12ed/attachment.bin>

From serg.brester at sebres.de  Mon Nov  7 11:37:30 2022
From: serg.brester at sebres.de (Sergey Brester)
Date: Mon, 07 Nov 2022 12:37:30 +0100
Subject: =?UTF-8?Q?ngx=5Flog=5Ferror=2C=20ngx=5Fslprintf=20signal=20sa?=  =?UTF-8?Q?fe?=
In-Reply-To: <CAAdXa_drNGGGAgPWyh+8YCc4MhjWLNE_OK5tEW=KS6SM6Sto-g@mail.gmail.com>
References: <CAAdXa_drNGGGAgPWyh+8YCc4MhjWLNE_OK5tEW=KS6SM6Sto-g@mail.gmail.com>
Message-ID: <5a0d47038c476570ec4873a6c2ae3888@sebres.de>

 

Hi, 

Function ngx_slprintf is conditionally async-signal safe (unless you'd
use the same buffer, supplied as first argument, or free such buffer or
some of the arguments in signal handler, because the function is not
atomic and can be interrupted by a signal). 

However regarding the function ngx_log_error (or rather
ngx_log_error_core invoked internally) it is a bit more complex (there
are still structures like log, log->connection etc used internally,
which must remain unchanged unless the function ends), but you can safe
call the function inside the handler, because it fills the buffer in
stack before logging it in one piece hereafter with function write
(being async-signal-safe corresponding POSIX) using stderr handler.

In my opinion it is safe to call both functions inside the async-signal
handler, but you should avoid to "touch" the structures and arguments
(e. g. release or modify them) that can be used in invocations of that
functions outside. 

Regards,
Sergey. 

05.11.2022 21:32, Nikhil Singhal wrote: 

> Hi All,
> 
> Wanted to know if ngx_log_error and ngx_slprintf functions are async signal safe or not. Any help would be appreciated.
> 
> Regards, 
> Nikhil Singhal 
> 
> _______________________________________________
> nginx-devel mailing list -- nginx-devel at nginx.org
> To unsubscribe send an email to nginx-devel-leave at nginx.org
 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20221107/e9afb1fa/attachment.htm>

From alx.manpages at gmail.com  Mon Nov  7 14:07:34 2022
From: alx.manpages at gmail.com (Alejandro Colomar)
Date: Mon,  7 Nov 2022 15:07:34 +0100
Subject: [PATCH 1/2] Core: added macro to calculate the length of an array.
Message-ID: <20221107140734.4630-1-alx@nginx.com>

This macro abstracts calculating how many items there are in an array.

Since recent versions of GCC and Clang provide a warning if we pass a
pointer to the construct in this macro, it is safe to use without any
further checks that the argument is an array.

Link: <https://stackoverflow.com/a/57537491>
Cc: Andrew Clayton <a.clayton at nginx.com>
Cc: Zhidao Hong <z.hong at f5.com>
Signed-off-by: Alejandro Colomar <alx at nginx.com>
---
 src/core/ngx_core.h | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/src/core/ngx_core.h b/src/core/ngx_core.h
index 7ecdca0c..bbca2df2 100644
--- a/src/core/ngx_core.h
+++ b/src/core/ngx_core.h
@@ -102,6 +102,9 @@ typedef void (*ngx_connection_handler_pt)(ngx_connection_t *c);
 #define ngx_max(val1, val2)  ((val1 < val2) ? (val2) : (val1))
 #define ngx_min(val1, val2)  ((val1 > val2) ? (val2) : (val1))
 
+#define ngx_nitems(arr)      (sizeof((arr)) / sizeof((arr)[0]))
+
+
 void ngx_cpuinfo(void);
 
 #if (NGX_HAVE_OPENAT)
-- 
2.37.2


From alx.manpages at gmail.com  Mon Nov  7 14:07:36 2022
From: alx.manpages at gmail.com (Alejandro Colomar)
Date: Mon,  7 Nov 2022 15:07:36 +0100
Subject: [PATCH 2/2] Core: replaced unsafe uses of sizeof() by ngx_nitems().
In-Reply-To: <20221107140734.4630-1-alx@nginx.com>
References: <20221107140734.4630-1-alx@nginx.com>
Message-ID: <20221107140734.4630-2-alx@nginx.com>

sizeof() should never be used to get the size of an array.  It is very
unsafe, since arrays easily decay to pointers, and sizeof() applied to
a pointer gives false results that compile and produce silent bugs.

It's especially important within macros that will be used in unplanned
cases, where a programmer would just expect it to work, rather than
inline in code where it can be more obvious that some combination is
not a good idea.

An important case where arrays can decay without the programmer
noticing is in the ternary operator (? :).  The ternary operator
applies default promotions and other undesired effects to the arguments,
which causes arrays to decay to pointers.

The following expression seems reasonable:

    ngx_string(tls ? "https://" : "http://")

And it is not.  The code above would be expanded (prior to this patch)
to:

    { sizeof(tls ? "https://" : "http://") - 1,
      (u_char *) tls ? "https://" : "http://" }

which evaluates to:

    { sizeof(char *) - 1, (u_char *) tls ? "https://" : "http://" }

which evaluates to:

    { 8 - 1, (u_char *) tls ? "https://" : "http://" }

Of course, a programmer would not want that, but rather:

    { (tls ? 9 : 8) - 1, (u_char *) tls ? "https://" : "http://" }

The worst part in this example is that since one of the strings has
exactly the same size as a pointer in most platforms, testing would
not report an issue in one of the paths of code (coincidentally, the
easier one to test), so it would be very difficult to detect this bug,
either in tests, or in code review.

This example is not a hypothetical one, but rather one that was found
by chance in Nginx Unit.  Now, imagine that both strings in the ternary
operator would have 8 bytes: tests would not possibly catch the bug,
and future changes to the code where one of the strings might change
would result in a completely unexpected bug that would be very hard to
track.

This patch makes such code trigger a compile-time warning that prevents
this class of bugs by using this macro.

This is also a recommendation that new code measuring length of arrays
uses the same macro instead of sizeof() directly.  A stackoverflow post
linked below details some more recommendations about sizeof() and
arrays.

Link: <https://stackoverflow.com/a/57537491>
Cc: Andrew Clayton <a.clayton at nginx.com>
Cc: Zhidao Hong <z.hong at f5.com>
Signed-off-by: Alejandro Colomar <alx at nginx.com>
---
 src/core/ngx_string.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/core/ngx_string.h b/src/core/ngx_string.h
index 0fb9be72..ad7b51ec 100644
--- a/src/core/ngx_string.h
+++ b/src/core/ngx_string.h
@@ -37,10 +37,10 @@ typedef struct {
 } ngx_variable_value_t;
 
 
-#define ngx_string(str)     { sizeof(str) - 1, (u_char *) str }
+#define ngx_string(str)     { ngx_nitems(str) - 1, (u_char *) str }
 #define ngx_null_string     { 0, NULL }
 #define ngx_str_set(str, text)                                               \
-    (str)->len = sizeof(text) - 1; (str)->data = (u_char *) text
+    (str)->len = ngx_nitems(text) - 1; (str)->data = (u_char *) text
 #define ngx_str_null(str)   (str)->len = 0; (str)->data = NULL
 
 
-- 
2.37.2


From v.zhestikov at f5.com  Mon Nov  7 22:10:31 2022
From: v.zhestikov at f5.com (Vadim Zhestikov)
Date: Mon, 07 Nov 2022 22:10:31 +0000
Subject: [njs] Fixed String.prototype.replace(re) if re.exec() returns
 non-flat array.
Message-ID: <hg.b80c24512157.1667859031.5965299922797593991@dev.nginx>

details:   https://hg.nginx.org/njs/rev/b80c24512157
branches:  
changeset: 1988:b80c24512157
user:      Vadim Zhestikov <v.zhestikov at f5.com>
date:      Mon Nov 07 14:08:28 2022 -0800
description:
Fixed String.prototype.replace(re) if re.exec() returns non-flat array.

This fixes #587 issue on Github.

diffstat:

 src/njs_regexp.c         |  58 +++++++++++++++--------------------------------
 src/test/njs_unit_test.c |  29 ++++++++++++++++++++++++
 2 files changed, 48 insertions(+), 39 deletions(-)

diffs (116 lines):

diff -r 4f66a66ef300 -r b80c24512157 src/njs_regexp.c
--- a/src/njs_regexp.c	Wed Oct 26 16:33:15 2022 -0700
+++ b/src/njs_regexp.c	Mon Nov 07 14:08:28 2022 -0800
@@ -1362,54 +1362,34 @@ njs_regexp_prototype_symbol_replace(njs_
 
         pos = njs_max(njs_min(pos, (int64_t) s.size), 0);
 
-        if (njs_fast_path(njs_is_fast_array(r) && njs_array_len(r) != 0)) {
-            array = njs_array(r);
+        ret = njs_object_length(vm, r, &ncaptures);
+        if (njs_slow_path(ret != NJS_OK)) {
+            goto exception;
+        }
 
-            arguments = array->start;
-            arguments[0] = matched;
-            ncaptures = njs_max((int64_t) array->length - 1, 0);
+        ncaptures = njs_min(njs_max(ncaptures - 1, 0), 99);
 
-            for (n = 1; n <= ncaptures; n++) {
-                if (njs_is_undefined(&arguments[n])) {
-                    continue;
-                }
+        array = njs_array_alloc(vm, 1, ncaptures + 1, 0);
+        if (njs_slow_path(array == NULL)) {
+            goto exception;
+        }
 
-                ret = njs_value_to_string(vm, &arguments[n], &arguments[n]);
-                if (njs_slow_path(ret == NJS_ERROR)) {
-                    goto exception;
-                }
-            }
+        arguments = array->start;
+        arguments[0] = matched;
 
-        } else {
-            ret = njs_object_length(vm, r, &ncaptures);
-            if (njs_slow_path(ret != NJS_OK)) {
+        for (n = 1; n <= ncaptures; n++) {
+            ret = njs_value_property_i64(vm, r, n, &arguments[n]);
+            if (njs_slow_path(ret == NJS_ERROR)) {
                 goto exception;
             }
 
-            ncaptures = njs_max(ncaptures - 1, 0);
-
-            array = njs_array_alloc(vm, 0, ncaptures + 1, 0);
-            if (njs_slow_path(array == NULL)) {
-                goto exception;
+            if (njs_is_undefined(&arguments[n])) {
+                continue;
             }
 
-            arguments = array->start;
-            arguments[0] = matched;
-
-            for (n = 1; n <= ncaptures; n++) {
-                ret = njs_value_property_i64(vm, r, n, &arguments[n]);
-                if (njs_slow_path(ret == NJS_ERROR)) {
-                    goto exception;
-                }
-
-                if (njs_is_undefined(&arguments[n])) {
-                    continue;
-                }
-
-                ret = njs_value_to_string(vm, &arguments[n], &arguments[n]);
-                if (njs_slow_path(ret == NJS_ERROR)) {
-                    goto exception;
-                }
+            ret = njs_value_to_string(vm, &arguments[n], &arguments[n]);
+            if (njs_slow_path(ret == NJS_ERROR)) {
+                goto exception;
             }
         }
 
diff -r 4f66a66ef300 -r b80c24512157 src/test/njs_unit_test.c
--- a/src/test/njs_unit_test.c	Wed Oct 26 16:33:15 2022 -0700
+++ b/src/test/njs_unit_test.c	Mon Nov 07 14:08:28 2022 -0800
@@ -8937,6 +8937,35 @@ static njs_unit_test_t  njs_test[] =
     { njs_str("String.bytesFrom([255,149,15,97,95]).replace(/_/g, 'X')[4]"),
       njs_str("X") },
 
+    { njs_str("var a = [];"
+              "a[2] = '';"
+              "var re = /any_regexp/;"
+              "re.exec = function () {"
+              "    return a;"
+              "};"
+              "var r = 'any_string'.replace(re);"),
+      njs_str("undefined") },
+
+    { njs_str("var a = [];"
+              "a[2] = {toString() {a[2**20] = 1; return 'X';}}; "
+              "a[4] = 'Y';"
+              "a[99] = 'Z';"
+              "a[100] = '*';"
+              "a[200] = '!';"
+              "var re = /b/;"
+              "re.exec = () => a;"
+              "'abc'.replace(re, '@$1|$2|$3|$4|$99|$100|@')"),
+      njs_str("@|X||Y|Z|0|@") },
+
+    { njs_str("var a = [];"
+              "Object.defineProperty(a, 32768, {});"
+              "var re = /any_regexp/;"
+              "re.exec = function () {"
+              "    return a;"
+              "};"
+              "var r = 'any_string'.replace(re);"),
+      njs_str("undefined") },
+
     { njs_str("/=/"),
       njs_str("/=/") },
 


From xeioex at nginx.com  Mon Nov  7 22:23:42 2022
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Mon, 07 Nov 2022 22:23:42 +0000
Subject: [njs] Fixed Array.prototype.fill() when start object changes "this".
Message-ID: <hg.94b370d9aa13.1667859822.5965299922797593991@dev.nginx>

details:   https://hg.nginx.org/njs/rev/94b370d9aa13
branches:  
changeset: 1989:94b370d9aa13
user:      Dmitry Volyntsev <xeioex at nginx.com>
date:      Mon Nov 07 14:22:41 2022 -0800
description:
Fixed Array.prototype.fill() when start object changes "this".

This fixed #589, #599 issues on Github.

diffstat:

 src/njs_array.c          |  27 ++++++++++-----------------
 src/njs_object.c         |   5 +++++
 src/test/njs_unit_test.c |   9 +++++++++
 3 files changed, 24 insertions(+), 17 deletions(-)

diffs (92 lines):

diff -r b80c24512157 -r 94b370d9aa13 src/njs_array.c
--- a/src/njs_array.c	Mon Nov 07 14:08:28 2022 -0800
+++ b/src/njs_array.c	Mon Nov 07 14:22:41 2022 -0800
@@ -1832,17 +1832,9 @@ njs_array_prototype_fill(njs_vm_t *vm, n
         return ret;
     }
 
-    array = NULL;
-
-    if (njs_is_fast_array(this)) {
-        array = njs_array(this);
-        length = array->length;
-
-    } else {
-        ret = njs_object_length(vm, this, &length);
-        if (njs_slow_path(ret == NJS_ERROR)) {
-            return ret;
-        }
+    ret = njs_object_length(vm, this, &length);
+    if (njs_slow_path(ret == NJS_ERROR)) {
+        return ret;
     }
 
     ret = njs_value_to_integer(vm, njs_arg(args, nargs, 2), &start);
@@ -1866,18 +1858,19 @@ njs_array_prototype_fill(njs_vm_t *vm, n
 
     value = njs_arg(args, nargs, 1);
 
-    if (array != NULL) {
+    if (njs_is_fast_array(this)) {
+        array = njs_array(this);
+        end = njs_min(end, array->length);
+
         for (i = start; i < end; i++) {
-            array->start[i] = *value;
+            njs_value_assign(&array->start[i], value);
         }
 
-        vm->retval = *this;
+        njs_value_assign(&vm->retval, this);
 
         return NJS_OK;
     }
 
-    value = njs_arg(args, nargs, 1);
-
     while (start < end) {
         ret = njs_value_property_i64_set(vm, this, start++, value);
         if (njs_slow_path(ret == NJS_ERROR)) {
@@ -1885,7 +1878,7 @@ njs_array_prototype_fill(njs_vm_t *vm, n
         }
     }
 
-    vm->retval = *this;
+    njs_value_assign(&vm->retval, this);
 
     return NJS_OK;
 }
diff -r b80c24512157 -r 94b370d9aa13 src/njs_object.c
--- a/src/njs_object.c	Mon Nov 07 14:08:28 2022 -0800
+++ b/src/njs_object.c	Mon Nov 07 14:22:41 2022 -0800
@@ -2505,6 +2505,11 @@ njs_object_length(njs_vm_t *vm, njs_valu
 
     const njs_value_t  string_length = njs_string("length");
 
+    if (njs_is_fast_array(value)) {
+        *length = njs_array(value)->length;
+        return NJS_OK;
+    }
+
     ret = njs_value_property(vm, value, njs_value_arg(&string_length),
                              &value_length);
     if (njs_slow_path(ret == NJS_ERROR)) {
diff -r b80c24512157 -r 94b370d9aa13 src/test/njs_unit_test.c
--- a/src/test/njs_unit_test.c	Mon Nov 07 14:08:28 2022 -0800
+++ b/src/test/njs_unit_test.c	Mon Nov 07 14:22:41 2022 -0800
@@ -5549,6 +5549,15 @@ static njs_unit_test_t  njs_test[] =
                  "Array.prototype.fill.call(o, 2).a"),
       njs_str("4") },
 
+    { njs_str("var a = (new Array(2**10)).fill(0);"
+              "var start = {valueOf() {"
+              "                 var len = a.length - 2;"
+              "                 for (var i = 0; i < len; i++) { a.shift(); }; "
+              "                 return 0;"
+              "            }};"
+              "a.fill('xxx', start)"),
+      njs_str("xxx,xxx") },
+
     { njs_str("Array.prototype.fill.call(new Int32Array(1))"),
       njs_str("0") },
 


From xeioex at nginx.com  Mon Nov  7 22:23:44 2022
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Mon, 07 Nov 2022 22:23:44 +0000
Subject: [njs] Fixed default module loader.
Message-ID: <hg.fa59bcc82142.1667859824.5965299922797593991@dev.nginx>

details:   https://hg.nginx.org/njs/rev/fa59bcc82142
branches:  
changeset: 1990:fa59bcc82142
user:      Dmitry Volyntsev <xeioex at nginx.com>
date:      Mon Nov 07 14:22:41 2022 -0800
description:
Fixed default module loader.

Previously, njs_mp_free() was called with invalid pointer because
njs_vm_compile_module() shifts start argument to the last reading
position.

This closes #601 issue on Github.

diffstat:

 src/njs_module.c |  7 +++++--
 1 files changed, 5 insertions(+), 2 deletions(-)

diffs (27 lines):

diff -r 94b370d9aa13 -r fa59bcc82142 src/njs_module.c
--- a/src/njs_module.c	Mon Nov 07 14:22:41 2022 -0800
+++ b/src/njs_module.c	Mon Nov 07 14:22:41 2022 -0800
@@ -361,10 +361,11 @@ static njs_mod_t *
 njs_default_module_loader(njs_vm_t *vm, njs_external_ptr_t external,
     njs_str_t *name)
 {
+    u_char             *start;
     njs_int_t          ret;
     njs_str_t          cwd, text;
+    njs_mod_t          *module;
     njs_parser_t       *prev;
-    njs_mod_t          *module;
     njs_module_info_t  info;
 
     prev = external;
@@ -388,7 +389,9 @@ njs_default_module_loader(njs_vm_t *vm, 
         return NULL;
     }
 
-    module = njs_vm_compile_module(vm, &info.file, &text.start,
+    start = text.start;
+
+    module = njs_vm_compile_module(vm, &info.file, &start,
                                    &text.start[text.length]);
 
     njs_mp_free(vm->mem_pool, text.start);


From xeioex at nginx.com  Mon Nov  7 22:23:46 2022
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Mon, 07 Nov 2022 22:23:46 +0000
Subject: [njs] Types: fixed description for fs.mkdir() and fs.rmdir() methods.
Message-ID: <hg.3a9415ae3330.1667859826.5965299922797593991@dev.nginx>

details:   https://hg.nginx.org/njs/rev/3a9415ae3330
branches:  
changeset: 1991:3a9415ae3330
user:      Dmitry Volyntsev <xeioex at nginx.com>
date:      Mon Nov 07 14:22:41 2022 -0800
description:
Types: fixed description for fs.mkdir() and fs.rmdir() methods.

This closes #600 issue on Github.

diffstat:

 test/ts/test.ts        |   6 ++++++
 ts/njs_modules/fs.d.ts |  22 ++++++++++++++++------
 2 files changed, 22 insertions(+), 6 deletions(-)

diffs (73 lines):

diff -r fa59bcc82142 -r 3a9415ae3330 test/ts/test.ts
--- a/test/ts/test.ts	Mon Nov 07 14:22:41 2022 -0800
+++ b/test/ts/test.ts	Mon Nov 07 14:22:41 2022 -0800
@@ -131,6 +131,12 @@ async function fs_module() {
     buffer[1] += 2;
     fs.writeSync(fd, buffer, 0, 16, 4);
     fs.closeSync(fd);
+
+    fs.mkdirSync('a/b/c', {recursive: true});
+    await fs.promises.mkdir('d/e/f', {recursive: false});
+
+    fs.rmdirSync('a/b/c', {recursive: true});
+    await fs.promises.rmdir('d/e/f', {recursive: false});
 }
 
 function qs_module(str: NjsByteString) {
diff -r fa59bcc82142 -r 3a9415ae3330 ts/njs_modules/fs.d.ts
--- a/ts/njs_modules/fs.d.ts	Mon Nov 07 14:22:41 2022 -0800
+++ b/ts/njs_modules/fs.d.ts	Mon Nov 07 14:22:41 2022 -0800
@@ -288,9 +288,12 @@ declare module "fs" {
          *
          * @since 0.4.2
          * @param path A path to a file.
-         * @param options The file mode (or an object specifying the file mode). Defaults to `0o777`.
+         * @param options A number specifying file mode, defaults to `0o777`.
+         * @param options An object with the following optional keys:
+         *   - `mode` - A number specifying file mode, defaults to `0o777`.
+         *   - `recursive` -  If true, perform a recursive directory creation, defaults to `false`.
          */
-        mkdir(path: PathLike, options?: { mode?: number } | number): Promise<void>;
+        mkdir(path: PathLike, options?: { mode?: number; recursive?: boolean; } | number): Promise<void>;
 
         /**
          * Asynchronously reads the contents of a directory at the specified `path`.
@@ -344,8 +347,10 @@ declare module "fs" {
          *
          * @since 0.4.2
          * @param path A path to a file.
+         * @param options An object with the following optional keys:
+         *   - `recursive` -  If true, perform a recursive directory removal, defaults to `false`.
          */
-        rmdir(path: PathLike): Promise<void>;
+        rmdir(path: PathLike, options?: { recursive?: boolean; }): Promise<void>;
 
         /**
          * Asynchronously retrieves `fs.Stats` object for the specified `path`.
@@ -537,9 +542,12 @@ declare module "fs" {
          *
          * @since 0.4.2
          * @param path A path to a file.
-         * @param options The file mode (or an object specifying the file mode). Defaults to `0o777`.
+         * @param options A number specifying file mode. defaults to `0o777`.
+         * @param options An object with the following optional keys:
+         *   - `mode` - A number specifying file mode, defaults to `0o777`.
+         *   - `recursive` -  If true, perform a recursive directory creation, defaults to `false`.
          */
-        mkdirSync(path: PathLike, options?: { mode?: number } | number): void;
+        mkdirSync(path: PathLike, options?: { mode?: number; recursive?: boolean; } | number): void;
 
         /**
          * Synchronously opens a file specified in the `path`.
@@ -635,8 +643,10 @@ declare module "fs" {
          *
          * @since 0.4.2
          * @param path A path to a file.
+         * @param options An object with the following optional keys:
+         *   - `recursive` -  If true, perform a recursive directory removal, defaults to `false`.
          */
-        rmdirSync(path: PathLike): void;
+        rmdirSync(path: PathLike, options?: { recursive?: boolean; }): void;
 
         /**
          * Synchronously retrieves `fs.Stats` object for the specified path.


From xeioex at nginx.com  Mon Nov  7 22:23:48 2022
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Mon, 07 Nov 2022 22:23:48 +0000
Subject: [njs] Fixed %TypedArray%.prototype.set(s) when s element changes
 "this".
Message-ID: <hg.62b475cce018.1667859828.5965299922797593991@dev.nginx>

details:   https://hg.nginx.org/njs/rev/62b475cce018
branches:  
changeset: 1992:62b475cce018
user:      Dmitry Volyntsev <xeioex at nginx.com>
date:      Mon Nov 07 14:22:41 2022 -0800
description:
Fixed %TypedArray%.prototype.set(s) when s element changes "this".

This closes #590 issue on Github.

diffstat:

 src/njs_typed_array.c    |  26 --------------------------
 src/test/njs_unit_test.c |   9 +++++++++
 2 files changed, 9 insertions(+), 26 deletions(-)

diffs (69 lines):

diff -r 3a9415ae3330 -r 62b475cce018 src/njs_typed_array.c
--- a/src/njs_typed_array.c	Mon Nov 07 14:22:41 2022 -0800
+++ b/src/njs_typed_array.c	Mon Nov 07 14:22:41 2022 -0800
@@ -694,7 +694,6 @@ njs_typed_array_prototype_set(njs_vm_t *
     int64_t             i, length, src_length, offset;
     njs_int_t           ret;
     njs_value_t         *this, *src, *value, prop;
-    njs_array_t         *array;
     njs_typed_array_t   *self, *src_tarray;
     njs_array_buffer_t  *buffer;
 
@@ -749,29 +748,6 @@ njs_typed_array_prototype_set(njs_vm_t *
         }
 
     } else {
-        if (njs_is_fast_array(src)) {
-            array = njs_array(src);
-            src_length = array->length;
-
-            if (njs_slow_path((src_length > length)
-                              || (offset > length - src_length)))
-            {
-                njs_range_error(vm, "source is too large");
-                return NJS_ERROR;
-            }
-
-            length = njs_min(array->length, length - offset);
-
-            for (i = 0; i < length; i++) {
-                ret = njs_value_to_number(vm, &array->start[i], &num);
-                if (ret == NJS_OK) {
-                    njs_typed_array_prop_set(vm, self, offset + i, num);
-                }
-            }
-
-            goto done;
-        }
-
         ret = njs_value_to_object(vm, src);
         if (njs_slow_path(ret != NJS_OK)) {
             return ret;
@@ -815,8 +791,6 @@ njs_typed_array_prototype_set(njs_vm_t *
         }
     }
 
-done:
-
     njs_set_undefined(&vm->retval);
 
     return NJS_OK;
diff -r 3a9415ae3330 -r 62b475cce018 src/test/njs_unit_test.c
--- a/src/test/njs_unit_test.c	Mon Nov 07 14:22:41 2022 -0800
+++ b/src/test/njs_unit_test.c	Mon Nov 07 14:22:41 2022 -0800
@@ -6585,6 +6585,15 @@ static njs_unit_test_t  njs_test[] =
               "           return a.toString() === '1,2,3,3'})"),
       njs_str("true") },
 
+    { njs_str(NJS_TYPED_ARRAY_LIST
+              ".every(v=>{ var a = [];"
+              "            var b = {toString() {a.length = 65535; return 99;}};"
+              "            for (var c = 0; c < 3; c++) { a[c] = b; }"
+              "            var ta = new v(6); ta.set(a);"
+              "            return ta.toString() == '99,99,99,0,0,0';"
+              "          })"),
+      njs_str("true") },
+
     { njs_str("(new Float32Array([255,255,NaN,3,NaN,Infinity,3,-Infinity,0,-0,2,1,-5])).slice(2).sort()"),
       njs_str("-Infinity,-5,0,0,1,2,3,3,Infinity,NaN,NaN") },
 


From mdounin at mdounin.ru  Tue Nov  8 06:51:37 2022
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 8 Nov 2022 09:51:37 +0300
Subject: [PATCH] SSI: ensure context of main request exists for
 subrequest using SSI
In-Reply-To: <-gUKUM8ZX15I_C0Ar8PG2FqZyzZg4agx-52A6kAx8_zLiAc-eXbg5903hwRPZxT_6wrpIvVyxTUqNlx9-2JmiTtjVlCoQwp2KM6igGnoddo=@ciel.dev>
References: <-SpypjU_KG4ByLBWOaBV_3rln7SC_v-4sAF5Z9nLtOr3EL7bvilrVTfbPXmVCw1XetJSevc6xGYxdGuqLfTfvnAYR2G1ZKcyZqTurHo0vRc=@ciel.dev>
 <Y2P1JK9cD+nMyRfh@mdounin.ru>
 <-gUKUM8ZX15I_C0Ar8PG2FqZyzZg4agx-52A6kAx8_zLiAc-eXbg5903hwRPZxT_6wrpIvVyxTUqNlx9-2JmiTtjVlCoQwp2KM6igGnoddo=@ciel.dev>
Message-ID: <Y2n8edVi8dKvTsV1@mdounin.ru>

Hello!

On Fri, Nov 04, 2022 at 09:00:33AM +0000, Ciel via nginx-devel wrote:

> Hi Maxim,
> 
> Thanks for the quick reply.
> 
> > It looks like an attempt to fix ticket #1263
> > (https://trac.nginx.org/nginx/ticket/1263). I've linked this
> > thread to the ticket. It might be a good idea to add a reference
> > into commit log.
> 
> I encountered this problem building my own website, and spent some hours to
> figure out the problem and patch. I didn't guess this bug have such history.
> Reference will be in the updated patch.
> 
> > In many cases the main request context is not needed. It might be
> > a good idea to create it only when needed, and avoid doing so if
> > it isn't.
> > 
> > Further, in theory, SSI processing of a (in-memory/background)
> > subrequest might happen even before main request is actually seen
> > by the SSI module, so the actual context will be created later.
> > This probably needs to be taken into account.
> 
> As we only use the main context for variables and stub blocks, it seems appreciable
> to have the context created on demand. However, we can't say if the SSI templates
> use these commands until we processed them. So I think this left us merely two 
> options:
> 
> a. create main context for every requests having SSI enabled (previous patch)
> b. check main context existence everywhere we use it
> 
> From my purview, most SSI templates have some variables involved, so just go ahead
> and create context can save many checks, while not introducing much waste of memory.

Certainly there are SSI templates which only use include commands, 
but you are probably right and creating the main request context 
would be easier.  With on-demand creation, even if we'll introduce 
a function to obtain/create main request context, we'll have to 
check for memory allocation errors in all places we use it.

Another option might be to link the first subrequest's context as 
the main one - till the main request context is created (if at 
all).  Especially given that we anyway have to upgrade the main 
request context if the main request is seen after the first 
subrequest.  This will imply an additional check in the body 
filter along with the flag, but a trivial one.

> > This does not look like a good solution to filter out processing
> > of a main request if SSI is not enabled for it. Rather, there
> > should be an explicit flag, so it would be possible to avoid
> > evaluation of complex constructs on each body filter call - there
> > can be a lot of such calls.
> 
> Totally agree, and this can be checked by `ngx_http_ssi_header_filter` in main request,
> to initialize the dummy context if needed. But I wonder shall I add a bit field into 
> `ngx_http_ssi_ctx_t` (as there're already some so no memory cost) or add an
> `ngx_flag_t` (8B on amd64) instead.

The ngx_flag_t type is configuration-specific, and shouldn't be 
used in runtime structures such as SSI contexts.

The general rule for runtime flags is that they are created as 
ngx_uint_t with appropriate comment if there are no other 
bit fields in the structure, for example
(src/http/modules/ngx_http_headers_filter_module.c):

    ngx_uint_t                 always;  /* unsigned  always:1 */

and upgraded to proper bit fields as long as there is more than 
one.

Given that ngx_http_ssi_ctx_t already contains bit fields, the new 
flag should be added as a bit field.

> In the issue tracker 6 years ago, you mentioned,
> 
> > It was written when there were no subrequests except subrequests created by the SSI
> > module itself, and assumes in many places that its context always exists in the main
> > request.
> 
> Though I'm not 100% familiar with the process of subrequests, this arise some more
> questions in my mind: If subrequests are processed by the SSI filter (before postpone)
> asynchronously and concurrently, could this introduce some out-of-order execution of
> SSI commands among subrequests, then lead to non-determined result? If this really
> happens, should we move the SSI filter past the postpone filter? Looking forward to
> some enlightenment.

The idea of the postpone filter is to glue all subrequests 
together in the correct order.  Basically, it makes it possible to 
execute SSI subrequests in parallel.

For sure this can be a problem if subrequests are used without 
taking this into account.  For example, if you'll set a variable 
in an SSI include virtual, trying to use it in the main SSI 
document or other included files might easily fail because it is 
not yet set.

To fix this, there is a special parameter of the include SSI 
command, "wait".  Quoting docs 
(https://nginx.org/en/docs/http/ngx_http_ssi_module.html#commands):

: wait
:
: a non-standard parameter that instructs to wait for a request to 
: fully complete before continuing with SSI processing, for example:
:
: <!--# include virtual="/remote/body.php?argument=value" wait="yes" -->

This makes it possible to add "sequence points" to SSI, resolving 
undefined behaviour due to parallel execution of requests.

-- 
Maxim Dounin
http://mdounin.ru/


From mdounin at mdounin.ru  Tue Nov  8 09:46:42 2022
From: mdounin at mdounin.ru (=?utf-8?q?Maxim_Dounin?=)
Date: Tue, 08 Nov 2022 12:46:42 +0300
Subject: [PATCH] Added logging to PROXY protocol write buffer check
Message-ID: <22c65e5f1c372f251e2c.1667900802@vm-bsd.mdounin.ru>

# HG changeset patch
# User Maxim Dounin <mdounin at mdounin.ru>
# Date 1667891773 -10800
#      Tue Nov 08 10:16:13 2022 +0300
# Node ID 22c65e5f1c372f251e2cefdd7aae743794ecfa9e
# Parent  17d6a537fb1bb587e4de22961bf5be5f0c648fa8
Added logging to PROXY protocol write buffer check.

The check is not expected to fail unless there is a bug in the calling
code.  But given the check is here, it should log an alert if it fails
instead of silently closing the connection.

diff --git a/src/core/ngx_proxy_protocol.c b/src/core/ngx_proxy_protocol.c
--- a/src/core/ngx_proxy_protocol.c
+++ b/src/core/ngx_proxy_protocol.c
@@ -282,6 +282,8 @@ ngx_proxy_protocol_write(ngx_connection_
     ngx_uint_t  port, lport;
 
     if (last - buf < NGX_PROXY_PROTOCOL_V1_MAX_HEADER) {
+        ngx_log_error(NGX_LOG_ALERT, c->log, 0,
+                      "too small buffer for PROXY protocol");
         return NULL;
     }
 


From mdounin at mdounin.ru  Tue Nov  8 09:48:52 2022
From: mdounin at mdounin.ru (=?utf-8?q?Maxim_Dounin?=)
Date: Tue, 08 Nov 2022 12:48:52 +0300
Subject: [PATCH] Fixed PROXY protocol to use ngx_memcpy()/ngx_memcmp()
Message-ID: <42bc158a47ecb3c2bd03.1667900932@vm-bsd.mdounin.ru>

# HG changeset patch
# User Maxim Dounin <mdounin at mdounin.ru>
# Date 1667900901 -10800
#      Tue Nov 08 12:48:21 2022 +0300
# Node ID 42bc158a47ecb3c2bd0396c723c307c757f2770e
# Parent  251daa98cc87111a62375941c988ed969f5d50dc
Fixed PROXY protocol to use ngx_memcpy()/ngx_memcmp().

diff --git a/src/core/ngx_proxy_protocol.c b/src/core/ngx_proxy_protocol.c
--- a/src/core/ngx_proxy_protocol.c
+++ b/src/core/ngx_proxy_protocol.c
@@ -109,7 +109,7 @@ ngx_proxy_protocol_read(ngx_connection_t
     len = last - buf;
 
     if (len >= sizeof(ngx_proxy_protocol_header_t)
-        && memcmp(p, signature, sizeof(signature) - 1) == 0)
+        && ngx_memcmp(p, signature, sizeof(signature) - 1) == 0)
     {
         return ngx_proxy_protocol_v2_read(c, buf, last);
     }
@@ -396,11 +396,11 @@ ngx_proxy_protocol_v2_read(ngx_connectio
 
         src_sockaddr.sockaddr_in.sin_family = AF_INET;
         src_sockaddr.sockaddr_in.sin_port = 0;
-        memcpy(&src_sockaddr.sockaddr_in.sin_addr, in->src_addr, 4);
+        ngx_memcpy(&src_sockaddr.sockaddr_in.sin_addr, in->src_addr, 4);
 
         dst_sockaddr.sockaddr_in.sin_family = AF_INET;
         dst_sockaddr.sockaddr_in.sin_port = 0;
-        memcpy(&dst_sockaddr.sockaddr_in.sin_addr, in->dst_addr, 4);
+        ngx_memcpy(&dst_sockaddr.sockaddr_in.sin_addr, in->dst_addr, 4);
 
         pp->src_port = ngx_proxy_protocol_parse_uint16(in->src_port);
         pp->dst_port = ngx_proxy_protocol_parse_uint16(in->dst_port);
@@ -423,11 +423,11 @@ ngx_proxy_protocol_v2_read(ngx_connectio
 
         src_sockaddr.sockaddr_in6.sin6_family = AF_INET6;
         src_sockaddr.sockaddr_in6.sin6_port = 0;
-        memcpy(&src_sockaddr.sockaddr_in6.sin6_addr, in6->src_addr, 16);
+        ngx_memcpy(&src_sockaddr.sockaddr_in6.sin6_addr, in6->src_addr, 16);
 
         dst_sockaddr.sockaddr_in6.sin6_family = AF_INET6;
         dst_sockaddr.sockaddr_in6.sin6_port = 0;
-        memcpy(&dst_sockaddr.sockaddr_in6.sin6_addr, in6->dst_addr, 16);
+        ngx_memcpy(&dst_sockaddr.sockaddr_in6.sin6_addr, in6->dst_addr, 16);
 
         pp->src_port = ngx_proxy_protocol_parse_uint16(in6->src_port);
         pp->dst_port = ngx_proxy_protocol_parse_uint16(in6->dst_port);


From mdounin at mdounin.ru  Tue Nov  8 09:50:08 2022
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 8 Nov 2022 12:50:08 +0300
Subject: [PATCH] Removed the unsafe ngx_memcmp() wrapper for memcmp(3)
In-Reply-To: <de47de8f-1d68-6c03-afa2-59d467446823@gmail.com>
References: <20221104152414.14208-1-alx@kernel.org>
 <Y2XM3nf7RZj5IMEb@mdounin.ru>
 <de47de8f-1d68-6c03-afa2-59d467446823@gmail.com>
Message-ID: <Y2omUOMNaYBFolqH@mdounin.ru>

Hello!

On Sun, Nov 06, 2022 at 11:50:38PM +0100, Alejandro Colomar wrote:

> On 11/5/22 03:39, Maxim Dounin wrote:
> > The ngx_memcpy() macro was introduced in 107:b5be4b0448d3
> > (nginx-0.0.1-2003-07-01-19:00:03 import) as an alias to memcmp(),
> > with the following explanation why it's just an alias:
> > 
> > +/* msvc and icc compile memcmp() to inline loop */
> > +#define ngx_memcmp                memcmp
> > 
> > This was along with the first use of memcmp in nginx, so clearly
> > the "ngx_" prefix is not an accident.
> 
> Hmm, interesting point.
> 
> > 
> > Indeed, the prefix is used in multiple places to provide optimized
> > or instrumented functions, even if normally they map to generally
> > available functions such as memcmp().  For example, ngx_memcpy()
> > usually maps to memcpy(), but instrumented to detect large copies
> > if NGX_MEMCPY_LIMIT is defined.
> 
> Yeah, I can understand a wrapper over memcpy() and some others for those reasons.
> 
> These days of heavily optimized assembly specializations in libc, it's not so 
> easy to outperform the libc implementation, but it might still be interesting to 
> check when there are large bottlenecks.
> 
> memcmp(3) is likely to be less of a candidate for instrumenting and even less 
> for optimization, but might be useful to have it around if its already there.  I 
> would not write a wrapper if it didn't exist, but probably I wouldn't remove it 
> now that it exists.  That makes some sense to me.
> 
> 
> >  Even if the particular function
> > is not instrumented, the prefix is important to make it possible
> > to change things without huge modifications of the code, like the
> > one in your patch.  Further, the prefix is also important for code
> > consistency.
> 
> You may be interested in replacing these few that I just found, for added 
> consistency:
> 
> src/core/ngx_proxy_protocol.c:397: 
> memcpy(&src_sockaddr.sockaddr_in.sin_addr, in->src_addr, 4);
> src/core/ngx_proxy_protocol.c:401: 
> memcpy(&dst_sockaddr.sockaddr_in.sin_addr, in->dst_addr, 4);
> src/core/ngx_proxy_protocol.c:424: 
> memcpy(&src_sockaddr.sockaddr_in6.sin6_addr, in6->src_addr, 16);
> src/core/ngx_proxy_protocol.c:428: 
> memcpy(&dst_sockaddr.sockaddr_in6.sin6_addr, in6->dst_addr, 16);

Sure, seems to be slipped in in the 7251:416953ef0428 commit 
introducing PROXY protocol v2 support.  I've submitted a patch to 
fix this.

> >  And, last but not least, it makes it possible to add
> > casts if needed on some platforms
> > (or remove them if they are
> > considered wrong).
> > 
> > In particular, casts to (const char *) for ngx_memcmp() were added
> > in 2007 by this commit (nginx 0.6.18):
> > 
> > changeset:   1648:89a47f19b9ec
> > user:        Igor Sysoev <igor at sysoev.ru>
> > date:        Fri Nov 23 17:00:11 2007 +0000
> > summary:     update ngx_memcmp()
> > 
> > Unfortunately, this commit contains no details about the
> > particular system which needed the cast, though it looks like
> > there were some at least at that time.
> 
> That date can make sense.  'void *' implementations were added after ANSI/ISO 
> C89.  Systems designed before that year, would mostly have 'char *'.  Since some 
> systems live up to 20+ years, by 2007 there would still be some pre-C89 systems 
> around, which nginx had to support with the casts.
> 
> Now in 2022 it's been 33 years after ISO C89, so all systems that lack 'void *' 
> are officially dead.  Nginx might still support some dead systems, though, so it 
> merits a bit more investigation to confirm.
> 
> Since memcmp(3) comes implemented together with the other mem...(3) functions, 
> checking those should help:
> 
> $ grepc ngx_memset
> ./src/core/ngx_string.h:89:
> #define ngx_memset(buf, c, n)     (void) memset(buf, c, n)
> $ grepc ngx_cpymem
> ./src/core/ngx_string.h:97:
> #define ngx_cpymem(dst, src, n)   (((u_char *) ngx_memcpy(dst, src, n)) + (n))
> 
> 
> ./src/core/ngx_string.h:107:
> #define ngx_cpymem(dst, src, n)   (((u_char *) memcpy(dst, src, n)) + (n))
> $ grepc ngx_memmove
> ./src/core/ngx_string.h:143:
> #define ngx_memmove(dst, src, n)   (void) memmove(dst, src, n)
> 
> 
> These have no casts, so we can bet that if those work without a cast, it's 
> because they are implemented in the system with 'void *'.  However, let's 
> confirm that; after all, it could be that we only call them with types 
> compatible with 'char *'.
> 
> `grep -rn ngx_memcpy.*data.*len` should find cases where ngx_str_t is being used 
> with ngx_str_t.  It finds, among others, the following line:
> 
> src/stream/ngx_stream_upstream_zone_module.c:300: 
> ngx_memcpy(dst->server.data, src->server.data, src->server.len);
> 
> ngx_str_t uses the incompatible 'u_char *', so a 'char *' memcpy(3) wouldn't 
> work without a cast, so memcpy(3) definitely must be implemented with 'void *', 
> which implies that all mem...(3) functions, including memcmp(3), are, with a 
> very strong certainty, implemented following ISO C prototypes (i.e., they use 
> 'void *').
> 
> Just to confirm, both 'dst' and 'src' are of type 
> 'ngx_stream_upstream_rr_peer_t', which is implemented as:
> 
> $ grepc ngx_stream_upstream_rr_peer_t
> ./src/stream/ngx_stream_upstream_round_robin.h:17:
> typedef struct ngx_stream_upstream_rr_peer_s   ngx_stream_upstream_rr_peer_t;
> 
> $ grepc ngx_stream_upstream_rr_peer_s
> ./src/stream/ngx_stream_upstream_round_robin.h:19:
> struct ngx_stream_upstream_rr_peer_s {
>      [...]
>      ngx_str_t                        server;
> 
>      [...]
> };
> 
> So that 'server' is definitely a ngx_str_t, which confirms that there's u_char* 
> involved.

The "stream" module isn't a good example, given that it's not 
built by default.  But I generally agree that given no casts in 
ngx_memcpy() it is highly unlikely that casts in ngx_memcmp() will 
do anything meaningful, at least with current nginx code.  Even if 
these casts were needed on some platforms in 2007.

Another possibility is that these casts were simply added based on 
the ngx_str*() macro definitions and weren't actually needed.  
Especially given that ngx_memcmp() was only used by ngx_memn2cmp() 
at that time, which is actually a string function.

> > Similar casts are used by almost all nginx string macro
> > definitions, since most C library functions accept "char *", while
> > nginx uses "u_char *" for strings, and this results in useless
> > warnings.
> 
> That's true of str...(3) functions from <string.h>, but the mem...(3) functions 
> from <string.h> use 'void *', so conversions are automatic, and casts are 
> unnecessary (after ANSI/ISO C89; previously, K&R C did use 'char *').

Sure.  The point is: similar casts are used by ngx_str*() macro 
definitions, so it might be copied from there even if not really 
needed.  And another point is: avoiding casts in ngx_memcmp() 
won't change anything, since there are lots of places with similar 
casts.

> >  While it might be seen as something better to avoid,
> 
> Even if it's a bit off-topic, I'm very curious about the reason for using 
> u_char.  It definitely requires a lot of extra work compared to 'char *': casts, 
> type-safety, reviewing that code just works when workarounding/disabling the 
> compiler warnings.  I'm guessing it was also some workaround for broken old 
> implementations and it has just continued like that for consistency, but am 
> curious if there are other better reasons.  Certainly, ASCII characters behave 
> well (at least nowadays) independently of the signedness of char, and usually 
> one doesn't do arithmetic with characters in strings.

Using signed chars for strings simply does not work as long as you 
consider 8-bit strings.  It results in wrong sorting unless you do 
care to compare characters as unsigned, requires careful handling 
of all range comparisons such as "ch <= 0x20", does not permit 
things like "ch < 0x80" or "c >= 0xc0", makes impossible to use 
table lookups such as "basis64[s[0]]" (all snippets are from nginx 
code).

The fact that signedness of "char" is not known adds even more 
fun: you can't really do anything without casting it to either 
unsigned char or signed char.

In general, using "char" for strings is a well known source of 
troubles at least in the Cyrillic world.  Writing the code which 
works with arbitrary chars is tricky and error-prone as long as 
you are doing anything more complex than just calling libc 
functions.  On the other hand, casts for external functions can be 
easily abstracted in most cases, and always trivial.

> > the only alternative would be to provide wrapper functions, which
> > might not be a good idea for performance and readability reasons.
> 
> Yeah, GNU always_inline or C99 inline functions could be better than macros for 
> performance, but those are not always available for systems supported by nginx, 
> so macros are the way to go here.
> 
> > On the other hand, macro definitions with casts are easily
> > readable, have no impact on performance, and provide limited to
> > no effect on the code quality as long as proper coding, testing
> > and review process is implemented.
> 
> Yes, they are the least evil, or even the only way, so that makes sense.
> 
> > 
> > Summing the above, certainly removing the "ngx_" prefix is a bad
> > idea.
> 
> It can make sense, especially considering that nginx has always had it.
> 
> >  Removing casts from the particular macro might be
> > considered, though in general it looks like a quest to prove these
> > are not needed (or no longer needed) given the wide range of
> > platforms nginx supports.
> 
> I don't know how far that support goes.  Maybe Windows XP?  Is that range even 
> documented?

Not really.  We certainly do support Windows XP, but nginx also 
happened to work on HP-UX, AIX, and even on QNX at some point 
(though there were warnings due to unsigned time_t, and likely 
associated bugs).  Some platforms are documented here:

http://nginx.org/en/#tested_os_and_platforms

And we certainly at least try to support anything reasonable / 
usable.

On the other hand, it might be good enough to require -Wno-error 
or an equivalent on ancient platforms.

> However, as far as the quest, I think the rationale above might be enough to 
> prove it even without investigating specific systems.  It could certainly be 
> possible that some system had a very brain-damaged <string.h> where memcmp(3) 
> used a 'char *' parameter and all other mem...(3) functions used 'void *', but I 
> don't think that's likely at all.

I tend to agree that we can drop casts from the ngx_memcmp() 
macro.

Note though that it's mostly nop change for the reasons given 
above.

-- 
Maxim Dounin
http://mdounin.ru/


From alx.manpages at gmail.com  Tue Nov  8 10:41:58 2022
From: alx.manpages at gmail.com (Alejandro Colomar)
Date: Tue, 8 Nov 2022 11:41:58 +0100
Subject: [PATCH] Fixed PROXY protocol to use ngx_memcpy()/ngx_memcmp()
In-Reply-To: <42bc158a47ecb3c2bd03.1667900932@vm-bsd.mdounin.ru>
References: <42bc158a47ecb3c2bd03.1667900932@vm-bsd.mdounin.ru>
Message-ID: <5e2c9c0e-80d4-deec-617e-07da5c0b0c49@gmail.com>

Hi Maxim!

On 11/8/22 10:48, Maxim Dounin wrote:
> # HG changeset patch
> # User Maxim Dounin <mdounin at mdounin.ru>
> # Date 1667900901 -10800
> #      Tue Nov 08 12:48:21 2022 +0300
> # Node ID 42bc158a47ecb3c2bd0396c723c307c757f2770e
> # Parent  251daa98cc87111a62375941c988ed969f5d50dc
> Fixed PROXY protocol to use ngx_memcpy()/ngx_memcmp().

LGTM.

If you use these:
Reviewed-by: Alejandro Colomar <alx at nginx.com>

Cheers,

Alex

> 
> diff --git a/src/core/ngx_proxy_protocol.c b/src/core/ngx_proxy_protocol.c
> --- a/src/core/ngx_proxy_protocol.c
> +++ b/src/core/ngx_proxy_protocol.c
> @@ -109,7 +109,7 @@ ngx_proxy_protocol_read(ngx_connection_t
>       len = last - buf;
>   
>       if (len >= sizeof(ngx_proxy_protocol_header_t)
> -        && memcmp(p, signature, sizeof(signature) - 1) == 0)
> +        && ngx_memcmp(p, signature, sizeof(signature) - 1) == 0)
>       {
>           return ngx_proxy_protocol_v2_read(c, buf, last);
>       }
> @@ -396,11 +396,11 @@ ngx_proxy_protocol_v2_read(ngx_connectio
>   
>           src_sockaddr.sockaddr_in.sin_family = AF_INET;
>           src_sockaddr.sockaddr_in.sin_port = 0;
> -        memcpy(&src_sockaddr.sockaddr_in.sin_addr, in->src_addr, 4);
> +        ngx_memcpy(&src_sockaddr.sockaddr_in.sin_addr, in->src_addr, 4);
>   
>           dst_sockaddr.sockaddr_in.sin_family = AF_INET;
>           dst_sockaddr.sockaddr_in.sin_port = 0;
> -        memcpy(&dst_sockaddr.sockaddr_in.sin_addr, in->dst_addr, 4);
> +        ngx_memcpy(&dst_sockaddr.sockaddr_in.sin_addr, in->dst_addr, 4);
>   
>           pp->src_port = ngx_proxy_protocol_parse_uint16(in->src_port);
>           pp->dst_port = ngx_proxy_protocol_parse_uint16(in->dst_port);
> @@ -423,11 +423,11 @@ ngx_proxy_protocol_v2_read(ngx_connectio
>   
>           src_sockaddr.sockaddr_in6.sin6_family = AF_INET6;
>           src_sockaddr.sockaddr_in6.sin6_port = 0;
> -        memcpy(&src_sockaddr.sockaddr_in6.sin6_addr, in6->src_addr, 16);
> +        ngx_memcpy(&src_sockaddr.sockaddr_in6.sin6_addr, in6->src_addr, 16);
>   
>           dst_sockaddr.sockaddr_in6.sin6_family = AF_INET6;
>           dst_sockaddr.sockaddr_in6.sin6_port = 0;
> -        memcpy(&dst_sockaddr.sockaddr_in6.sin6_addr, in6->dst_addr, 16);
> +        ngx_memcpy(&dst_sockaddr.sockaddr_in6.sin6_addr, in6->dst_addr, 16);
>   
>           pp->src_port = ngx_proxy_protocol_parse_uint16(in6->src_port);
>           pp->dst_port = ngx_proxy_protocol_parse_uint16(in6->dst_port);
> 
> _______________________________________________
> nginx-devel mailing list -- nginx-devel at nginx.org
> To unsubscribe send an email to nginx-devel-leave at nginx.org

-- 
<http://www.alejandro-colomar.es/>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20221108/bea8841c/attachment.bin>

From arut at nginx.com  Tue Nov  8 10:45:10 2022
From: arut at nginx.com (Roman Arutyunyan)
Date: Tue, 8 Nov 2022 14:45:10 +0400
Subject: [PATCH] Added logging to PROXY protocol write buffer check
In-Reply-To: <22c65e5f1c372f251e2c.1667900802@vm-bsd.mdounin.ru>
References: <22c65e5f1c372f251e2c.1667900802@vm-bsd.mdounin.ru>
Message-ID: <20221108104510.stbjwbq4zqo2si6v@N00W24XTQX>

On Tue, Nov 08, 2022 at 12:46:42PM +0300, Maxim Dounin wrote:
> # HG changeset patch
> # User Maxim Dounin <mdounin at mdounin.ru>
> # Date 1667891773 -10800
> #      Tue Nov 08 10:16:13 2022 +0300
> # Node ID 22c65e5f1c372f251e2cefdd7aae743794ecfa9e
> # Parent  17d6a537fb1bb587e4de22961bf5be5f0c648fa8
> Added logging to PROXY protocol write buffer check.
> 
> The check is not expected to fail unless there is a bug in the calling
> code.  But given the check is here, it should log an alert if it fails
> instead of silently closing the connection.
> 
> diff --git a/src/core/ngx_proxy_protocol.c b/src/core/ngx_proxy_protocol.c
> --- a/src/core/ngx_proxy_protocol.c
> +++ b/src/core/ngx_proxy_protocol.c
> @@ -282,6 +282,8 @@ ngx_proxy_protocol_write(ngx_connection_
>      ngx_uint_t  port, lport;
>  
>      if (last - buf < NGX_PROXY_PROTOCOL_V1_MAX_HEADER) {
> +        ngx_log_error(NGX_LOG_ALERT, c->log, 0,
> +                      "too small buffer for PROXY protocol");
>          return NULL;
>      }
>  
> 
> _______________________________________________
> nginx-devel mailing list -- nginx-devel at nginx.org
> To unsubscribe send an email to nginx-devel-leave at nginx.org

Looks fine


From alx.manpages at gmail.com  Tue Nov  8 10:48:35 2022
From: alx.manpages at gmail.com (Alejandro Colomar)
Date: Tue, 8 Nov 2022 11:48:35 +0100
Subject: [PATCH] Removed the unsafe ngx_memcmp() wrapper for memcmp(3)
In-Reply-To: <Y2omUOMNaYBFolqH@mdounin.ru>
References: <20221104152414.14208-1-alx@kernel.org>
 <Y2XM3nf7RZj5IMEb@mdounin.ru>
 <de47de8f-1d68-6c03-afa2-59d467446823@gmail.com>
 <Y2omUOMNaYBFolqH@mdounin.ru>
Message-ID: <d8fed8e2-0316-5c5e-cea1-6a5d894cf76c@gmail.com>

Hi Maxim!

On 11/8/22 10:50, Maxim Dounin wrote:
[...]

> 
> Another possibility is that these casts were simply added based on
> the ngx_str*() macro definitions and weren't actually needed.

Yeah, that was my most-likely guess.  But wanted to consider other possibilities 
just in case.

> Especially given that ngx_memcmp() was only used by ngx_memn2cmp()
> at that time, which is actually a string function.
> 
>>> Similar casts are used by almost all nginx string macro
>>> definitions, since most C library functions accept "char *", while
>>> nginx uses "u_char *" for strings, and this results in useless
>>> warnings.
>>
>> That's true of str...(3) functions from <string.h>, but the mem...(3) functions
>> from <string.h> use 'void *', so conversions are automatic, and casts are
>> unnecessary (after ANSI/ISO C89; previously, K&R C did use 'char *').
> 
> Sure.  The point is: similar casts are used by ngx_str*() macro
> definitions, so it might be copied from there even if not really
> needed.  And another point is: avoiding casts in ngx_memcmp()
> won't change anything, since there are lots of places with similar
> casts.

Yes, this one is (almost) a no-op change.  However, I'll still send the patch, 
since it's just a one-line patch, and more because of readability.  Not 
readability of the wrapper, which is as readable with or without the cast, but 
about programmers trying to understand why the cast is there and what can it be 
doing, since it actually does nothing.  I think that's an improvement, even if 
small in this case.

[...]

> Not really.  We certainly do support Windows XP, but nginx also
> happened to work on HP-UX, AIX, and even on QNX at some point
> (though there were warnings due to unsigned time_t, and likely
> associated bugs).  Some platforms are documented here:
> 
> http://nginx.org/en/#tested_os_and_platforms
> 
> And we certainly at least try to support anything reasonable /
> usable.
> 
> On the other hand, it might be good enough to require -Wno-error
> or an equivalent on ancient platforms.

Yeah, that makes sense.

[...]

> I tend to agree that we can drop casts from the ngx_memcmp()
> macro.
> 
> Note though that it's mostly nop change for the reasons given
> above.
> 

So, I'll send the patch in a moment.

Cheers,

Alex

-- 
<http://www.alejandro-colomar.es/>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20221108/e1e0c45b/attachment.bin>

From alx.manpages at gmail.com  Tue Nov  8 10:55:40 2022
From: alx.manpages at gmail.com (Alejandro Colomar)
Date: Tue,  8 Nov 2022 11:55:40 +0100
Subject: [PATCH v2] Removed the casts within ngx_memcmp()
In-Reply-To: <d8fed8e2-0316-5c5e-cea1-6a5d894cf76c@gmail.com>
References: <d8fed8e2-0316-5c5e-cea1-6a5d894cf76c@gmail.com>
Message-ID: <20221108105539.3924-1-alx@nginx.com>

From: Alejandro Colomar <alx at kernel.org>

The casts are unnecessary, since memcmp(3)'s arguments are
'const void *', which allows implicit conversion from any pointer type.
It might have been necessary in the times of K&R C, where 'void *'
didn't exist yet, up until the early 2000s, because some old systems
still had limited or no support for ISO C89.  Those systems passed away
a long time ago, and current systems, even the oldest living ones, have
support for ISO C89.

The cast in this case is (almost) innocuous, because it only hides
warnings for conversions from integer to pointer such as:

    nxt_memcmp(n, "foo", 3);  // no warnings
    memcmp(n, "foo", 3);      // warning: integer to pointer conversion

which is a difficult bug to write, since it's too obvious.  Such code
will probably be caught in a code review, but there's always a small
risk.  Since there's no reason to keep the small risk around, when we
can just avoid it by removing the cast.

In general, it's better to avoid a cast if possible, since casts will
disable many compiler warnings regarding type safety.

Apart from the small risk, there's a bigger concern about readability.
Casts tend to disable compiler warnings, and so are unsafe by nature.
A lot of care needs to be taken when casts are used, and a programmer
reading the cast might wonder what was that specific cast trying to
achieve.  The answer is clear: nothing.  So it's better to just prevent
that moment of "wtf is this doing here".

Signed-off-by: Alejandro Colomar <alx at nginx.com>
---
 src/core/ngx_string.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/core/ngx_string.h b/src/core/ngx_string.h
index 0fb9be72..6c218e22 100644
--- a/src/core/ngx_string.h
+++ b/src/core/ngx_string.h
@@ -145,7 +145,7 @@ ngx_copy(u_char *dst, u_char *src, size_t len)
 
 
 /* msvc and icc7 compile memcmp() to the inline loop */
-#define ngx_memcmp(s1, s2, n)  memcmp((const char *) s1, (const char *) s2, n)
+#define ngx_memcmp(s1, s2, n)  memcmp(s1, s2, n)
 
 
 u_char *ngx_cpystrn(u_char *dst, u_char *src, size_t n);
-- 
2.37.2


From mdounin at mdounin.ru  Tue Nov  8 11:12:13 2022
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 8 Nov 2022 14:12:13 +0300
Subject: [PATCH 2/2] Core: replaced unsafe uses of sizeof() by
 ngx_nitems().
In-Reply-To: <20221107140734.4630-2-alx@nginx.com>
References: <20221107140734.4630-1-alx@nginx.com>
 <20221107140734.4630-2-alx@nginx.com>
Message-ID: <Y2o5jWdJdtK1lyIO@mdounin.ru>

Hello!

On Mon, Nov 07, 2022 at 03:07:36PM +0100, Alejandro Colomar wrote:

> sizeof() should never be used to get the size of an array.  It is very
> unsafe, since arrays easily decay to pointers, and sizeof() applied to
> a pointer gives false results that compile and produce silent bugs.
> 
> It's especially important within macros that will be used in unplanned
> cases, where a programmer would just expect it to work, rather than
> inline in code where it can be more obvious that some combination is
> not a good idea.
> 
> An important case where arrays can decay without the programmer
> noticing is in the ternary operator (? :).  The ternary operator
> applies default promotions and other undesired effects to the arguments,
> which causes arrays to decay to pointers.
> 
> The following expression seems reasonable:
> 
>     ngx_string(tls ? "https://" : "http://")
> 
> And it is not.  The code above would be expanded (prior to this patch)
> to:
> 
>     { sizeof(tls ? "https://" : "http://") - 1,
>       (u_char *) tls ? "https://" : "http://" }
> 
> which evaluates to:
> 
>     { sizeof(char *) - 1, (u_char *) tls ? "https://" : "http://" }
> 
> which evaluates to:
> 
>     { 8 - 1, (u_char *) tls ? "https://" : "http://" }
> 
> Of course, a programmer would not want that, but rather:
> 
>     { (tls ? 9 : 8) - 1, (u_char *) tls ? "https://" : "http://" }
> 
> The worst part in this example is that since one of the strings has
> exactly the same size as a pointer in most platforms, testing would
> not report an issue in one of the paths of code (coincidentally, the
> easier one to test), so it would be very difficult to detect this bug,
> either in tests, or in code review.
> 
> This example is not a hypothetical one, but rather one that was found
> by chance in Nginx Unit.  Now, imagine that both strings in the ternary
> operator would have 8 bytes: tests would not possibly catch the bug,
> and future changes to the code where one of the strings might change
> would result in a completely unexpected bug that would be very hard to
> track.
> 
> This patch makes such code trigger a compile-time warning that prevents
> this class of bugs by using this macro.
> 
> This is also a recommendation that new code measuring length of arrays
> uses the same macro instead of sizeof() directly.  A stackoverflow post
> linked below details some more recommendations about sizeof() and
> arrays.
> 
> Link: <https://stackoverflow.com/a/57537491>
> Cc: Andrew Clayton <a.clayton at nginx.com>
> Cc: Zhidao Hong <z.hong at f5.com>
> Signed-off-by: Alejandro Colomar <alx at nginx.com>
> ---
>  src/core/ngx_string.h | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/src/core/ngx_string.h b/src/core/ngx_string.h
> index 0fb9be72..ad7b51ec 100644
> --- a/src/core/ngx_string.h
> +++ b/src/core/ngx_string.h
> @@ -37,10 +37,10 @@ typedef struct {
>  } ngx_variable_value_t;
>  
>  
> -#define ngx_string(str)     { sizeof(str) - 1, (u_char *) str }
> +#define ngx_string(str)     { ngx_nitems(str) - 1, (u_char *) str }
>  #define ngx_null_string     { 0, NULL }
>  #define ngx_str_set(str, text)                                               \
> -    (str)->len = sizeof(text) - 1; (str)->data = (u_char *) text
> +    (str)->len = ngx_nitems(text) - 1; (str)->data = (u_char *) text
>  #define ngx_str_null(str)   (str)->len = 0; (str)->data = NULL
>  
>  

Thanks, but no.

It's unfortunate to see that new developers in NGINX Unit actually 
do allow such trivial bugs to happen, but on the other hand that's 
somewhat expected effect of dumping the whole development team.

Either way, suggested change looks bad to me.  It obfuscates 
what's going on in the macro, making it harder to understand it 
and to use it correctly.  While it might save some newbie 
developers from making silly mistakes, it will also make the code 
harder to understand for a lot more developers.

Note well that using ternary operators in macro arguments is a bad 
coding practice regardless of whether it works with the particular 
macro or not, and should be avoided.

Hope this helps.

-- 
Maxim Dounin
http://mdounin.ru/


From alx.manpages at gmail.com  Tue Nov  8 11:15:51 2022
From: alx.manpages at gmail.com (Alejandro Colomar)
Date: Tue, 8 Nov 2022 12:15:51 +0100
Subject: u_char vs char (was: [PATCH] Removed the unsafe ngx_memcmp() wrapper
 for memcmp(3))
In-Reply-To: <Y2omUOMNaYBFolqH@mdounin.ru>
References: <20221104152414.14208-1-alx@kernel.org>
 <Y2XM3nf7RZj5IMEb@mdounin.ru>
 <de47de8f-1d68-6c03-afa2-59d467446823@gmail.com>
 <Y2omUOMNaYBFolqH@mdounin.ru>
Message-ID: <524092e5-09c7-f973-03e6-85eb7637245e@gmail.com>

Hello!

On 11/8/22 10:50, Maxim Dounin wrote:
>> Even if it's a bit off-topic, I'm very curious about the reason for using
>> u_char.  It definitely requires a lot of extra work compared to 'char *': casts,
>> type-safety, reviewing that code just works when workarounding/disabling the
>> compiler warnings.  I'm guessing it was also some workaround for broken old
>> implementations and it has just continued like that for consistency, but am
>> curious if there are other better reasons.  Certainly, ASCII characters behave
>> well (at least nowadays) independently of the signedness of char, and usually
>> one doesn't do arithmetic with characters in strings.
> 
> Using signed chars for strings simply does not work as long as you
> consider 8-bit strings.  It results in wrong sorting unless you do
> care to compare characters as unsigned, requires careful handling
> of all range comparisons such as "ch <= 0x20", does not permit
> things like "ch < 0x80" or "c >= 0xc0", makes impossible to use
> table lookups such as "basis64[s[0]]" (all snippets are from nginx
> code).
> 
> The fact that signedness of "char" is not known adds even more
> fun: you can't really do anything without casting it to either
> unsigned char or signed char.
> 
> In general, using "char" for strings is a well known source of
> troubles at least in the Cyrillic world.  Writing the code which
> works with arbitrary chars is tricky and error-prone as long as
> you are doing anything more complex than just calling libc
> functions.  On the other hand, casts for external functions can be
> easily abstracted in most cases, and always trivial.

Hmm, yeah, it makes sense.  The libc design around char instead of u_char is 
broken by design, and the requirement that libc macros need to be called with a 
cast (e.g., toupper(3)) shows that.

If nginx does things with chars other than calling libc, it makes a lot of sense 
to also use u_char.

Thanks for the rationale!  It certainly helps to understand why it was done that 
way.

Cheers,

Alex

-- 
<http://www.alejandro-colomar.es/>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20221108/034cf527/attachment.bin>

From i at ciel.dev  Tue Nov  8 17:31:25 2022
From: i at ciel.dev (Ciel)
Date: Tue, 08 Nov 2022 17:31:25 +0000
Subject: [PATCH] SSI: ensure context of main request exists for subrequest using SSI
In-Reply-To: <Y2n8edVi8dKvTsV1@mdounin.ru>
References: <-SpypjU_KG4ByLBWOaBV_3rln7SC_v-4sAF5Z9nLtOr3EL7bvilrVTfbPXmVCw1XetJSevc6xGYxdGuqLfTfvnAYR2G1ZKcyZqTurHo0vRc=@ciel.dev> <Y2P1JK9cD+nMyRfh@mdounin.ru> <-gUKUM8ZX15I_C0Ar8PG2FqZyzZg4agx-52A6kAx8_zLiAc-eXbg5903hwRPZxT_6wrpIvVyxTUqNlx9-2JmiTtjVlCoQwp2KM6igGnoddo=@ciel.dev> <Y2n8edVi8dKvTsV1@mdounin.ru>
Message-ID: <4MwuNFVwHxE7Xfc57rVEktH4gLKCciPjR6yGjd5zowzkBnEmz9Q-8THWnUDmsbtljEbclcg_Vwp0MC36DZuPhhVaXqGGR_T7DLgiOrX9fdc=@ciel.dev>

Hello! Thanks for your detailed explanation, really helps a lot!

> Another option might be to link the first subrequest's context as 
> the main one - till the main request context is created (if at 
> all).  Especially given that we anyway have to upgrade the main 
> request context if the main request is seen after the first 
> subrequest.  This will imply an additional check in the body 
> filter along with the flag, but a trivial one.

Yes, you're right, this is a valid *option C* and I have implemented that.
This can truly save bytes, if someone have SSI only on main request disabled,
but use templates without variables or blocks.

However, due to the context stealing, we should mark whether the context is for
main/subrequests, and distinguish that in body filter.  Honestly, it's kind of
counter-intuitive for me to share one context and use (type, flag) tuple to
check for validity.

> This makes it possible to add "sequence points" to SSI, resolving 
> undefined behaviour due to parallel execution of requests.

But what if the parallel subrequests do not share a common SSI parent, i.e.
introduced concurrently by other modules?  The `wait` parameter seems have dealt
with intra-module concurrency, but not inter-module ones.  If that truly is a
problem, I'm not going to cover this case in this patch.  So answer at your
interest or leave it alone.

I've implemented two patch here, the first is the just-create way fixed, and the
second is the context-stealing way.  Up to you to choose, or let me know for
other problems.

********************************************************************************
Patch A:

# HG changeset patch
# User Ciel Zhao <i at ciel.dev>
# Date 1667927876 -28800
#      Wed Nov 09 01:17:56 2022 +0800
# Node ID f7046e9deabef8c1d3caa4809a4ed5f93c17cf99
# Parent  17d6a537fb1bb587e4de22961bf5be5f0c648fa8
SSI: ensure context of main request exists for subrequest using SSI

As the SSI parser always uses the context from the main request for storing
variables and blocks, that context should always exist for subrequests using
SSI, even though the main request does not necessarily have SSI enabled.

However, `ngx_http_get_module_ctx(r->main, ...)` is getting NULL in such cases,
resulting in the worker crashing SIGSEGV when accessing its attributes.

This patch checks the context of the main request after initializing the context
for subrequests, and creates one if not exists.

diff -r 17d6a537fb1b -r f7046e9deabe src/http/modules/ngx_http_ssi_filter_module.c
--- a/src/http/modules/ngx_http_ssi_filter_module.c     Wed Nov 02 13:46:16 2022 +0400
+++ b/src/http/modules/ngx_http_ssi_filter_module.c     Wed Nov 09 01:17:56 2022 +0800
@@ -329,7 +329,7 @@
 static ngx_int_t
 ngx_http_ssi_header_filter(ngx_http_request_t *r)
 {
-    ngx_http_ssi_ctx_t       *ctx;
+    ngx_http_ssi_ctx_t       *ctx, *mctx;
     ngx_http_ssi_loc_conf_t  *slcf;
 
     slcf = ngx_http_get_module_loc_conf(r, ngx_http_ssi_filter_module);
@@ -341,14 +341,26 @@
         return ngx_http_next_header_filter(r);
     }
 
-    ctx = ngx_pcalloc(r->pool, sizeof(ngx_http_ssi_ctx_t));
+    ctx = ngx_http_get_module_ctx(r, ngx_http_ssi_filter_module);
     if (ctx == NULL) {
-        return NGX_ERROR;
+        ctx = ngx_pcalloc(r->pool, sizeof(ngx_http_ssi_ctx_t));
+        if (ctx == NULL) {
+            return NGX_ERROR;
+        }
+        ngx_http_set_ctx(r, ctx, ngx_http_ssi_filter_module);
     }
 
-    ngx_http_set_ctx(r, ctx, ngx_http_ssi_filter_module);
-
-
+    mctx = ngx_http_get_module_ctx(r->main, ngx_http_ssi_filter_module);
+    if (mctx == NULL) {
+        mctx = ngx_pcalloc(r->main->pool, sizeof(ngx_http_ssi_ctx_t));
+        if (mctx == NULL) {
+            return NGX_ERROR;
+        }
+        ngx_http_set_ctx(r->main, mctx, ngx_http_ssi_filter_module);
+    }
+
+
+    ctx->enabled = 1;
     ctx->value_len = slcf->value_len;
     ctx->last_out = &ctx->out;
 
@@ -405,6 +417,7 @@
     ctx = ngx_http_get_module_ctx(r, ngx_http_ssi_filter_module);
 
     if (ctx == NULL
+        || !ctx->enabled
         || (in == NULL
             && ctx->buf == NULL
             && ctx->in == NULL
diff -r 17d6a537fb1b -r f7046e9deabe src/http/modules/ngx_http_ssi_filter_module.h
--- a/src/http/modules/ngx_http_ssi_filter_module.h     Wed Nov 02 13:46:16 2022 +0400
+++ b/src/http/modules/ngx_http_ssi_filter_module.h     Wed Nov 09 01:17:56 2022 +0800
@@ -71,6 +71,7 @@
     u_char                   *captures_data;
 #endif
 
+    unsigned                  enabled:1;
     unsigned                  conditional:2;
     unsigned                  encoding:2;
     unsigned                  block:1;


********************************************************************************
Patch C:

# HG changeset patch
# User Ciel Zhao <i at ciel.dev>
# Date 1667928380 -28800
#      Wed Nov 09 01:26:20 2022 +0800
# Node ID 4b6f88048a6104478709b5bd9a6cc6c0c343b36c
# Parent  17d6a537fb1bb587e4de22961bf5be5f0c648fa8
SSI: ensure context of main request exists for subrequest using SSI

As the SSI parser always uses the context from the main request for storing
variables and blocks, that context should always exist for subrequests using
SSI, even though the main request does not necessarily have SSI enabled.

However, `ngx_http_get_module_ctx(r->main, ...)` is getting NULL in such cases,
resulting in the worker crashing SIGSEGV when accessing its attributes.

This patch links the first initialized context to the main request, and upgrades
it only when main context is initialized.

diff -r 17d6a537fb1b -r 4b6f88048a61 src/http/modules/ngx_http_ssi_filter_module.c
--- a/src/http/modules/ngx_http_ssi_filter_module.c     Wed Nov 02 13:46:16 2022 +0400
+++ b/src/http/modules/ngx_http_ssi_filter_module.c     Wed Nov 09 01:26:20 2022 +0800
@@ -329,7 +329,7 @@
 static ngx_int_t
 ngx_http_ssi_header_filter(ngx_http_request_t *r)
 {
-    ngx_http_ssi_ctx_t       *ctx;
+    ngx_http_ssi_ctx_t       *ctx, *sctx;
     ngx_http_ssi_loc_conf_t  *slcf;
 
     slcf = ngx_http_get_module_loc_conf(r, ngx_http_ssi_filter_module);
@@ -346,9 +346,21 @@
         return NGX_ERROR;
     }
 
+    sctx = ngx_http_get_module_ctx(r, ngx_http_ssi_filter_module);
+    if (sctx != NULL) {
+        // migrate to main context if needed, see below
+        ctx->blocks = sctx->blocks;
+        ctx->variables = sctx->variables;
+    }
+
     ngx_http_set_ctx(r, ctx, ngx_http_ssi_filter_module);
-
-
+    if (ngx_http_get_module_ctx(r->main, ngx_http_ssi_filter_module) == NULL) {
+        // set main context to current context if not exists
+        ngx_http_set_ctx(r->main, ctx, ngx_http_ssi_filter_module);
+    }
+
+
+    ctx->is_main = (r == r->main);
     ctx->value_len = slcf->value_len;
     ctx->last_out = &ctx->out;
 
@@ -405,6 +417,7 @@
     ctx = ngx_http_get_module_ctx(r, ngx_http_ssi_filter_module);
 
     if (ctx == NULL
+        || (!ctx->is_main && r == r->main)
         || (in == NULL
             && ctx->buf == NULL
             && ctx->in == NULL
diff -r 17d6a537fb1b -r 4b6f88048a61 src/http/modules/ngx_http_ssi_filter_module.h
--- a/src/http/modules/ngx_http_ssi_filter_module.h     Wed Nov 02 13:46:16 2022 +0400
+++ b/src/http/modules/ngx_http_ssi_filter_module.h     Wed Nov 09 01:26:20 2022 +0800
@@ -71,6 +71,7 @@
     u_char                   *captures_data;
 #endif
 
+    unsigned                  is_main:1;
     unsigned                  conditional:2;
     unsigned                  encoding:2;
     unsigned                  block:1;


From mdounin at mdounin.ru  Wed Nov  9 10:18:51 2022
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Wed, 9 Nov 2022 13:18:51 +0300
Subject: [PATCH] Added logging to PROXY protocol write buffer check
In-Reply-To: <20221108104510.stbjwbq4zqo2si6v@N00W24XTQX>
References: <22c65e5f1c372f251e2c.1667900802@vm-bsd.mdounin.ru>
 <20221108104510.stbjwbq4zqo2si6v@N00W24XTQX>
Message-ID: <Y2t+i8h0Ic32fWRT@mdounin.ru>

Hello!

On Tue, Nov 08, 2022 at 02:45:10PM +0400, Roman Arutyunyan wrote:

> On Tue, Nov 08, 2022 at 12:46:42PM +0300, Maxim Dounin wrote:
> > # HG changeset patch
> > # User Maxim Dounin <mdounin at mdounin.ru>
> > # Date 1667891773 -10800
> > #      Tue Nov 08 10:16:13 2022 +0300
> > # Node ID 22c65e5f1c372f251e2cefdd7aae743794ecfa9e
> > # Parent  17d6a537fb1bb587e4de22961bf5be5f0c648fa8
> > Added logging to PROXY protocol write buffer check.
> > 
> > The check is not expected to fail unless there is a bug in the calling
> > code.  But given the check is here, it should log an alert if it fails
> > instead of silently closing the connection.
> > 
> > diff --git a/src/core/ngx_proxy_protocol.c b/src/core/ngx_proxy_protocol.c
> > --- a/src/core/ngx_proxy_protocol.c
> > +++ b/src/core/ngx_proxy_protocol.c
> > @@ -282,6 +282,8 @@ ngx_proxy_protocol_write(ngx_connection_
> >      ngx_uint_t  port, lport;
> >  
> >      if (last - buf < NGX_PROXY_PROTOCOL_V1_MAX_HEADER) {
> > +        ngx_log_error(NGX_LOG_ALERT, c->log, 0,
> > +                      "too small buffer for PROXY protocol");
> >          return NULL;
> >      }
> >  
> 
> Looks fine

Pushed to http://mdounin.ru/hg/nginx along with the second patch, 
thnx.

-- 
Maxim Dounin
http://mdounin.ru/


From mdounin at mdounin.ru  Wed Nov  9 15:03:24 2022
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Wed, 9 Nov 2022 18:03:24 +0300
Subject: [PATCH v2] Removed the casts within ngx_memcmp()
In-Reply-To: <20221108105539.3924-1-alx@nginx.com>
References: <d8fed8e2-0316-5c5e-cea1-6a5d894cf76c@gmail.com>
 <20221108105539.3924-1-alx@nginx.com>
Message-ID: <Y2vBPDFSnKfWNmDX@mdounin.ru>

Hello!

On Tue, Nov 08, 2022 at 11:55:40AM +0100, Alejandro Colomar wrote:

> From: Alejandro Colomar <alx at kernel.org>
> 
> The casts are unnecessary, since memcmp(3)'s arguments are
> 'const void *', which allows implicit conversion from any pointer type.
> It might have been necessary in the times of K&R C, where 'void *'
> didn't exist yet, up until the early 2000s, because some old systems
> still had limited or no support for ISO C89.  Those systems passed away
> a long time ago, and current systems, even the oldest living ones, have
> support for ISO C89.
> 
> The cast in this case is (almost) innocuous, because it only hides
> warnings for conversions from integer to pointer such as:
> 
>     nxt_memcmp(n, "foo", 3);  // no warnings
>     memcmp(n, "foo", 3);      // warning: integer to pointer conversion
> 
> which is a difficult bug to write, since it's too obvious.  Such code
> will probably be caught in a code review, but there's always a small
> risk.  Since there's no reason to keep the small risk around, when we
> can just avoid it by removing the cast.
> 
> In general, it's better to avoid a cast if possible, since casts will
> disable many compiler warnings regarding type safety.
> 
> Apart from the small risk, there's a bigger concern about readability.
> Casts tend to disable compiler warnings, and so are unsafe by nature.
> A lot of care needs to be taken when casts are used, and a programmer
> reading the cast might wonder what was that specific cast trying to
> achieve.  The answer is clear: nothing.  So it's better to just prevent
> that moment of "wtf is this doing here".
> 
> Signed-off-by: Alejandro Colomar <alx at nginx.com>
> ---
>  src/core/ngx_string.h | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/src/core/ngx_string.h b/src/core/ngx_string.h
> index 0fb9be72..6c218e22 100644
> --- a/src/core/ngx_string.h
> +++ b/src/core/ngx_string.h
> @@ -145,7 +145,7 @@ ngx_copy(u_char *dst, u_char *src, size_t len)
>  
>  
>  /* msvc and icc7 compile memcmp() to the inline loop */
> -#define ngx_memcmp(s1, s2, n)  memcmp((const char *) s1, (const char *) s2, n)
> +#define ngx_memcmp(s1, s2, n)  memcmp(s1, s2, n)
>  
>  
>  u_char *ngx_cpystrn(u_char *dst, u_char *src, size_t n);

The suggested commit log seems to be referring to non-nginx code, 
and in general seems to be overcomplicated on the one hand, and 
lacks details supporting the claim that these casts are not needed 
in the nginx code as discussed in this thread on the other hand.

The suggested change, while being mostly style, does not take into 
account expected alignment of adjacent macro definitions: since 
the ngx_memcmp() macro definition is now short enough, it can be 
aligned similarly to ngx_memzero()/ngx_memcpy()/etc.

While here, fixed alignment of ngx_memmove() and ngx_movemem() 
macro definitions, these were off by 1 character.

# HG changeset patch
# User Maxim Dounin <mdounin at mdounin.ru>
# Date 1668004692 -10800
#      Wed Nov 09 17:38:12 2022 +0300
# Node ID fc79ea0724a92c1f463625a11ed4cb785cd342b7
# Parent  42bc158a47ecb3c2bd0396c723c307c757f2770e
Fixed alignment of ngx_memmove()/ngx_movemem() macro definitions.

diff --git a/src/core/ngx_string.h b/src/core/ngx_string.h
--- a/src/core/ngx_string.h
+++ b/src/core/ngx_string.h
@@ -140,8 +140,8 @@ ngx_copy(u_char *dst, u_char *src, size_
 #endif
 
 
-#define ngx_memmove(dst, src, n)   (void) memmove(dst, src, n)
-#define ngx_movemem(dst, src, n)   (((u_char *) memmove(dst, src, n)) + (n))
+#define ngx_memmove(dst, src, n)  (void) memmove(dst, src, n)
+#define ngx_movemem(dst, src, n)  (((u_char *) memmove(dst, src, n)) + (n))
 
 
 /* msvc and icc7 compile memcmp() to the inline loop */
# HG changeset patch
# User Maxim Dounin <mdounin at mdounin.ru>
# Date 1668005196 -10800
#      Wed Nov 09 17:46:36 2022 +0300
# Node ID 5269880f00df1e5ae08299165ec43435b759c5a3
# Parent  fc79ea0724a92c1f463625a11ed4cb785cd342b7
Removed casts from ngx_memcmp() macro.

Casts are believed to be not needed, since memcmp() has "const void *"
arguments since introduction of the "void" type in C89.  And on pre-C89
platforms nginx is unlikely to compile without warnings anyway, as there
are no casts in memcpy() and memmove() calls.

These casts were added in 1648:89a47f19b9ec without any details on why they
were added, and Igor does not remember details either.  The most plausible
explanation is that they were copied from ngx_strcmp() and were not really
needed even at that time.

Prodded by Alejandro Colomar.

diff --git a/src/core/ngx_string.h b/src/core/ngx_string.h
--- a/src/core/ngx_string.h
+++ b/src/core/ngx_string.h
@@ -145,7 +145,7 @@ ngx_copy(u_char *dst, u_char *src, size_
 
 
 /* msvc and icc7 compile memcmp() to the inline loop */
-#define ngx_memcmp(s1, s2, n)  memcmp((const char *) s1, (const char *) s2, n)
+#define ngx_memcmp(s1, s2, n)     memcmp(s1, s2, n)
 
 
 u_char *ngx_cpystrn(u_char *dst, u_char *src, size_t n);

-- 
Maxim Dounin
http://mdounin.ru/


From xeioex at nginx.com  Thu Nov 10 17:34:10 2022
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Thu, 10 Nov 2022 17:34:10 +0000
Subject: [njs] Change: the default JS stack size is reduced to 64k.
Message-ID: <hg.581c321d4015.1668101650.5965299922797593991@dev.nginx>

details:   https://hg.nginx.org/njs/rev/581c321d4015
branches:  
changeset: 1993:581c321d4015
user:      Dmitry Volyntsev <xeioex at nginx.com>
date:      Thu Nov 10 09:33:36 2022 -0800
description:
Change: the default JS stack size is reduced to 64k.

After 86784a68e8c8 (Computed goto) the size of a stack frame
njs_vmcode_interpreter() when -O0 is specified during compilation
increased significantly.  This might cause system stack overflow for very
deep recursive calls because system stack is exausted faster than JS
stack.

It is possible now to specify JS stack size for CLI.

diffstat:

 src/njs.h                |   2 ++
 src/njs_function.c       |   6 +++---
 src/njs_shell.c          |  18 ++++++++++++++++--
 src/njs_vm.c             |   3 +++
 src/njs_vm.h             |   4 ++--
 src/njs_vmcode.c         |   2 +-
 src/test/njs_unit_test.c |   2 +-
 7 files changed, 28 insertions(+), 9 deletions(-)

diffs (163 lines):

diff -r 62b475cce018 -r 581c321d4015 src/njs.h
--- a/src/njs.h	Mon Nov 07 14:22:41 2022 -0800
+++ b/src/njs.h	Thu Nov 10 09:33:36 2022 -0800
@@ -254,6 +254,8 @@ typedef struct {
     char                            **argv;
     njs_uint_t                      argc;
 
+    njs_uint_t                      max_stack_size;
+
     njs_log_level_t                 log_level;
 
 #define NJS_VM_OPT_UNHANDLED_REJECTION_IGNORE   0
diff -r 62b475cce018 -r 581c321d4015 src/njs_function.c
--- a/src/njs_function.c	Mon Nov 07 14:22:41 2022 -0800
+++ b/src/njs_function.c	Thu Nov 10 09:33:36 2022 -0800
@@ -464,7 +464,7 @@ njs_function_frame_alloc(njs_vm_t *vm, s
         spare_size = size + NJS_FRAME_SPARE_SIZE;
         spare_size = njs_align_size(spare_size, NJS_FRAME_SPARE_SIZE);
 
-        if (vm->stack_size + spare_size > NJS_MAX_STACK_SIZE) {
+        if (spare_size > vm->spare_stack_size) {
             njs_range_error(vm, "Maximum call stack size exceeded");
             return NULL;
         }
@@ -476,7 +476,7 @@ njs_function_frame_alloc(njs_vm_t *vm, s
         }
 
         chunk_size = spare_size;
-        vm->stack_size += spare_size;
+        vm->spare_stack_size -= spare_size;
     }
 
     njs_memzero(frame, sizeof(njs_native_frame_t));
@@ -702,7 +702,7 @@ njs_function_frame_free(njs_vm_t *vm, nj
         /* GC: free frame->local, etc. */
 
         if (native->size != 0) {
-            vm->stack_size -= native->size;
+            vm->spare_stack_size += native->size;
             njs_mp_free(vm->mem_pool, native);
         }
 
diff -r 62b475cce018 -r 581c321d4015 src/njs_shell.c
--- a/src/njs_shell.c	Mon Nov 07 14:22:41 2022 -0800
+++ b/src/njs_shell.c	Thu Nov 10 09:33:36 2022 -0800
@@ -39,6 +39,7 @@ typedef struct {
     uint8_t                 opcode_debug;
     uint8_t                 generator_debug;
     int                     exit_code;
+    int                     stack_size;
 
     char                    *file;
     char                    *command;
@@ -292,6 +293,10 @@ main(int argc, char **argv)
     vm_options.ast = opts.ast;
     vm_options.unhandled_rejection = opts.unhandled_rejection;
 
+    if (opts.stack_size != 0) {
+        vm_options.max_stack_size = opts.stack_size;
+    }
+
 #ifdef NJS_HAVE_READLINE
 
     if (opts.interactive) {
@@ -343,15 +348,16 @@ njs_options_parse(njs_opts_t *opts, int 
         "  -a                print AST.\n"
         "  -c                specify the command to execute.\n"
         "  -d                print disassembled code.\n"
-        "  -e                set failure exit code.\n"
+        "  -e <code>         set failure exit code.\n"
         "  -f                disabled denormals mode.\n"
 #ifdef NJS_DEBUG_GENERATOR
         "  -g                enable generator debug.\n"
 #endif
+        "  -j <size>         set the maximum stack size in bytes.\n"
 #ifdef NJS_DEBUG_OPCODE
         "  -o                enable opcode debug.\n"
 #endif
-        "  -p                set path prefix for modules.\n"
+        "  -p <path>         set path prefix for modules.\n"
         "  -q                disable interactive introduction prompt.\n"
         "  -r                ignore unhandled promise rejection.\n"
         "  -s                sandbox mode.\n"
@@ -432,6 +438,14 @@ njs_options_parse(njs_opts_t *opts, int 
             opts->generator_debug = 1;
             break;
 #endif
+        case 'j':
+            if (++i < argc) {
+                opts->stack_size = atoi(argv[i]);
+                break;
+            }
+
+            njs_stderror("option \"-j\" requires argument\n");
+            return NJS_ERROR;
 
 #ifdef NJS_DEBUG_OPCODE
         case 'o':
diff -r 62b475cce018 -r 581c321d4015 src/njs_vm.c
--- a/src/njs_vm.c	Mon Nov 07 14:22:41 2022 -0800
+++ b/src/njs_vm.c	Thu Nov 10 09:33:36 2022 -0800
@@ -26,6 +26,7 @@ njs_vm_opt_init(njs_vm_opt_t *options)
     njs_memzero(options, sizeof(njs_vm_opt_t));
 
     options->log_level = NJS_LOG_LEVEL_INFO;
+    options->max_stack_size = NJS_MAX_STACK_SIZE;
 }
 
 
@@ -71,6 +72,8 @@ njs_vm_create(njs_vm_opt_t *options)
 
     vm->external = options->external;
 
+    vm->spare_stack_size = options->max_stack_size;
+
     vm->trace.level = NJS_LEVEL_TRACE;
     vm->trace.size = 2048;
     vm->trace.data = vm;
diff -r 62b475cce018 -r 581c321d4015 src/njs_vm.h
--- a/src/njs_vm.h	Mon Nov 07 14:22:41 2022 -0800
+++ b/src/njs_vm.h	Thu Nov 10 09:33:36 2022 -0800
@@ -8,7 +8,7 @@
 #define _NJS_VM_H_INCLUDED_
 
 
-#define NJS_MAX_STACK_SIZE       (256 * 1024)
+#define NJS_MAX_STACK_SIZE       (64 * 1024)
 
 
 typedef struct njs_frame_s            njs_frame_t;
@@ -160,7 +160,7 @@ struct njs_vm_s {
     njs_mp_t                 *mem_pool;
 
     u_char                   *start;
-    size_t                   stack_size;
+    size_t                   spare_stack_size;
 
     njs_vm_shared_t          *shared;
 
diff -r 62b475cce018 -r 581c321d4015 src/njs_vmcode.c
--- a/src/njs_vmcode.c	Mon Nov 07 14:22:41 2022 -0800
+++ b/src/njs_vmcode.c	Thu Nov 10 09:33:36 2022 -0800
@@ -1837,7 +1837,7 @@ error:
         njs_vm_scopes_restore(vm, native, previous);
 
         if (native->size != 0) {
-            vm->stack_size -= native->size;
+            vm->spare_stack_size += native->size;
             njs_mp_free(vm->mem_pool, native);
         }
 
diff -r 62b475cce018 -r 581c321d4015 src/test/njs_unit_test.c
--- a/src/test/njs_unit_test.c	Mon Nov 07 14:22:41 2022 -0800
+++ b/src/test/njs_unit_test.c	Thu Nov 10 09:33:36 2022 -0800
@@ -23218,7 +23218,7 @@ njs_vm_value_test(njs_unit_test_t unused
 
     for (i = 0; i < njs_nitems(tests); i++) {
 
-        memset(&options, 0, sizeof(njs_vm_opt_t));
+        njs_vm_opt_init(&options);
         options.init = 1;
 
         vm = njs_vm_create(&options);


From xeioex at nginx.com  Fri Nov 11 01:55:05 2022
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Fri, 11 Nov 2022 01:55:05 +0000
Subject: [njs] Improved njs_object_prop_define() for fast-arrays.
Message-ID: <hg.ecf6c3e56857.1668131705.5965299922797593991@dev.nginx>

details:   https://hg.nginx.org/njs/rev/ecf6c3e56857
branches:  
changeset: 1994:ecf6c3e56857
user:      Dmitry Volyntsev <xeioex at nginx.com>
date:      Thu Nov 10 17:51:32 2022 -0800
description:
Improved njs_object_prop_define() for fast-arrays.

Previously, any Object.defineProperty() for fast-arrays converted them
to slow ones. Now, it is only done when it is necessary.

diffstat:

 src/njs_object.h         |    7 +-
 src/njs_object_prop.c    |  293 +++++++++++++++++++++++++++++-----------------
 src/njs_value.c          |   10 +-
 src/njs_value.h          |    1 +
 src/test/njs_unit_test.c |   25 +++-
 5 files changed, 216 insertions(+), 120 deletions(-)

diffs (639 lines):

diff -r 581c321d4015 -r ecf6c3e56857 src/njs_object.h
--- a/src/njs_object.h	Thu Nov 10 09:33:36 2022 -0800
+++ b/src/njs_object.h	Thu Nov 10 17:51:32 2022 -0800
@@ -18,6 +18,7 @@ typedef enum {
     NJS_OBJECT_PROP_ENUMERABLE = 8,
     NJS_OBJECT_PROP_CONFIGURABLE = 16,
     NJS_OBJECT_PROP_WRITABLE = 32,
+    NJS_OBJECT_PROP_UNSET = 64,
 #define NJS_OBJECT_PROP_VALUE_ECW (NJS_OBJECT_PROP_VALUE                     \
                                    | NJS_OBJECT_PROP_ENUMERABLE              \
                                    | NJS_OBJECT_PROP_CONFIGURABLE            \
@@ -280,13 +281,9 @@ njs_inline njs_int_t
 njs_value_create_data_prop_i64(njs_vm_t *vm, njs_value_t *value, int64_t index,
     njs_value_t *setval, uint32_t hash)
 {
-    njs_int_t    ret;
     njs_value_t  key;
 
-    ret = njs_int64_to_string(vm, &key, index);
-    if (njs_slow_path(ret != NJS_OK)) {
-        return ret;
-    }
+    njs_set_number(&key, index);
 
     return njs_value_create_data_prop(vm, value, &key, setval, hash);
 }
diff -r 581c321d4015 -r ecf6c3e56857 src/njs_object_prop.c
--- a/src/njs_object_prop.c	Thu Nov 10 09:33:36 2022 -0800
+++ b/src/njs_object_prop.c	Thu Nov 10 17:51:32 2022 -0800
@@ -8,34 +8,85 @@
 #include <njs_main.h>
 
 
-static njs_int_t njs_descriptor_prop(njs_vm_t *vm,
-    njs_object_prop_t *prop, const njs_value_t *desc);
+static njs_object_prop_t *njs_object_prop_alloc2(njs_vm_t *vm,
+    const njs_value_t *name, njs_object_prop_type_t type, unsigned flags);
+static njs_object_prop_t *njs_descriptor_prop(njs_vm_t *vm,
+    const njs_value_t *name, const njs_value_t *desc);
 
 
 njs_object_prop_t *
 njs_object_prop_alloc(njs_vm_t *vm, const njs_value_t *name,
     const njs_value_t *value, uint8_t attributes)
 {
+    unsigned           flags;
+    njs_object_prop_t  *prop;
+
+    switch (attributes) {
+    case NJS_ATTRIBUTE_FALSE:
+    case NJS_ATTRIBUTE_TRUE:
+        flags = attributes ? NJS_OBJECT_PROP_VALUE_ECW : 0;
+        break;
+
+    case NJS_ATTRIBUTE_UNSET:
+    default:
+        flags = NJS_OBJECT_PROP_UNSET;
+        break;
+    }
+
+    prop = njs_object_prop_alloc2(vm, name, NJS_PROPERTY, flags);
+    if (njs_slow_path(prop == NULL)) {
+        return NULL;
+    }
+
+    njs_value_assign(njs_prop_value(prop), value);
+
+    return prop;
+}
+
+
+static njs_object_prop_t *
+njs_object_prop_alloc2(njs_vm_t *vm, const njs_value_t *name,
+    njs_object_prop_type_t type, unsigned flags)
+{
+    njs_int_t          ret;
     njs_object_prop_t  *prop;
 
     prop = njs_mp_align(vm->mem_pool, sizeof(njs_value_t),
                         sizeof(njs_object_prop_t));
-
-    if (njs_fast_path(prop != NULL)) {
-        njs_value_assign(&prop->name, name);
-        njs_value_assign(njs_prop_value(prop), value);
+    if (njs_slow_path(prop == NULL)) {
+        njs_memory_error(vm);
+        return NULL;
+    }
 
-        prop->type = NJS_PROPERTY;
-        prop->writable = attributes;
-        prop->enumerable = attributes;
-        prop->configurable = attributes;
+    njs_value_assign(&prop->name, name);
 
-        return prop;
+    if (njs_slow_path(!njs_is_key(&prop->name))) {
+        ret = njs_value_to_key(vm, &prop->name, &prop->name);
+        if (njs_slow_path(ret != NJS_OK)) {
+            return NULL;
+        }
     }
 
-    njs_memory_error(vm);
+    prop->type = type;
+
+    if (flags != NJS_OBJECT_PROP_UNSET) {
+        prop->enumerable = !!(flags & NJS_OBJECT_PROP_ENUMERABLE);
+        prop->configurable = !!(flags & NJS_OBJECT_PROP_CONFIGURABLE);
+
+        if (type == NJS_PROPERTY) {
+            prop->writable = !!(flags & NJS_OBJECT_PROP_WRITABLE);
 
-    return NULL;
+        } else {
+            prop->writable = NJS_ATTRIBUTE_UNSET;
+        }
+
+    } else {
+        prop->enumerable = NJS_ATTRIBUTE_UNSET;
+        prop->configurable = NJS_ATTRIBUTE_UNSET;
+        prop->writable = NJS_ATTRIBUTE_UNSET;
+    }
+
+    return prop;
 }
 
 
@@ -131,7 +182,7 @@ njs_int_t
 njs_object_prop_define(njs_vm_t *vm, njs_value_t *object,
     njs_value_t *name, njs_value_t *value, unsigned flags, uint32_t hash)
 {
-    uint32_t              length;
+    uint32_t              length, index;
     njs_int_t             ret;
     njs_array_t           *array;
     njs_object_prop_t     *prop, *prev;
@@ -139,28 +190,13 @@ njs_object_prop_define(njs_vm_t *vm, njs
 
     static const njs_str_t  length_key = njs_str("length");
 
-    if (njs_slow_path(!njs_is_key(name))) {
+    if (njs_slow_path(!njs_is_index_or_key(name))) {
         ret = njs_value_to_key(vm, name, name);
         if (njs_slow_path(ret != NJS_OK)) {
             return ret;
         }
     }
 
-    if (njs_slow_path(njs_is_fast_array(object))) {
-        array = njs_array(object);
-        length = array->length;
-
-        ret = njs_array_convert_to_slow_array(vm, array);
-        if (ret != NJS_OK) {
-            return NJS_ERROR;
-        }
-
-        ret = njs_array_length_redefine(vm, object, length, 1);
-        if (njs_slow_path(ret != NJS_OK)) {
-            return ret;
-        }
-    }
-
 again:
 
     njs_property_query_init(&pq, NJS_PROPERTY_QUERY_SET, hash, 1);
@@ -173,53 +209,59 @@ again:
         return ret;
     }
 
-    prop = njs_object_prop_alloc(vm, name, &njs_value_invalid,
-                                 NJS_ATTRIBUTE_UNSET);
-    if (njs_slow_path(prop == NULL)) {
-        return NJS_ERROR;
-    }
-
     switch (njs_prop_type(flags)) {
     case NJS_OBJECT_PROP_DESCRIPTOR:
-        if (njs_descriptor_prop(vm, prop, value) != NJS_OK) {
+        prop = njs_descriptor_prop(vm, name, value);
+        if (njs_slow_path(prop == NULL)) {
             return NJS_ERROR;
         }
 
         break;
 
     case NJS_OBJECT_PROP_VALUE:
-        njs_value_assign(njs_prop_value(prop), value);
-        prop->enumerable = !!(flags & NJS_OBJECT_PROP_ENUMERABLE);
-        prop->configurable = !!(flags & NJS_OBJECT_PROP_CONFIGURABLE);
-        prop->writable = !!(flags & NJS_OBJECT_PROP_WRITABLE);
+        if (((flags & NJS_OBJECT_PROP_VALUE_ECW) == NJS_OBJECT_PROP_VALUE_ECW)
+            && njs_is_fast_array(object)
+            && njs_is_number(name))
+        {
+            if (njs_number_is_integer_index(njs_number(name))) {
+                array = njs_array(object);
+                index = (uint32_t) njs_number(name);
 
-        break;
+                if (index < array->length) {
+                    njs_value_assign(&array->start[index], value);
+                    return NJS_OK;
+                }
+            }
+        }
 
-    case NJS_OBJECT_PROP_GETTER:
-        if (!njs_is_function(value)) {
-            njs_type_error(vm, "Getter must be a function");
+        prop = njs_object_prop_alloc2(vm, name, NJS_PROPERTY,
+                                      flags & NJS_OBJECT_PROP_VALUE_ECW);
+        if (njs_slow_path(prop == NULL)) {
             return NJS_ERROR;
         }
 
-        prop->type = NJS_ACCESSOR;
-        njs_prop_getter(prop) = njs_function(value);
-        njs_prop_setter(prop) = NJS_PROP_PTR_UNSET;
-        prop->enumerable = NJS_ATTRIBUTE_TRUE;
-        prop->configurable = NJS_ATTRIBUTE_TRUE;
-
+        njs_value_assign(njs_prop_value(prop), value);
         break;
 
+    case NJS_OBJECT_PROP_GETTER:
     case NJS_OBJECT_PROP_SETTER:
-        if (!njs_is_function(value)) {
-            njs_type_error(vm, "Setter must be a function");
+    default:
+        njs_assert(njs_is_function(value));
+
+        prop = njs_object_prop_alloc2(vm, name, NJS_ACCESSOR,
+                                      NJS_OBJECT_PROP_VALUE_EC);
+        if (njs_slow_path(prop == NULL)) {
             return NJS_ERROR;
         }
 
-        prop->type = NJS_ACCESSOR;
-        njs_prop_getter(prop) = NJS_PROP_PTR_UNSET;
-        njs_prop_setter(prop) = njs_function(value);
-        prop->enumerable = NJS_ATTRIBUTE_TRUE;
-        prop->configurable = NJS_ATTRIBUTE_TRUE;
+        if (njs_prop_type(flags) == NJS_OBJECT_PROP_GETTER) {
+            njs_prop_getter(prop) = njs_function(value);
+            njs_prop_setter(prop) = NJS_PROP_PTR_UNSET;
+
+        } else {
+            njs_prop_getter(prop) = NJS_PROP_PTR_UNSET;
+            njs_prop_setter(prop) = njs_function(value);
+        }
 
         break;
     }
@@ -330,35 +372,36 @@ set_prop:
         break;
 
     case NJS_PROPERTY_REF:
-        if (njs_is_accessor_descriptor(prop)
-            || prop->configurable == NJS_ATTRIBUTE_FALSE
-            || prop->enumerable == NJS_ATTRIBUTE_FALSE
-            || prop->writable == NJS_ATTRIBUTE_FALSE)
+    case NJS_PROPERTY_PLACE_REF:
+        if (prev->type == NJS_PROPERTY_REF
+            && !njs_is_accessor_descriptor(prop)
+            && prop->configurable != NJS_ATTRIBUTE_FALSE
+            && prop->enumerable != NJS_ATTRIBUTE_FALSE
+            && prop->writable != NJS_ATTRIBUTE_FALSE)
         {
-            array = njs_array(object);
-            length = array->length;
-
-            ret = njs_array_convert_to_slow_array(vm, array);
-            if (njs_slow_path(ret != NJS_OK)) {
-                return ret;
+            if (njs_is_valid(njs_prop_value(prop))) {
+                njs_value_assign(njs_prop_ref(prev), njs_prop_value(prop));
             }
 
-            ret = njs_array_length_redefine(vm, object, length, 1);
-            if (njs_slow_path(ret != NJS_OK)) {
-                return ret;
-            }
-
-            goto again;
+            return NJS_OK;
         }
 
-        if (njs_is_valid(njs_prop_value(prop))) {
-            njs_value_assign(njs_prop_ref(prop), njs_prop_value(prop));
+        array = njs_array(object);
+        length = array->length;
 
-        } else {
-            njs_set_undefined(njs_prop_ref(prop));
+        ret = njs_array_convert_to_slow_array(vm, array);
+        if (njs_slow_path(ret != NJS_OK)) {
+            return ret;
         }
 
-        return NJS_OK;
+        ret = njs_array_length_redefine(vm, object, length, 1);
+        if (njs_slow_path(ret != NJS_OK)) {
+            return ret;
+        }
+
+        flags &= ~NJS_OBJECT_PROP_CREATE;
+
+        goto again;
 
     case NJS_PROPERTY_TYPED_ARRAY_REF:
         if (njs_is_accessor_descriptor(prop)) {
@@ -473,6 +516,27 @@ set_prop:
 
 done:
 
+    if (njs_slow_path(njs_is_fast_array(object)
+                      && pq.lhq.key_hash == NJS_LENGTH_HASH)
+                      && njs_strstr_eq(&pq.lhq.key, &length_key)
+                      && prop->writable == NJS_ATTRIBUTE_FALSE)
+    {
+        array = njs_array(object);
+        length = array->length;
+
+        ret = njs_array_convert_to_slow_array(vm, array);
+        if (njs_slow_path(ret != NJS_OK)) {
+            return ret;
+        }
+
+        ret = njs_array_length_redefine(vm, object, length, 1);
+        if (njs_slow_path(ret != NJS_OK)) {
+            return ret;
+        }
+
+        goto again;
+    }
+
     if (njs_is_accessor_descriptor(prop)) {
         prev->type = prop->type;
 
@@ -506,29 +570,26 @@ done:
             }
 
         } else {
-            if (njs_is_array(object)) {
-                if (njs_slow_path(pq.lhq.key_hash == NJS_LENGTH_HASH)) {
-
-                    if (njs_strstr_eq(&pq.lhq.key, &length_key)) {
 
-                        if (prev->configurable != 1 &&
-                            prev->writable != 1 &&
-                            !njs_values_strict_equal(njs_prop_value(prev),
-                                                     njs_prop_value(prop)))
-                        {
-                            njs_type_error(vm, "Cannot redefine "
-                                           "property: \"length\"");
-                            return NJS_ERROR;
-                        }
+            if (njs_slow_path(njs_is_array(object)
+                              && pq.lhq.key_hash == NJS_LENGTH_HASH)
+                              && njs_strstr_eq(&pq.lhq.key, &length_key))
+            {
+                if (prev->configurable != NJS_ATTRIBUTE_TRUE
+                    && prev->writable != NJS_ATTRIBUTE_TRUE
+                    && !njs_values_strict_equal(njs_prop_value(prev),
+                                                 njs_prop_value(prop)))
+                {
+                    njs_type_error(vm, "Cannot redefine property: \"length\"");
+                    return NJS_ERROR;
+                }
 
-                        if (prop->writable != NJS_ATTRIBUTE_UNSET) {
-                            prev->writable = prop->writable;
-                        }
+                if (prop->writable != NJS_ATTRIBUTE_UNSET) {
+                    prev->writable = prop->writable;
+                }
 
-                        return njs_array_length_set(vm, object, prev,
-                                                    njs_prop_value(prop));
-                    }
-                }
+                return njs_array_length_set(vm, object, prev,
+                                            njs_prop_value(prop));
             }
 
             njs_value_assign(njs_prop_value(prev), njs_prop_value(prop));
@@ -652,21 +713,28 @@ njs_prop_private_copy(njs_vm_t *vm, njs_
 }
 
 
-static njs_int_t
-njs_descriptor_prop(njs_vm_t *vm, njs_object_prop_t *prop,
+static njs_object_prop_t *
+njs_descriptor_prop(njs_vm_t *vm, const njs_value_t *name,
     const njs_value_t *desc)
 {
     njs_int_t           ret;
     njs_bool_t          data, accessor;
     njs_value_t         value;
     njs_function_t      *getter, *setter;
+    njs_object_prop_t   *prop;
     njs_lvlhsh_query_t  lhq;
 
     static const njs_value_t  get_string = njs_string("get");
 
     if (!njs_is_object(desc)) {
         njs_type_error(vm, "property descriptor must be an object");
-        return NJS_ERROR;
+        return NULL;
+    }
+
+    prop = njs_object_prop_alloc(vm, name, &njs_value_invalid,
+                                 NJS_ATTRIBUTE_UNSET);
+    if (njs_slow_path(prop == NULL)) {
+        return NULL;
     }
 
     data = 0;
@@ -678,13 +746,13 @@ njs_descriptor_prop(njs_vm_t *vm, njs_ob
 
     ret = njs_object_property(vm, desc, &lhq, &value);
     if (njs_slow_path(ret == NJS_ERROR)) {
-        return NJS_ERROR;
+        return NULL;
     }
 
     if (ret == NJS_OK) {
         if (njs_is_defined(&value) && !njs_is_function(&value)) {
             njs_type_error(vm, "Getter must be a function");
-            return NJS_ERROR;
+            return NULL;
         }
 
         accessor = 1;
@@ -696,13 +764,13 @@ njs_descriptor_prop(njs_vm_t *vm, njs_ob
 
     ret = njs_object_property(vm, desc, &lhq, &value);
     if (njs_slow_path(ret == NJS_ERROR)) {
-        return ret;
+        return NULL;
     }
 
     if (ret == NJS_OK) {
         if (njs_is_defined(&value) && !njs_is_function(&value)) {
             njs_type_error(vm, "Setter must be a function");
-            return NJS_ERROR;
+            return NULL;
         }
 
         accessor = 1;
@@ -714,7 +782,7 @@ njs_descriptor_prop(njs_vm_t *vm, njs_ob
 
     ret = njs_object_property(vm, desc, &lhq, &value);
     if (njs_slow_path(ret == NJS_ERROR)) {
-        return ret;
+        return NULL;
     }
 
     if (ret == NJS_OK) {
@@ -727,7 +795,7 @@ njs_descriptor_prop(njs_vm_t *vm, njs_ob
 
     ret = njs_object_property(vm, desc, &lhq, &value);
     if (njs_slow_path(ret == NJS_ERROR)) {
-        return ret;
+        return NULL;
     }
 
     if (ret == NJS_OK) {
@@ -738,7 +806,7 @@ njs_descriptor_prop(njs_vm_t *vm, njs_ob
     if (accessor && data) {
         njs_type_error(vm, "Cannot both specify accessors "
                            "and a value or writable attribute");
-        return NJS_ERROR;
+        return NULL;
     }
 
     lhq.key = njs_str_value("enumerable");
@@ -746,7 +814,7 @@ njs_descriptor_prop(njs_vm_t *vm, njs_ob
 
     ret = njs_object_property(vm, desc, &lhq, &value);
     if (njs_slow_path(ret == NJS_ERROR)) {
-        return ret;
+        return NULL;
     }
 
     if (ret == NJS_OK) {
@@ -758,7 +826,7 @@ njs_descriptor_prop(njs_vm_t *vm, njs_ob
 
     ret = njs_object_property(vm, desc, &lhq, &value);
     if (njs_slow_path(ret == NJS_ERROR)) {
-        return ret;
+        return NULL;
     }
 
     if (ret == NJS_OK) {
@@ -771,7 +839,7 @@ njs_descriptor_prop(njs_vm_t *vm, njs_ob
         njs_prop_setter(prop) = setter;
     }
 
-    return NJS_OK;
+    return prop;
 }
 
 
@@ -983,6 +1051,7 @@ njs_prop_type_string(njs_object_prop_typ
 {
     switch (type) {
     case NJS_PROPERTY_REF:
+    case NJS_PROPERTY_PLACE_REF:
         return "property_ref";
 
     case NJS_PROPERTY_HANDLER:
diff -r 581c321d4015 -r ecf6c3e56857 src/njs_value.c
--- a/src/njs_value.c	Thu Nov 10 09:33:36 2022 -0800
+++ b/src/njs_value.c	Thu Nov 10 17:51:32 2022 -0800
@@ -519,7 +519,7 @@ njs_value_is_buffer(const njs_value_t *v
  *     in NJS_PROPERTY_QUERY_GET
  *       prop->type is NJS_PROPERTY or NJS_PROPERTY_HANDLER.
  *     in NJS_PROPERTY_QUERY_SET, NJS_PROPERTY_QUERY_DELETE
- *       prop->type is NJS_PROPERTY, NJS_PROPERTY_REF,
+ *       prop->type is NJS_PROPERTY, NJS_PROPERTY_REF, NJS_PROPERTY_PLACE_REF,
  *       NJS_PROPERTY_TYPED_ARRAY_REF or
  *       NJS_PROPERTY_HANDLER.
  *   NJS_DECLINED         property was not found in object,
@@ -737,9 +737,12 @@ njs_array_property_query(njs_vm_t *vm, n
     int64_t            length;
     uint64_t           size;
     njs_int_t          ret;
+    njs_bool_t         resized;
     njs_value_t        *setval, value;
     njs_object_prop_t  *prop;
 
+    resized = 0;
+
     if (pq->query == NJS_PROPERTY_QUERY_SET) {
         if (!array->object.extensible) {
             return NJS_DECLINED;
@@ -764,6 +767,7 @@ njs_array_property_query(njs_vm_t *vm, n
                     }
 
                     array->length = index + 1;
+                    resized = 1;
                 }
 
                 goto prop;
@@ -829,7 +833,7 @@ prop:
 
     } else {
         njs_prop_ref(prop) = &array->start[index];
-        prop->type = NJS_PROPERTY_REF;
+        prop->type = resized ? NJS_PROPERTY_PLACE_REF : NJS_PROPERTY_REF;
     }
 
     njs_set_number(&prop->name, index);
@@ -1229,6 +1233,7 @@ slow_path:
                 goto found;
 
             case NJS_PROPERTY_REF:
+            case NJS_PROPERTY_PLACE_REF:
                 njs_value_assign(njs_prop_ref(prop), setval);
                 return NJS_OK;
 
@@ -1385,6 +1390,7 @@ njs_value_property_delete(njs_vm_t *vm, 
         return njs_function_apply(vm, njs_prop_getter(prop), value, 1, removed);
 
     case NJS_PROPERTY_REF:
+    case NJS_PROPERTY_PLACE_REF:
         if (removed != NULL) {
             njs_value_assign(removed, njs_prop_ref(prop));
         }
diff -r 581c321d4015 -r ecf6c3e56857 src/njs_value.h
--- a/src/njs_value.h	Thu Nov 10 09:33:36 2022 -0800
+++ b/src/njs_value.h	Thu Nov 10 17:51:32 2022 -0800
@@ -330,6 +330,7 @@ typedef enum {
     NJS_PROPERTY = 0,
     NJS_ACCESSOR,
     NJS_PROPERTY_REF,
+    NJS_PROPERTY_PLACE_REF,
     NJS_PROPERTY_TYPED_ARRAY_REF,
     NJS_PROPERTY_HANDLER,
     NJS_WHITEOUT,
diff -r 581c321d4015 -r ecf6c3e56857 src/test/njs_unit_test.c
--- a/src/test/njs_unit_test.c	Thu Nov 10 09:33:36 2022 -0800
+++ b/src/test/njs_unit_test.c	Thu Nov 10 17:51:32 2022 -0800
@@ -4521,6 +4521,14 @@ static njs_unit_test_t  njs_test[] =
               "Object.defineProperty(a, 'length', {writable:true})"),
       njs_str("TypeError: Cannot redefine property: \"length\"") },
 
+    { njs_str("var a = [0,1]; Object.defineProperty(a, 'length', {writable: false}); "
+              "Object.defineProperty(a, 'length', {value:12})"),
+      njs_str("TypeError: Cannot redefine property: \"length\"") },
+
+    { njs_str("var a = [0,1]; Object.defineProperty(a, 'length', {writable: false}); "
+              "Object.defineProperty(a, 'length', {value:2}); a.length"),
+      njs_str("2") },
+
     { njs_str ("var a =[0,1,2]; Object.defineProperty(a, 100, {value:100});"
               "njs.dump(a);"),
       njs_str("[0,1,2,<97 empty items>,100]") },
@@ -4718,6 +4726,17 @@ static njs_unit_test_t  njs_test[] =
               "Array.prototype.pop.call(a); [a.length, a[a.length - 1]]"),
       njs_str("15,y") },
 
+    { njs_str("var a = new Array(1), arrayPrototypeGet0Calls = 0;"
+              "Object.defineProperty(Array.prototype, '0', {"
+              "    get() { Object.defineProperty(a, 'length', {writable: false});"
+              "            arrayPrototypeGet0Calls++;"
+              "    },"
+              "});"
+              "var e = null;"
+              "try { a.pop(); } catch (ee) { e = ee.name };"
+              "[e, a.length, arrayPrototypeGet0Calls]"),
+      njs_str("TypeError,1,1") },
+
     { njs_str("[0,1].slice()"),
       njs_str("0,1") },
 
@@ -14919,7 +14938,7 @@ static njs_unit_test_t  njs_test[] =
       njs_str("a,b") },
 
     { njs_str("Object.getOwnPropertyNames(Object.defineProperty([], 'b', {}))"),
-      njs_str("length,b") },
+      njs_str("b,length") },
 
     { njs_str("Object.getOwnPropertyNames(Object.defineProperty(new String(), 'b', {}))"),
       njs_str("b,length") },
@@ -15002,6 +15021,10 @@ static njs_unit_test_t  njs_test[] =
     { njs_str("Object.defineProperty([1,2], 'a', {value:1}).a"),
       njs_str("1") },
 
+    { njs_str("var a = []; a[0] = 101; Object.defineProperty(a, 0, {});"
+              "a[0]"),
+      njs_str("101") },
+
     { njs_str("var a = Object.freeze([1,2]);"
                  "Object.defineProperty(a, 'a', {value:1}).a"),
       njs_str("TypeError: Cannot add property \"a\", object is not extensible") },


From xeioex at nginx.com  Fri Nov 11 01:55:08 2022
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Fri, 11 Nov 2022 01:55:08 +0000
Subject: [njs] Improved njs_value_property_i64_delete() for fast-arrays.
Message-ID: <hg.5964ac864676.1668131708.5965299922797593991@dev.nginx>

details:   https://hg.nginx.org/njs/rev/5964ac864676
branches:  
changeset: 1995:5964ac864676
user:      Dmitry Volyntsev <xeioex at nginx.com>
date:      Thu Nov 10 17:53:35 2022 -0800
description:
Improved njs_value_property_i64_delete() for fast-arrays.

diffstat:

 src/njs_value.c  |  33 +++++++++++++++++++++++++++------
 src/njs_value.h  |   6 +-----
 src/njs_vmcode.c |  14 ++++++++++++++
 3 files changed, 42 insertions(+), 11 deletions(-)

diffs (93 lines):

diff -r ecf6c3e56857 -r 5964ac864676 src/njs_value.c
--- a/src/njs_value.c	Thu Nov 10 17:51:32 2022 -0800
+++ b/src/njs_value.c	Thu Nov 10 17:53:35 2022 -0800
@@ -1329,20 +1329,41 @@ njs_int_t
 njs_value_property_delete(njs_vm_t *vm, njs_value_t *value, njs_value_t *key,
     njs_value_t *removed, njs_bool_t thrw)
 {
+    double                num;
+    uint32_t              index;
     njs_int_t             ret;
-    njs_value_t           primitive;
+    njs_array_t           *array;
     njs_object_prop_t     *prop;
     njs_property_query_t  pq;
 
-    if (njs_slow_path(!njs_is_key(key))) {
-        ret = njs_value_to_key(vm, &primitive, key);
-        if (njs_slow_path(ret != NJS_OK)) {
-            return NJS_ERROR;
+    njs_assert(njs_is_index_or_key(key));
+
+    if (njs_fast_path(njs_is_number(key))) {
+        if (njs_slow_path(!(njs_is_fast_array(value)))) {
+            goto slow_path;
+        }
+
+        num = njs_number(key);
+
+        if (njs_slow_path(!njs_number_is_integer_index(num))) {
+            goto slow_path;
         }
 
-        key = &primitive;
+        index = (uint32_t) num;
+
+        array = njs_array(value);
+
+        if (njs_slow_path(index >= array->length)) {
+            goto slow_path;
+        }
+
+        njs_value_assign(&array->start[index], &njs_value_invalid);
+
+        return NJS_OK;
     }
 
+slow_path:
+
     njs_property_query_init(&pq, NJS_PROPERTY_QUERY_DELETE, 0, 1);
 
     ret = njs_property_query(vm, &pq, value, key);
diff -r ecf6c3e56857 -r 5964ac864676 src/njs_value.h
--- a/src/njs_value.h	Thu Nov 10 17:51:32 2022 -0800
+++ b/src/njs_value.h	Thu Nov 10 17:53:35 2022 -0800
@@ -1142,13 +1142,9 @@ njs_inline njs_int_t
 njs_value_property_i64_delete(njs_vm_t *vm, njs_value_t *value, int64_t index,
     njs_value_t *removed)
 {
-    njs_int_t    ret;
     njs_value_t  key;
 
-    ret = njs_int64_to_string(vm, &key, index);
-    if (njs_slow_path(ret != NJS_OK)) {
-        return ret;
-    }
+    njs_set_number(&key, index);
 
     return njs_value_property_delete(vm, value, &key, removed, 1);
 }
diff -r ecf6c3e56857 -r 5964ac864676 src/njs_vmcode.c
--- a/src/njs_vmcode.c	Thu Nov 10 17:51:32 2022 -0800
+++ b/src/njs_vmcode.c	Thu Nov 10 17:53:35 2022 -0800
@@ -932,6 +932,20 @@ NEXT_LBL;
         njs_vmcode_operand(vm, vmcode->operand3, value2);
         njs_vmcode_operand(vm, vmcode->operand2, value1);
 
+        if (njs_slow_path(!njs_is_index_or_key(value2))) {
+            if (njs_slow_path(njs_is_null_or_undefined(value1))) {
+                (void) njs_throw_cannot_property(vm, value1, value2, "delete");
+                goto error;
+            }
+
+            ret = njs_value_to_key(vm, &primitive1, value2);
+            if (njs_slow_path(ret != NJS_OK)) {
+                goto error;
+            }
+
+            value2 = &primitive1;
+        }
+
         ret = njs_value_property_delete(vm, value1, value2, NULL, 1);
         if (njs_fast_path(ret != NJS_ERROR)) {
             vm->retval = njs_value_true;


From xeioex at nginx.com  Fri Nov 11 01:55:09 2022
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Fri, 11 Nov 2022 01:55:09 +0000
Subject: [njs] Fixed Array.prototype.splice(s,
 d) when d resizes "this" during eval.
Message-ID: <hg.1d6cea817ef4.1668131709.5965299922797593991@dev.nginx>

details:   https://hg.nginx.org/njs/rev/1d6cea817ef4
branches:  
changeset: 1996:1d6cea817ef4
user:      Dmitry Volyntsev <xeioex at nginx.com>
date:      Thu Nov 10 17:53:36 2022 -0800
description:
Fixed Array.prototype.splice(s,d) when d resizes "this" during eval.

This closes #592 and #595 issues on Github.

diffstat:

 src/njs_array.c          |  116 ++++++++++++++++++----------------------------
 src/test/njs_unit_test.c |   21 ++++++++
 2 files changed, 67 insertions(+), 70 deletions(-)

diffs (201 lines):

diff -r 5964ac864676 -r 1d6cea817ef4 src/njs_array.c
--- a/src/njs_array.c	Thu Nov 10 17:53:35 2022 -0800
+++ b/src/njs_array.c	Thu Nov 10 17:53:36 2022 -0800
@@ -274,18 +274,24 @@ static njs_int_t
 njs_array_copy_within(njs_vm_t *vm, njs_value_t *array, int64_t to_pos,
     int64_t from_pos, int64_t count, njs_bool_t forward)
 {
-    int64_t      i, from, to;
+    int64_t      i, from, to, len;
     njs_int_t    ret;
     njs_array_t  *arr;
     njs_value_t  value;
 
+    njs_assert(to_pos >= 0);
+    njs_assert(from_pos >= 0);
+
     if (njs_fast_path(njs_is_fast_array(array) && count > 0)) {
         arr = njs_array(array);
-
-        memmove(&arr->start[to_pos], &arr->start[from_pos],
-                count * sizeof(njs_value_t));
-
-        return NJS_OK;
+        len = arr->length;
+
+        if (to_pos + count < len && from_pos + count < len) {
+            memmove(&arr->start[to_pos], &arr->start[from_pos],
+                    count * sizeof(njs_value_t));
+
+            return NJS_OK;
+        }
     }
 
     if (!forward) {
@@ -1240,15 +1246,19 @@ njs_array_prototype_splice(njs_vm_t *vm,
         return NJS_ERROR;
     }
 
-    if (njs_fast_path(njs_is_fast_array(this) && deleted->object.fast_array)) {
+    njs_set_array(&del_object, deleted);
+
+    if (njs_fast_path(njs_is_fast_array(this)
+                      && deleted->object.fast_array
+                      && delete <= deleted->length
+                      && start + delete <= njs_array_len(this)))
+    {
         array = njs_array(this);
         for (i = 0, n = start; i < delete; i++, n++) {
-            deleted->start[i] = array->start[n];
+            njs_value_assign(&deleted->start[i], &array->start[n]);
         }
 
     } else {
-        njs_set_array(&del_object, deleted);
-
         for (i = 0, n = start; i < delete; i++, n++) {
             ret = njs_value_property_i64(vm, this, n, &value);
             if (njs_slow_path(ret == NJS_ERROR)) {
@@ -1256,8 +1266,8 @@ njs_array_prototype_splice(njs_vm_t *vm,
             }
 
             if (ret == NJS_OK) {
-                /* TODO:  CreateDataPropertyOrThrow(). */
-                ret = njs_value_property_i64_set(vm, &del_object, i, &value);
+                ret = njs_value_create_data_prop_i64(vm, &del_object, i, &value,
+                                                     0);
                 if (njs_slow_path(ret == NJS_ERROR)) {
                     return ret;
                 }
@@ -1268,76 +1278,42 @@ njs_array_prototype_splice(njs_vm_t *vm,
                 }
             }
         }
-
-        ret = njs_object_length_set(vm, &del_object, delete);
-        if (njs_slow_path(ret != NJS_OK)) {
-            return NJS_ERROR;
-        }
+    }
+
+    ret = njs_object_length_set(vm, &del_object, delete);
+    if (njs_slow_path(ret != NJS_OK)) {
+        return NJS_ERROR;
     }
 
-    if (njs_fast_path(njs_is_fast_array(this))) {
-        array = njs_array(this);
-
-        if (delta != 0) {
-            /*
-             * Relocate the rest of items.
-             * Index of the first item is in "n".
-             */
-            if (delta > 0) {
-                ret = njs_array_expand(vm, array, 0, delta);
-                if (njs_slow_path(ret != NJS_OK)) {
-                    return ret;
-                }
-            }
-
-            ret = njs_array_copy_within(vm, this, start + items, start + delete,
-                                        array->length - (start + delete), 0);
-            if (njs_slow_path(ret != NJS_OK)) {
-                return ret;
-            }
-
-            array->length += delta;
+   if (delta != 0) {
+       ret = njs_array_copy_within(vm, this, start + items, start + delete,
+                                   length - (start + delete), delta < 0);
+        if (njs_slow_path(ret != NJS_OK)) {
+            return ret;
         }
 
-        /* Copy new items. */
-
-        if (items > 0) {
-            memcpy(&array->start[start], &args[3],
-                   items * sizeof(njs_value_t));
-        }
-
-    } else {
-
-       if (delta != 0) {
-           ret = njs_array_copy_within(vm, this, start + items, start + delete,
-                                       length - (start + delete), delta < 0);
-            if (njs_slow_path(ret != NJS_OK)) {
-                return ret;
-            }
-
-            for (i = length - 1; i >= length + delta; i--) {
-                ret = njs_value_property_i64_delete(vm, this, i, NULL);
-                if (njs_slow_path(ret == NJS_ERROR)) {
-                    return NJS_ERROR;
-                }
-            }
-       }
-
-        /* Copy new items. */
-
-        for (i = 3, n = start; items-- > 0; i++, n++) {
-            ret = njs_value_property_i64_set(vm, this, n, &args[i]);
+        for (i = length - 1; i >= length + delta; i--) {
+            ret = njs_value_property_i64_delete(vm, this, i, NULL);
             if (njs_slow_path(ret == NJS_ERROR)) {
                 return NJS_ERROR;
             }
         }
-
-        ret = njs_object_length_set(vm, this, length + delta);
-        if (njs_slow_path(ret != NJS_OK)) {
+   }
+
+    /* Copy new items. */
+
+    for (i = 3, n = start; items-- > 0; i++, n++) {
+        ret = njs_value_property_i64_set(vm, this, n, &args[i]);
+        if (njs_slow_path(ret == NJS_ERROR)) {
             return NJS_ERROR;
         }
     }
 
+    ret = njs_object_length_set(vm, this, length + delta);
+    if (njs_slow_path(ret != NJS_OK)) {
+        return NJS_ERROR;
+    }
+
     njs_set_array(&vm->retval, deleted);
 
     return NJS_OK;
diff -r 5964ac864676 -r 1d6cea817ef4 src/test/njs_unit_test.c
--- a/src/test/njs_unit_test.c	Thu Nov 10 17:53:35 2022 -0800
+++ b/src/test/njs_unit_test.c	Thu Nov 10 17:53:36 2022 -0800
@@ -4980,6 +4980,27 @@ static njs_unit_test_t  njs_test[] =
               ".map(v=>v.join(''))"),
       njs_str(",1345,,1,13,13,13") },
 
+    { njs_str("var a = ['x'];"
+              "var d = a.splice(0, { valueOf() {  a.length = 0; return 10; } });"
+              "njs.dump(d)"),
+      njs_str("[<empty>]") },
+
+    { njs_str("var a = ['a', 'b', 'c'];"
+              "var d = a.splice(0, { valueOf() {  a.length = 2; return 3; } });"
+              "njs.dump(d)"),
+      njs_str("['a','b',<empty>]") },
+
+#if NJS_HAVE_LARGE_STACK
+    { njs_str("let arr = [ 'x' ];"
+              "let a = { toString() {"
+               "          new Float64Array(100).set(["
+               "            {toString() {Array.prototype.splice.call(arr, a)}}"
+               "          ])"
+               "        }};"
+               "a.toString()"),
+      njs_str("RangeError: Maximum call stack size exceeded") },
+#endif
+
     { njs_str("var o = { toString: () => {"
               "             for (var i = 0; i < 0x10; i++) {a.push(1)};"
               "             return {};"


From v.zhestikov at f5.com  Mon Nov 14 17:26:38 2022
From: v.zhestikov at f5.com (Vadim Zhestikov)
Date: Mon, 14 Nov 2022 17:26:38 +0000
Subject: [njs] Fixed for-in loop with left and right hand side expressions.
Message-ID: <hg.283ae119d121.1668446798.5965299922797593991@dev.nginx>

details:   https://hg.nginx.org/njs/rev/283ae119d121
branches:  
changeset: 1997:283ae119d121
user:      Vadim Zhestikov <v.zhestikov at f5.com>
date:      Mon Nov 14 09:18:37 2022 -0800
description:
Fixed for-in loop with left and right hand side expressions.

This fixes #351 issue on Github.

diffstat:

 src/njs_generator.c      |  365 ++++++++++++++++++++++++++++++++++++++++++++--
 src/njs_lexer.c          |  112 ++++++++++++++-
 src/njs_lexer.h          |   14 +-
 src/njs_parser.c         |  309 ++++++++++++++++++++++++++++++++++++--
 src/njs_parser.h         |    2 +
 src/test/njs_unit_test.c |   62 +++++++
 6 files changed, 822 insertions(+), 42 deletions(-)

diffs (truncated from 1087 to 1000 lines):

diff -r 1d6cea817ef4 -r 283ae119d121 src/njs_generator.c
--- a/src/njs_generator.c	Thu Nov 10 17:53:36 2022 -0800
+++ b/src/njs_generator.c	Mon Nov 14 09:18:37 2022 -0800
@@ -86,6 +86,7 @@ typedef struct {
     njs_vmcode_jump_t           *jump;
     njs_variable_t              *var;
     njs_index_t                 index;
+    njs_index_t                 index_next_value;
 } njs_generator_loop_ctx_t;
 
 
@@ -176,10 +177,22 @@ static njs_int_t njs_generate_for_resolv
     njs_parser_node_t *node);
 static njs_int_t njs_generate_for_in_statement(njs_vm_t *vm,
     njs_generator_t *generator, njs_parser_node_t *node);
+static njs_int_t njs_generate_for_in_set_prop_block(njs_vm_t *vm,
+    njs_generator_t *generator, njs_parser_node_t *node);
+static njs_int_t njs_generate_for_in_name_assign(njs_vm_t *vm,
+    njs_generator_t *generator, njs_parser_node_t *node);
 static njs_int_t njs_generate_for_in_object(njs_vm_t *vm,
     njs_generator_t *generator, njs_parser_node_t *node);
+static njs_int_t njs_generate_for_in_object_wo_decl(njs_vm_t *vm,
+    njs_generator_t *generator, njs_parser_node_t *node);
+static njs_int_t njs_generate_for_in_object_left_hand_expr(njs_vm_t *vm,
+    njs_generator_t *generator, njs_parser_node_t *node);
 static njs_int_t njs_generate_for_in_body(njs_vm_t *vm,
     njs_generator_t *generator, njs_parser_node_t *node);
+static njs_int_t njs_generate_for_in_body_wo_decl(njs_vm_t *vm,
+    njs_generator_t *generator, njs_parser_node_t *node);
+static njs_int_t njs_generate_for_in_body_left_hand_expr(njs_vm_t *vm,
+    njs_generator_t *generator, njs_parser_node_t *node);
 static njs_int_t njs_generate_start_block(njs_vm_t *vm,
     njs_generator_t *generator, njs_generator_block_type_t type,
     const njs_str_t *label);
@@ -1994,6 +2007,181 @@ njs_generate_for_resolve_closure(njs_vm_
 
 
 static njs_int_t
+njs_generate_for_in_name_assign(njs_vm_t *vm, njs_generator_t *generator,
+    njs_parser_node_t *node)
+{
+    njs_int_t                 ret;
+    njs_variable_t            *var;
+    njs_parser_node_t         *foreach, *lvalue, *expr;
+    njs_vmcode_move_t         *move;
+    njs_generator_loop_ctx_t  *ctx;
+
+    ctx = generator->context;
+
+    foreach = node->left;
+    lvalue = foreach->left;
+    expr = node->right;
+
+    var = njs_variable_reference(vm, lvalue);
+
+    if (var != NULL) {
+        ctx->index_next_value = lvalue->index;
+
+    } else {
+        ctx->index_next_value = njs_generate_temp_index_get(vm, generator,
+                                                            foreach->left);
+        if (njs_slow_path(ctx->index_next_value == NJS_INDEX_ERROR)) {
+            return NJS_ERROR;
+        }
+
+        if (expr != NULL) {
+            expr->index = ctx->index_next_value;
+
+            /*
+             * lvalue and expression indexes are equal if the expression is an
+             * empty object or expression result is stored directly in variable.
+             */
+            if (lvalue->index != expr->index) {
+                njs_generate_code_move(generator, move, lvalue->index,
+                                       expr->index, expr);
+            }
+
+            ret = njs_generate_global_property_set(vm, generator, foreach->left,
+                                                   expr);
+            if (njs_slow_path(ret != NJS_OK)) {
+                return ret;
+            }
+        }
+    }
+    return njs_generator_stack_pop(vm, generator, NULL);
+}
+
+
+static njs_int_t
+njs_generate_for_in_body_wo_decl(njs_vm_t *vm, njs_generator_t *generator,
+    njs_parser_node_t *node)
+{
+    njs_int_t                 ret;
+    njs_jump_off_t            prop_offset;
+    njs_parser_node_t         *foreach, *name;
+    njs_vmcode_prop_next_t    *prop_next;
+    njs_generator_loop_ctx_t  *ctx;
+
+    ctx = generator->context;
+
+    foreach = node->left;
+    name = foreach->left->right;
+
+    /* The loop iterator. */
+
+    if (name != NULL) {
+        ret = njs_generate_for_let_update(vm, generator, foreach->left);
+        if (njs_slow_path(ret != NJS_OK)) {
+            return ret;
+        }
+    }
+
+    njs_generate_patch_block(vm, generator, generator->block,
+                             NJS_GENERATOR_CONTINUATION);
+
+    njs_code_set_jump_offset(generator, njs_vmcode_prop_foreach_t,
+                             ctx->jump_offset);
+
+    njs_generate_code(generator, njs_vmcode_prop_next_t, prop_next,
+                      NJS_VMCODE_PROPERTY_NEXT, 3, node->left->left);
+    prop_offset = njs_code_offset(generator, prop_next);
+    prop_next->retval = ctx->index_next_value;
+    prop_next->object = foreach->right->index;
+    prop_next->next = ctx->index;
+    prop_next->offset = ctx->loop_offset - prop_offset;
+
+    njs_generate_patch_block_exit(vm, generator);
+
+    /*
+     * Release object and iterator indexes: an object can be a function result
+     * or a property of another object and an iterator can be given with "let".
+     */
+    ret = njs_generate_children_indexes_release(vm, generator, foreach);
+    if (njs_slow_path(ret != NJS_OK)) {
+        return ret;
+    }
+
+    ret = njs_generate_index_release(vm, generator, ctx->index);
+    if (njs_slow_path(ret != NJS_OK)) {
+        return ret;
+    }
+
+    return njs_generator_stack_pop(vm, generator, ctx);
+}
+
+
+static njs_int_t
+njs_generate_for_in_object_wo_decl(njs_vm_t *vm, njs_generator_t *generator,
+    njs_parser_node_t *node)
+{
+    njs_int_t                  ret;
+    njs_parser_node_t          *foreach, *name;
+    njs_generator_loop_ctx_t   *ctx;
+    njs_vmcode_prop_foreach_t  *prop_foreach;
+
+    ctx = generator->context;
+
+    foreach = node->left;
+    name = foreach->left->right;
+
+    if (name != NULL) {
+        ctx->var->init = 1;
+    }
+
+    njs_generate_code(generator, njs_vmcode_prop_foreach_t, prop_foreach,
+                      NJS_VMCODE_PROPERTY_FOREACH, 2, foreach);
+    ctx->jump_offset = njs_code_offset(generator, prop_foreach);
+    prop_foreach->object = foreach->right->index;
+
+    ctx->index = njs_generate_temp_index_get(vm, generator, foreach->right);
+    if (njs_slow_path(ctx->index == NJS_INDEX_ERROR)) {
+        return NJS_ERROR;
+    }
+
+    prop_foreach->next = ctx->index;
+
+    /* The loop body. */
+
+    ctx->loop_offset = njs_code_offset(generator, generator->code_end);
+
+
+    /* 1) left. */
+
+    njs_generator_next(generator, njs_generate, foreach->left);
+
+    /* 4) loop-body-end. */
+
+    ret = njs_generator_after(vm, generator,
+                               njs_queue_first(&generator->stack), node,
+                               njs_generate_for_in_body_wo_decl, ctx, 0);
+    if (ret != NJS_OK) {
+        return ret;
+    }
+
+    /* 3) loop-body. */
+
+    ret = njs_generator_after(vm, generator,
+                               njs_queue_first(&generator->stack), node->right,
+                               njs_generate, ctx, 0);
+    if (ret != NJS_OK) {
+        return ret;
+    }
+
+    /* 2) assign value to name. */
+
+    return njs_generator_after(vm, generator,
+                               njs_queue_first(&generator->stack), node,
+                               njs_generate_for_in_name_assign, ctx, 0);
+
+}
+
+
+static njs_int_t
 njs_generate_for_in_statement(njs_vm_t *vm, njs_generator_t *generator,
     njs_parser_node_t *node)
 {
@@ -2010,40 +2198,135 @@ njs_generate_for_in_statement(njs_vm_t *
     /* The object. */
 
     foreach = node->left;
-    name = foreach->left->right;
-
-    if (name != NULL) {
-        name = name->left;
-
-        ret = njs_generate_variable_wo_dest(vm, generator, name,
+
+    if (foreach->left->token_type != NJS_TOKEN_PROPERTY) {
+        name = foreach->left->right;
+
+        if (name != NULL) {
+            name = name->left;
+
+            ret = njs_generate_variable_wo_dest(vm, generator, name,
                                             NJS_DECLARATION, &ctx.var);
-        if (njs_slow_path(ret != NJS_OK)) {
-            return NJS_ERROR;
+            if (njs_slow_path(ret != NJS_OK)) {
+                return NJS_ERROR;
+            }
+
+            foreach->left->index = name->index;
+
+            njs_generator_next(generator, njs_generate, foreach->right);
+
+            return njs_generator_after(vm, generator,
+                                       njs_queue_first(&generator->stack), node,
+                                       njs_generate_for_in_object,
+                                       &ctx, sizeof(njs_generator_loop_ctx_t));
         }
 
-        foreach->left->index = name->index;
+    } else {
+
+        /* foreach->right is object in 'in object'. */
 
         njs_generator_next(generator, njs_generate, foreach->right);
 
         return njs_generator_after(vm, generator,
                                    njs_queue_first(&generator->stack), node,
-                                   njs_generate_for_in_object,
+                                   njs_generate_for_in_object_left_hand_expr,
                                    &ctx, sizeof(njs_generator_loop_ctx_t));
-    }
-
-    njs_generator_next(generator, njs_generate, foreach->left);
-
-    ret = njs_generator_after(vm, generator,
+
+    }
+
+    njs_generator_next(generator, njs_generate, foreach->right);
+
+    return  njs_generator_after(vm, generator,
                               njs_queue_first(&generator->stack), node,
-                              njs_generate_for_in_object,
+                              njs_generate_for_in_object_wo_decl,
                               &ctx, sizeof(njs_generator_loop_ctx_t));
+}
+
+
+static njs_int_t
+njs_generate_for_in_object_left_hand_expr(njs_vm_t *vm,
+    njs_generator_t *generator, njs_parser_node_t *node)
+{
+    njs_int_t                 ret;
+    njs_parser_node_t          *foreach;
+    njs_generator_loop_ctx_t   *ctx;
+    njs_vmcode_prop_foreach_t  *prop_foreach;
+
+    ctx = generator->context;
+
+    foreach = node->left;
+
+    njs_generate_code(generator, njs_vmcode_prop_foreach_t, prop_foreach,
+                      NJS_VMCODE_PROPERTY_FOREACH, 2, foreach);
+    ctx->jump_offset = njs_code_offset(generator, prop_foreach);
+    prop_foreach->object = foreach->right->index;
+
+    ctx->index = njs_generate_temp_index_get(vm, generator, foreach->right);
+    if (njs_slow_path(ctx->index == NJS_INDEX_ERROR)) {
+        return NJS_ERROR;
+    }
+
+    ctx->index_next_value = njs_generate_temp_index_get(vm, generator,
+                                                        foreach->left);
+    if (njs_slow_path(ctx->index_next_value == NJS_INDEX_ERROR)) {
+        return NJS_ERROR;
+    }
+
+    prop_foreach->next = ctx->index;
+
+    ctx->loop_offset = njs_code_offset(generator, generator->code_end);
+
+    /* Object part calculation. */
+
+    njs_generator_next(generator, njs_generate, foreach->left->left);
+
+    /* The loop body. */
+
+    ret = njs_generator_after(vm, generator, njs_queue_first(&generator->stack),
+                              node, njs_generate_for_in_body_left_hand_expr,
+                              ctx, sizeof(njs_generator_loop_ctx_t));
     if (njs_slow_path(ret != NJS_OK)) {
         return ret;
     }
 
+    /* set-property and block. */
+
+    ret = njs_generator_after(vm, generator, njs_queue_first(&generator->stack),
+                              node, njs_generate_for_in_set_prop_block, ctx,
+                              sizeof(njs_generator_loop_ctx_t));
+    if (njs_slow_path(ret != NJS_OK)) {
+        return ret;
+    }
+
+    /* Key part calculation. */
+
     return njs_generator_after(vm, generator,
                                njs_queue_first(&generator->stack),
-                               foreach->right, njs_generate, NULL, 0);
+                               foreach->left->right, njs_generate, NULL, 0);
+}
+
+
+static njs_int_t
+njs_generate_for_in_set_prop_block(njs_vm_t *vm, njs_generator_t *generator,
+    njs_parser_node_t *node)
+{
+    njs_parser_node_t      *foreach;
+    njs_vmcode_prop_set_t  *prop_set;
+    njs_generator_loop_ctx_t   *ctx;
+
+    ctx = generator->context;
+
+    foreach = node->left;
+
+    njs_generate_code(generator, njs_vmcode_prop_set_t, prop_set,
+                      NJS_VMCODE_PROPERTY_SET, 3, foreach);
+    prop_set->object = foreach->left->left->index;
+    prop_set->property = foreach->left->right->index;
+    prop_set->value = ctx->index_next_value;
+
+    njs_generator_next(generator, njs_generate, node->right);
+
+    return NJS_OK;
 }
 
 
@@ -2089,6 +2372,54 @@ njs_generate_for_in_object(njs_vm_t *vm,
 
 
 static njs_int_t
+njs_generate_for_in_body_left_hand_expr(njs_vm_t *vm,
+    njs_generator_t *generator, njs_parser_node_t *node)
+{
+    njs_int_t                 ret;
+    njs_jump_off_t            prop_offset;
+    njs_parser_node_t         *foreach;
+    njs_vmcode_prop_next_t    *prop_next;
+    njs_generator_loop_ctx_t  *ctx;
+
+    ctx = generator->context;
+
+    foreach = node->left;
+
+    njs_generate_patch_block(vm, generator, generator->block,
+                             NJS_GENERATOR_CONTINUATION);
+
+    njs_code_set_jump_offset(generator, njs_vmcode_prop_foreach_t,
+                             ctx->jump_offset);
+
+    njs_generate_code(generator, njs_vmcode_prop_next_t, prop_next,
+                      NJS_VMCODE_PROPERTY_NEXT, 3, node->left->left);
+    prop_offset = njs_code_offset(generator, prop_next);
+    prop_next->retval = ctx->index_next_value;
+    prop_next->object = foreach->right->index;
+    prop_next->next = ctx->index;
+    prop_next->offset = ctx->loop_offset - prop_offset;
+
+    njs_generate_patch_block_exit(vm, generator);
+
+    /*
+     * Release object and iterator indexes: an object can be a function result
+     * or a property of another object and an iterator can be given with "let".
+     */
+    ret = njs_generate_children_indexes_release(vm, generator, foreach);
+    if (njs_slow_path(ret != NJS_OK)) {
+        return ret;
+    }
+
+    ret = njs_generate_index_release(vm, generator, ctx->index);
+    if (njs_slow_path(ret != NJS_OK)) {
+        return ret;
+    }
+
+    return njs_generator_stack_pop(vm, generator, ctx);
+}
+
+
+static njs_int_t
 njs_generate_for_in_body(njs_vm_t *vm, njs_generator_t *generator,
     njs_parser_node_t *node)
 {
diff -r 1d6cea817ef4 -r 283ae119d121 src/njs_lexer.c
--- a/src/njs_lexer.c	Thu Nov 10 17:53:36 2022 -0800
+++ b/src/njs_lexer.c	Mon Nov 14 09:18:37 2022 -0800
@@ -290,9 +290,13 @@ static const njs_lexer_multi_t  njs_assi
 
 njs_int_t
 njs_lexer_init(njs_vm_t *vm, njs_lexer_t *lexer, njs_str_t *file,
-    u_char *start, u_char *end, njs_uint_t runtime)
+    u_char *start, u_char *end, njs_uint_t runtime,
+    njs_int_t init_lexer_memory)
 {
-    njs_memzero(lexer, sizeof(njs_lexer_t));
+    if (init_lexer_memory) {
+        njs_memzero(lexer, sizeof(njs_lexer_t));
+
+    }
 
     lexer->file = *file;
     lexer->start = start;
@@ -304,6 +308,105 @@ njs_lexer_init(njs_vm_t *vm, njs_lexer_t
 
     njs_queue_init(&lexer->preread);
 
+    return njs_lexer_in_stack_init(lexer);
+}
+
+
+njs_int_t
+njs_lexer_in_stack_init(njs_lexer_t *lexer)
+{
+    lexer->in_stack_size = 128;
+    lexer->in_stack = njs_mp_zalloc(lexer->mem_pool, lexer->in_stack_size);
+    if (lexer->in_stack == NULL) {
+        return NJS_ERROR;
+    }
+
+    lexer->in_stack_ptr = 0;
+
+    return NJS_OK;
+}
+
+
+njs_int_t
+njs_lexer_in_stack_push(njs_lexer_t *lexer)
+{
+    u_char  *tmp;
+    size_t  size;
+
+    lexer->in_stack_ptr++;
+
+    if (lexer->in_stack_ptr < lexer->in_stack_size) {
+        lexer->in_stack[lexer->in_stack_ptr] = 0;
+        return NJS_OK;
+    }
+
+    /* Realloc in_stack, it is up to higher layer generate error if any. */
+
+    size = lexer->in_stack_size;
+    lexer->in_stack_size = size * 2;
+
+    tmp = njs_mp_alloc(lexer->mem_pool, size * 2);
+    if (tmp == NULL) {
+        return NJS_ERROR;
+    }
+
+    memcpy(tmp, lexer->in_stack, size);
+    memset(&tmp[size], 0, size);
+
+    njs_mp_free(lexer->mem_pool, lexer->in_stack);
+    lexer->in_stack = tmp;
+
+    return NJS_OK;
+}
+
+
+void
+njs_lexer_in_stack_pop(njs_lexer_t *lexer)
+{
+    /**
+     * if in_stack_ptr <= 0 do nothing, it is up to higher layer
+     * generate error.
+     */
+
+    if (lexer->in_stack_ptr > 0) {
+        lexer->in_stack_ptr--;
+    }
+}
+
+
+njs_int_t
+njs_lexer_in_fail_get(njs_lexer_t *lexer)
+{
+    return lexer->in_stack[lexer->in_stack_ptr];
+}
+
+
+void
+njs_lexer_in_fail_set(njs_lexer_t *lexer, njs_int_t flag)
+{
+    lexer->in_stack[lexer->in_stack_ptr] = flag;
+}
+
+
+njs_inline njs_int_t
+njs_lexer_in_stack(njs_lexer_t *lexer, njs_lexer_token_t  *token)
+{
+    switch (token->type) {
+    case NJS_TOKEN_OPEN_PARENTHESIS:
+    case NJS_TOKEN_OPEN_BRACKET:
+    case NJS_TOKEN_OPEN_BRACE:
+        return njs_lexer_in_stack_push(lexer);
+
+    case NJS_TOKEN_CLOSE_PARENTHESIS:
+    case NJS_TOKEN_CLOSE_BRACKET:
+    case NJS_TOKEN_CLOSE_BRACE:
+        njs_lexer_in_stack_pop(lexer);
+        break;
+
+    default:
+        break;
+    }
+
     return NJS_OK;
 }
 
@@ -329,6 +432,11 @@ njs_lexer_next_token(njs_lexer_t *lexer)
 
     njs_queue_insert_tail(&lexer->preread, &token->link);
 
+    ret = njs_lexer_in_stack(lexer, token);
+    if (njs_slow_path(ret != NJS_OK)) {
+        return NULL;
+    }
+
     return token;
 }
 
diff -r 1d6cea817ef4 -r 283ae119d121 src/njs_lexer.h
--- a/src/njs_lexer.h	Thu Nov 10 17:53:36 2022 -0800
+++ b/src/njs_lexer.h	Mon Nov 14 09:18:37 2022 -0800
@@ -267,11 +267,17 @@ typedef struct {
 
     u_char                          *start;
     u_char                          *end;
+
+#define NJS_INITIAL_IN_STACK_SIZE 128
+    uint8_t                        *in_stack;
+    njs_int_t                       in_stack_ptr;
+    njs_int_t                       in_stack_size;
 } njs_lexer_t;
 
 
 njs_int_t njs_lexer_init(njs_vm_t *vm, njs_lexer_t *lexer, njs_str_t *file,
-    u_char *start, u_char *end, njs_uint_t runtime);
+    u_char *start, u_char *end, njs_uint_t runtime,
+    njs_int_t init_lexer_memory);
 
 njs_lexer_token_t *njs_lexer_token(njs_lexer_t *lexer,
     njs_bool_t with_end_line);
@@ -279,6 +285,12 @@ njs_lexer_token_t *njs_lexer_peek_token(
     njs_lexer_token_t *current, njs_bool_t with_end_line);
 void njs_lexer_consume_token(njs_lexer_t *lexer, unsigned length);
 njs_int_t njs_lexer_make_token(njs_lexer_t *lexer, njs_lexer_token_t *token);
+njs_int_t njs_lexer_in_stack_init(njs_lexer_t *lexer);
+njs_int_t njs_lexer_in_stack_push(njs_lexer_t *lexer);
+void njs_lexer_in_stack_pop(njs_lexer_t *lexer);
+void njs_lexer_in_fail_set(njs_lexer_t *lexer, njs_int_t flag);
+njs_int_t njs_lexer_in_fail_get(njs_lexer_t *lexer);
+
 
 const njs_lexer_keyword_entry_t *njs_lexer_keyword(const u_char *key,
     size_t length);
diff -r 1d6cea817ef4 -r 283ae119d121 src/njs_parser.c
--- a/src/njs_parser.c	Thu Nov 10 17:53:36 2022 -0800
+++ b/src/njs_parser.c	Mon Nov 14 09:18:37 2022 -0800
@@ -294,6 +294,16 @@ static njs_int_t njs_parser_while_after(
 
 static njs_int_t njs_parser_iteration_statement_for(njs_parser_t *parser,
     njs_lexer_token_t *token, njs_queue_link_t *current);
+static njs_int_t njs_parser_for_left_hand_side_expression_map(
+    njs_parser_t *parser, njs_lexer_token_t *token,
+    njs_queue_link_t *current);
+static njs_int_t njs_parser_expression_continue_op(njs_parser_t *parser,
+    njs_lexer_token_t *token, njs_queue_link_t *current);
+static njs_int_t njs_parser_expression_continue_assign_comma(
+    njs_parser_t *parser, njs_lexer_token_t *token,
+    njs_queue_link_t *current);
+static njs_int_t njs_parser_for_in_statement_statement(njs_parser_t *parser,
+    njs_lexer_token_t *token, njs_queue_link_t *current);
 static njs_int_t njs_parser_iteration_statement_for_map(njs_parser_t *parser,
     njs_lexer_token_t *token, njs_queue_link_t *current);
 static njs_int_t njs_parser_for_var_binding_or_var_list(njs_parser_t *parser,
@@ -527,17 +537,9 @@ njs_parser_init(njs_vm_t *vm, njs_parser
     lexer = &parser->lexer0;
     parser->lexer = lexer;
 
-    lexer->file = *file;
-    lexer->start = start;
-    lexer->end = end;
-    lexer->line = 1;
-    lexer->keywords_hash = (runtime) ? &vm->keywords_hash
-                                     : &vm->shared->keywords_hash;
-    lexer->mem_pool = vm->mem_pool;
-
-    njs_queue_init(&lexer->preread);
-
-    return NJS_OK;
+    parser->use_lhs = 0;
+
+    return njs_lexer_init(vm, lexer, file, start, end, runtime, 0);
 }
 
 
@@ -3620,11 +3622,17 @@ njs_parser_exponentiation_expression(njs
 {
     parser->target = NULL;
 
-    njs_parser_next(parser, njs_parser_unary_expression);
-
-    /* For UpdateExpression, see njs_parser_unary_expression_after. */
-
-    return NJS_OK;
+    if (parser->use_lhs == 0) {
+        njs_parser_next(parser, njs_parser_unary_expression);
+
+        /* For UpdateExpression, see njs_parser_unary_expression_after. */
+
+        return NJS_OK;
+    } else {
+        parser->use_lhs = 0;
+
+        return njs_parser_update_expression_post(parser, token, current);
+    }
 }
 
 
@@ -3899,6 +3907,10 @@ njs_parser_relational_expression_match(n
         break;
 
     case NJS_TOKEN_IN:
+        if (njs_lexer_in_fail_get(parser->lexer)) {
+            njs_parser_syntax_error(parser, "Invalid left-hand side in for-loop");
+            return NJS_ERROR;
+        }
         operation = NJS_VMCODE_PROPERTY_IN;
         break;
 
@@ -4215,6 +4227,11 @@ njs_parser_conditional_question_mark(njs
     cond->right = node;
 
     njs_lexer_consume_token(parser->lexer, 1);
+
+    if (njs_lexer_in_stack_push(parser->lexer) != NJS_OK) {
+        return NJS_ERROR;
+    }
+
     njs_parser_next(parser, njs_parser_assignment_expression);
 
     return njs_parser_after(parser, current, cond, 1,
@@ -4232,6 +4249,8 @@ njs_parser_conditional_colon(njs_parser_
         return njs_parser_failed(parser);
     }
 
+    njs_lexer_in_stack_pop(parser->lexer);
+
     njs_lexer_consume_token(parser->lexer, 1);
 
     node = parser->target->right;
@@ -5470,6 +5489,175 @@ njs_parser_iteration_statement_for(njs_p
 
 
 static njs_int_t
+njs_parser_for_left_hand_side_expression_map(njs_parser_t *parser,
+    njs_lexer_token_t *token, njs_queue_link_t *current)
+{
+    njs_int_t          operation;
+    njs_str_t          *text;
+    njs_parser_node_t  *node;
+
+    if (parser->node == NULL) {
+        njs_lexer_in_fail_set(parser->lexer, 1);
+
+        njs_parser_next(parser, njs_parser_expression);
+
+        /*
+         * Here we pass not a node, but a token, this is important.
+         * This is necessary for correct error output.
+         */
+
+        text = njs_mp_alloc(parser->vm->mem_pool, sizeof(njs_str_t));
+        if (text == NULL) {
+            return NJS_ERROR;
+        }
+
+        *text = token->text;
+
+        return njs_parser_after(parser, current, text, 1,
+                                njs_parser_for_var_in_of_expression);
+
+    }
+
+    if (token->type != NJS_TOKEN_IN) {
+        njs_lexer_in_fail_set(parser->lexer, 1);
+
+        /* Continue parsing of expr1 in "for (expr1;[expr2];[expr3])". */
+
+        njs_parser_next(parser, njs_parser_expression_continue_op);
+
+        /*
+         * Here we pass not a node, but a token, this is important.
+         * This is necessary for correct error output.
+         */
+
+        text = njs_mp_alloc(parser->vm->mem_pool, sizeof(njs_str_t));
+        if (text == NULL) {
+            return NJS_ERROR;
+        }
+
+        *text = token->text;
+
+        return njs_parser_after(parser, current, text, 1,
+                                njs_parser_for_var_in_of_expression);
+
+    } else {
+
+        /* for-in */
+
+        if (parser->node->token_type != NJS_TOKEN_NAME &&
+            parser->node->token_type != NJS_TOKEN_PROPERTY)
+        {
+            text = (njs_str_t *) parser->target;
+
+            njs_parser_ref_error(parser, "Invalid left-hand side \"%V\" "
+                                 "in for-in statement", text);
+
+            njs_mp_free(parser->vm->mem_pool, text);
+
+            return NJS_DONE;
+        }
+
+        operation = NJS_VMCODE_PROPERTY_IN;
+
+        node = njs_parser_node_new(parser, token->type);
+        if (node == NULL) {
+            return NJS_ERROR;
+        }
+
+        node->token_line = token->line;
+        node->u.operation = operation;
+        node->left = parser->node;
+        node->left->dest = node;
+
+        njs_lexer_consume_token(parser->lexer, 1);
+
+        njs_parser_next(parser, njs_parser_expression);
+
+        return njs_parser_after(parser, current, node, 0,
+                                njs_parser_for_in_statement_statement);
+    }
+
+}
+
+
+static njs_int_t
+njs_parser_after_expr(njs_parser_t *parser,
+    njs_lexer_token_t *token, njs_queue_link_t *current)
+{
+    parser->target->right = parser->node;
+    parser->node = parser->target;
+
+    return njs_parser_stack_pop(parser);
+}
+
+
+static njs_int_t
+njs_parser_comma_expression_comma(njs_parser_t *parser,
+    njs_lexer_token_t *token, njs_queue_link_t *current)
+{
+    njs_parser_node_t  *node;
+
+    if (parser->target != NULL) {
+        parser->target->right = parser->node;
+        parser->target->right->dest = parser->target;
+        parser->node = parser->target;
+    }
+
+    if (token->type != NJS_TOKEN_COMMA) {
+        return njs_parser_stack_pop(parser);
+    }
+
+    node = njs_parser_node_new(parser, NJS_TOKEN_COMMA);
+    if (node == NULL) {
+        return NJS_ERROR;
+    }
+
+    node->token_line = token->line;
+    node->u.operation = 0;
+    node->left = parser->node;
+    node->left->dest = node;
+
+    njs_lexer_consume_token(parser->lexer, 1);
+
+    njs_parser_next(parser, njs_parser_expression);
+
+    return njs_parser_after(parser, current, node, 1, njs_parser_after_expr);
+}
+
+
+static njs_int_t
+njs_parser_expression_continue_op(njs_parser_t *parser,
+    njs_lexer_token_t *token, njs_queue_link_t *current)
+{
+    if (token->type == NJS_TOKEN_CONDITIONAL) {
+        njs_parser_next(parser, njs_parser_conditional_question_mark);
+        return njs_parser_after(parser, current, NULL, 0,
+                                njs_parser_expression_continue_assign_comma);
+    } else {
+        parser->target = NULL;
+
+        parser->use_lhs = 1;
+
+        njs_parser_next(parser, njs_parser_expression);
+
+        return njs_parser_after(parser, current, NULL, 1,
+                                njs_parser_comma_expression_comma);
+    }
+}
+
+
+static njs_int_t
+njs_parser_expression_continue_assign_comma(njs_parser_t *parser,
+    njs_lexer_token_t *token, njs_queue_link_t *current)
+{
+    njs_parser_next(parser, njs_parser_assignment_expression_after);
+
+    return njs_parser_after(parser, current, NULL, 1,
+                            njs_parser_expression_comma);
+}
+
+
+static njs_int_t
 njs_parser_iteration_statement_for_map(njs_parser_t *parser,
     njs_lexer_token_t *token, njs_queue_link_t *current)
 {
@@ -5543,12 +5731,43 @@ njs_parser_iteration_statement_for_map(n
             return ret;
         }
 
-        break;
+        goto expression_after;
+
+    case NJS_TOKEN_AWAIT:
+        njs_parser_next(parser, njs_parser_expression);
+
+        goto expression_after;
 
     default:
-        njs_parser_next(parser, njs_parser_expression);
-        break;
-    }
+        ret = njs_parser_match_arrow_expression(parser, token);
+        if (ret == NJS_OK) {
+            parser->target = NULL;
+            njs_parser_next(parser, njs_parser_expression);
+            goto expression_after;
+        } else if (ret == NJS_ERROR) {
+            return NJS_ERROR;
+        }
+
+        parser->target = NULL;
+        njs_parser_next(parser, njs_parser_left_hand_side_expression);
+
+        /*
+         * Here we pass not a node, but a token, this is important.
+         * This is necessary for correct error output.
+         */
+
+        text = njs_mp_alloc(parser->vm->mem_pool, sizeof(njs_str_t));
+        if (text == NULL) {
+            return NJS_ERROR;
+        }
+
+        *text = token->text;
+
+        return njs_parser_after(parser, current, text, 0,
+                                njs_parser_for_left_hand_side_expression_map);
+    }
+
+expression_after:
 
     /*
      * Here we pass not a node, but a token, this is important.
@@ -5617,6 +5836,8 @@ njs_parser_for_var_binding_or_var_list(n
             if (next->type != NJS_TOKEN_IN) {
                 parser->var_type = type;
 
+                njs_lexer_in_fail_set(parser->lexer, 1);
+
                 njs_parser_next(parser, njs_parser_variable_declaration_list);
                 return NJS_OK;
             }
@@ -5726,10 +5947,16 @@ njs_parser_for_var_in_of_expression(njs_
      * "of" <AssignmentExpression> ")" <Statement>
      */
 
-    if (parser->node->token_type == NJS_TOKEN_IN) {
+    if (token->type != NJS_TOKEN_SEMICOLON &&
+        token->type != NJS_TOKEN_CLOSE_PARENTHESIS &&
+        parser->node != NULL && parser->node->token_type == NJS_TOKEN_IN)
+    {
         node = parser->node->left;
 
-        if (node->token_type != NJS_TOKEN_NAME) {
+        if (node->token_type != NJS_TOKEN_NAME &&
+            node->token_type != NJS_TOKEN_PROPERTY)
+        {
+
             text = (njs_str_t *) parser->target;
 
             njs_parser_ref_error(parser, "Invalid left-hand side \"%V\" "
@@ -5752,6 +5979,8 @@ njs_parser_for_var_in_of_expression(njs_
 
     switch (token->type) {
     case NJS_TOKEN_SEMICOLON:
+        njs_lexer_in_fail_set(parser->lexer, 0);
+
         token = njs_lexer_peek_token(parser->lexer, token, 0);
         if (token == NULL) {
             return NJS_ERROR;
@@ -5820,6 +6049,36 @@ njs_parser_for_in_statement(njs_parser_t
 
 
 static njs_int_t
+njs_parser_for_in_statement_statement(njs_parser_t *parser,
+    njs_lexer_token_t *token, njs_queue_link_t *current)
+{
+    njs_parser_node_t  *forin;
+
+    if (token->type != NJS_TOKEN_CLOSE_PARENTHESIS) {
+        return njs_parser_failed(parser);
+    }
+
+    njs_lexer_consume_token(parser->lexer, 1);
+
+    parser->target->right = parser->node;
+
+    forin = njs_parser_node_new(parser, NJS_TOKEN_FOR_IN);
+    if (forin == NULL) {
+        return NJS_ERROR;
+    }
+
+    forin->left = parser->target;
+
+    parser->node = NULL;
+
+    njs_parser_next(parser, njs_parser_statement_wo_node);
+
+    return njs_parser_after(parser, current, forin, 1,
+                            njs_parser_for_in_statement_after);
+}
+
+
+static njs_int_t
 njs_parser_for_in_statement_after(njs_parser_t *parser,
     njs_lexer_token_t *token, njs_queue_link_t *current)
 {
@@ -8263,6 +8522,12 @@ njs_parser_template_string(njs_parser_t 
             if (p < lexer->end && *p == '{') {
                 p++;
                 text->length = p - text->start - 2;
+
+                ret = njs_lexer_in_stack_push(lexer);
+                if (njs_slow_path(ret != NJS_OK)) {
+                    return NJS_ERROR;
+                }
+


From v.zhestikov at f5.com  Tue Nov 15 01:32:42 2022
From: v.zhestikov at f5.com (Vadim Zhestikov)
Date: Tue, 15 Nov 2022 01:32:42 +0000
Subject: [njs] Masks definition corrected and made regularized.
Message-ID: <hg.993f19791a71.1668475962.5965299922797593991@dev.nginx>

details:   https://hg.nginx.org/njs/rev/993f19791a71
branches:  
changeset: 1998:993f19791a71
user:      Vadim Zhestikov <v.zhestikov at f5.com>
date:      Mon Nov 14 17:31:03 2022 -0800
description:
Masks definition corrected and made regularized.

diffstat:

 src/njs_scope.h |  26 +++++++++++++++++---------
 1 files changed, 17 insertions(+), 9 deletions(-)

diffs (57 lines):

diff -r 283ae119d121 -r 993f19791a71 src/njs_scope.h
--- a/src/njs_scope.h	Mon Nov 14 09:18:37 2022 -0800
+++ b/src/njs_scope.h	Mon Nov 14 17:31:03 2022 -0800
@@ -7,12 +7,20 @@
 #ifndef _NJS_SCOPE_H_INCLUDED_
 #define _NJS_SCOPE_H_INCLUDED_
 
-
+#define NJS_SCOPE_VAR_OFFSET    0
 #define NJS_SCOPE_VAR_SIZE      4
-#define NJS_SCOPE_TYPE_OFFSET   (NJS_SCOPE_VAR_SIZE + 4)
-#define NJS_SCOPE_VALUE_OFFSET  (NJS_SCOPE_TYPE_OFFSET + 1)
-#define NJS_SCOPE_VALUE_MAX     ((1 << (32 - NJS_SCOPE_VALUE_OFFSET)) - 1)
-#define NJS_SCOPE_TYPE_MASK     ((NJS_SCOPE_VALUE_MAX) << NJS_SCOPE_VAR_SIZE)
+
+#define NJS_SCOPE_TYPE_OFFSET   (NJS_SCOPE_VAR_OFFSET + NJS_SCOPE_VAR_SIZE)
+#define NJS_SCOPE_TYPE_SIZE     4
+
+#define NJS_SCOPE_VALUE_OFFSET  (NJS_SCOPE_TYPE_OFFSET + NJS_SCOPE_TYPE_SIZE)
+#define NJS_SCOPE_VALUE_SIZE    24
+
+#define NJS_SCOPE_VALUE_MASK    ((1 << NJS_SCOPE_VALUE_SIZE) - 1)
+#define NJS_SCOPE_VAR_MASK      ((1 << NJS_SCOPE_VAR_SIZE) - 1)
+#define NJS_SCOPE_TYPE_MASK     ((1 << NJS_SCOPE_TYPE_SIZE) - 1)
+
+#define NJS_SCOPE_VALUE_MAX     NJS_SCOPE_VALUE_MASK
 
 #define NJS_INDEX_NONE          ((njs_index_t) 0)
 #define NJS_INDEX_ERROR         ((njs_index_t) -1)
@@ -41,7 +49,7 @@ njs_scope_index(njs_scope_t scope, njs_i
         type = NJS_LEVEL_GLOBAL;
     }
 
-    return (index << NJS_SCOPE_VALUE_OFFSET) | (type << NJS_SCOPE_VAR_SIZE)
+    return (index << NJS_SCOPE_VALUE_OFFSET) | (type << NJS_SCOPE_TYPE_OFFSET)
             | var_type;
 }
 
@@ -49,15 +57,15 @@ njs_scope_index(njs_scope_t scope, njs_i
 njs_inline njs_variable_type_t
 njs_scope_index_var(njs_index_t index)
 {
-    return (njs_variable_type_t) (index & ~NJS_SCOPE_TYPE_MASK);
+    return (njs_variable_type_t) (index & NJS_SCOPE_VAR_MASK);
 }
 
 
 njs_inline njs_level_type_t
 njs_scope_index_type(njs_index_t index)
 {
-    return (njs_level_type_t) ((index >> NJS_SCOPE_VAR_SIZE)
-                               & ~NJS_SCOPE_TYPE_MASK);
+    return (njs_level_type_t) ((index >> NJS_SCOPE_TYPE_OFFSET)
+                               & NJS_SCOPE_TYPE_MASK);
 }
 
 


From mdounin at mdounin.ru  Tue Nov 15 02:59:14 2022
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 15 Nov 2022 05:59:14 +0300
Subject: [PATCH] SSI: ensure context of main request exists for
 subrequest using SSI
In-Reply-To: <4MwuNFVwHxE7Xfc57rVEktH4gLKCciPjR6yGjd5zowzkBnEmz9Q-8THWnUDmsbtljEbclcg_Vwp0MC36DZuPhhVaXqGGR_T7DLgiOrX9fdc=@ciel.dev>
References: <-SpypjU_KG4ByLBWOaBV_3rln7SC_v-4sAF5Z9nLtOr3EL7bvilrVTfbPXmVCw1XetJSevc6xGYxdGuqLfTfvnAYR2G1ZKcyZqTurHo0vRc=@ciel.dev>
 <Y2P1JK9cD+nMyRfh@mdounin.ru>
 <-gUKUM8ZX15I_C0Ar8PG2FqZyzZg4agx-52A6kAx8_zLiAc-eXbg5903hwRPZxT_6wrpIvVyxTUqNlx9-2JmiTtjVlCoQwp2KM6igGnoddo=@ciel.dev>
 <Y2n8edVi8dKvTsV1@mdounin.ru>
 <4MwuNFVwHxE7Xfc57rVEktH4gLKCciPjR6yGjd5zowzkBnEmz9Q-8THWnUDmsbtljEbclcg_Vwp0MC36DZuPhhVaXqGGR_T7DLgiOrX9fdc=@ciel.dev>
Message-ID: <Y3MAglSvf3L6O56g@mdounin.ru>

Hello!

On Tue, Nov 08, 2022 at 05:31:25PM +0000, Ciel via nginx-devel wrote:

> Hello! Thanks for your detailed explanation, really helps a lot!
> 
> > Another option might be to link the first subrequest's context as 
> > the main one - till the main request context is created (if at 
> > all).  Especially given that we anyway have to upgrade the main 
> > request context if the main request is seen after the first 
> > subrequest.  This will imply an additional check in the body 
> > filter along with the flag, but a trivial one.
> 
> Yes, you're right, this is a valid *option C* and I have implemented that.
> This can truly save bytes, if someone have SSI only on main request disabled,
> but use templates without variables or blocks.
> 
> However, due to the context stealing, we should mark whether the context is for
> main/subrequests, and distinguish that in body filter.  Honestly, it's kind of
> counter-intuitive for me to share one context and use (type, flag) tuple to
> check for validity.

While sharing contexts certainly requires some checking, I don't 
think there is major difference, especially with slightly more 
intuitive flag name (see below).

On the other hand, I tend to think it's somewhat wrong to keep a 
created but half-initialized context, which happens with your 
original approach.

> > This makes it possible to add "sequence points" to SSI, resolving 
> > undefined behaviour due to parallel execution of requests.
> 
> But what if the parallel subrequests do not share a common SSI parent, i.e.
> introduced concurrently by other modules?  The `wait` parameter seems have dealt
> with intra-module concurrency, but not inter-module ones.  If that truly is a
> problem, I'm not going to cover this case in this patch.  So answer at your
> interest or leave it alone.

If there are dependencies between subrequests created by some 
other module, these are to be addressed with appropriate 
synchronization within the module which created these subrequests.  
Certainly does not look like something to be addressed in this 
work.

[...]

> ********************************************************************************
> Patch C:
> 
> # HG changeset patch
> # User Ciel Zhao <i at ciel.dev>
> # Date 1667928380 -28800
> #      Wed Nov 09 01:26:20 2022 +0800
> # Node ID 4b6f88048a6104478709b5bd9a6cc6c0c343b36c
> # Parent  17d6a537fb1bb587e4de22961bf5be5f0c648fa8
> SSI: ensure context of main request exists for subrequest using SSI
> 
> As the SSI parser always uses the context from the main request for storing
> variables and blocks, that context should always exist for subrequests using
> SSI, even though the main request does not necessarily have SSI enabled.
> 
> However, `ngx_http_get_module_ctx(r->main, ...)` is getting NULL in such cases,
> resulting in the worker crashing SIGSEGV when accessing its attributes.
> 
> This patch links the first initialized context to the main request, and upgrades
> it only when main context is initialized.
> 
> diff -r 17d6a537fb1b -r 4b6f88048a61 src/http/modules/ngx_http_ssi_filter_module.c
> --- a/src/http/modules/ngx_http_ssi_filter_module.c     Wed Nov 02 13:46:16 2022 +0400
> +++ b/src/http/modules/ngx_http_ssi_filter_module.c     Wed Nov 09 01:26:20 2022 +0800
> @@ -329,7 +329,7 @@
>  static ngx_int_t
>  ngx_http_ssi_header_filter(ngx_http_request_t *r)
>  {
> -    ngx_http_ssi_ctx_t       *ctx;
> +    ngx_http_ssi_ctx_t       *ctx, *sctx;
>      ngx_http_ssi_loc_conf_t  *slcf;
>  
>      slcf = ngx_http_get_module_loc_conf(r, ngx_http_ssi_filter_module);
> @@ -346,9 +346,21 @@
>          return NGX_ERROR;
>      }
>  
> +    sctx = ngx_http_get_module_ctx(r, ngx_http_ssi_filter_module);
> +    if (sctx != NULL) {
> +        // migrate to main context if needed, see below
> +        ctx->blocks = sctx->blocks;
> +        ctx->variables = sctx->variables;
> +    }
> +
>      ngx_http_set_ctx(r, ctx, ngx_http_ssi_filter_module);
> -
> -
> +    if (ngx_http_get_module_ctx(r->main, ngx_http_ssi_filter_module) == NULL) {
> +        // set main context to current context if not exists
> +        ngx_http_set_ctx(r->main, ctx, ngx_http_ssi_filter_module);
> +    }
> +
> +
> +    ctx->is_main = (r == r->main);
>      ctx->value_len = slcf->value_len;
>      ctx->last_out = &ctx->out;
>  
> @@ -405,6 +417,7 @@
>      ctx = ngx_http_get_module_ctx(r, ngx_http_ssi_filter_module);
>  
>      if (ctx == NULL
> +        || (!ctx->is_main && r == r->main)
>          || (in == NULL
>              && ctx->buf == NULL
>              && ctx->in == NULL
> diff -r 17d6a537fb1b -r 4b6f88048a61 src/http/modules/ngx_http_ssi_filter_module.h
> --- a/src/http/modules/ngx_http_ssi_filter_module.h     Wed Nov 02 13:46:16 2022 +0400
> +++ b/src/http/modules/ngx_http_ssi_filter_module.h     Wed Nov 09 01:26:20 2022 +0800
> @@ -71,6 +71,7 @@
>      u_char                   *captures_data;
>  #endif
>  
> +    unsigned                  is_main:1;
>      unsigned                  conditional:2;
>      unsigned                  encoding:2;
>      unsigned                  block:1;
> 

Below is slightly cleaned up version of this patch, please take a 
look:

# HG changeset patch
# User Ciel Zhao <i at ciel.dev>
# Date 1667928380 -28800
#      Wed Nov 09 01:26:20 2022 +0800
# Node ID 41c67de61c7a916c01f0a1f5054d11821991ffce
# Parent  42bc158a47ecb3c2bd0396c723c307c757f2770e
SSI: handling of subrequests from other modules (ticket #1263).

As the SSI parser always uses the context from the main request for storing
variables and blocks, that context should always exist for subrequests using
SSI, even though the main request does not necessarily have SSI enabled.

However, `ngx_http_get_module_ctx(r->main, ...)` is getting NULL in such cases,
resulting in the worker crashing SIGSEGV when accessing its attributes.

This patch links the first initialized context to the main request, and
upgrades it only when main context is initialized.

diff --git a/src/http/modules/ngx_http_ssi_filter_module.c b/src/http/modules/ngx_http_ssi_filter_module.c
--- a/src/http/modules/ngx_http_ssi_filter_module.c
+++ b/src/http/modules/ngx_http_ssi_filter_module.c
@@ -329,7 +329,7 @@ static ngx_http_variable_t  ngx_http_ssi
 static ngx_int_t
 ngx_http_ssi_header_filter(ngx_http_request_t *r)
 {
-    ngx_http_ssi_ctx_t       *ctx;
+    ngx_http_ssi_ctx_t       *ctx, *mctx;
     ngx_http_ssi_loc_conf_t  *slcf;
 
     slcf = ngx_http_get_module_loc_conf(r, ngx_http_ssi_filter_module);
@@ -341,6 +341,8 @@ ngx_http_ssi_header_filter(ngx_http_requ
         return ngx_http_next_header_filter(r);
     }
 
+    mctx = ngx_http_get_module_ctx(r->main, ngx_http_ssi_filter_module);
+
     ctx = ngx_pcalloc(r->pool, sizeof(ngx_http_ssi_ctx_t));
     if (ctx == NULL) {
         return NGX_ERROR;
@@ -367,6 +369,20 @@ ngx_http_ssi_header_filter(ngx_http_requ
     r->filter_need_in_memory = 1;
 
     if (r == r->main) {
+
+        if (mctx) {
+
+            /*
+             * if there was a shared context previously used as main,
+             * copy variables and blocks
+             */
+
+            ctx->variables = mctx->variables;
+            ctx->blocks = mctx->blocks;
+
+            mctx->shared = 0;
+        }
+
         ngx_http_clear_content_length(r);
         ngx_http_clear_accept_ranges(r);
 
@@ -379,6 +395,10 @@ ngx_http_ssi_header_filter(ngx_http_requ
         } else {
             ngx_http_weak_etag(r);
         }
+
+    } else if (mctx == NULL) {
+        ngx_http_set_ctx(r->main, ctx, ngx_http_ssi_filter_module);
+        ctx->shared = 1;
     }
 
     return ngx_http_next_header_filter(r);
@@ -405,6 +425,7 @@ ngx_http_ssi_body_filter(ngx_http_reques
     ctx = ngx_http_get_module_ctx(r, ngx_http_ssi_filter_module);
 
     if (ctx == NULL
+        || (ctx->shared && r == r->main)
         || (in == NULL
             && ctx->buf == NULL
             && ctx->in == NULL
diff --git a/src/http/modules/ngx_http_ssi_filter_module.h b/src/http/modules/ngx_http_ssi_filter_module.h
--- a/src/http/modules/ngx_http_ssi_filter_module.h
+++ b/src/http/modules/ngx_http_ssi_filter_module.h
@@ -71,6 +71,7 @@ typedef struct {
     u_char                   *captures_data;
 #endif
 
+    unsigned                  shared:1;
     unsigned                  conditional:2;
     unsigned                  encoding:2;
     unsigned                  block:1;


-- 
Maxim Dounin
http://mdounin.ru/


From i at ciel.dev  Tue Nov 15 04:40:46 2022
From: i at ciel.dev (Ciel)
Date: Tue, 15 Nov 2022 04:40:46 +0000
Subject: [PATCH] SSI: ensure context of main request exists for subrequest using SSI
In-Reply-To: <Y3MAglSvf3L6O56g@mdounin.ru>
References: <-SpypjU_KG4ByLBWOaBV_3rln7SC_v-4sAF5Z9nLtOr3EL7bvilrVTfbPXmVCw1XetJSevc6xGYxdGuqLfTfvnAYR2G1ZKcyZqTurHo0vRc=@ciel.dev> <Y2P1JK9cD+nMyRfh@mdounin.ru> <-gUKUM8ZX15I_C0Ar8PG2FqZyzZg4agx-52A6kAx8_zLiAc-eXbg5903hwRPZxT_6wrpIvVyxTUqNlx9-2JmiTtjVlCoQwp2KM6igGnoddo=@ciel.dev> <Y2n8edVi8dKvTsV1@mdounin.ru> <4MwuNFVwHxE7Xfc57rVEktH4gLKCciPjR6yGjd5zowzkBnEmz9Q-8THWnUDmsbtljEbclcg_Vwp0MC36DZuPhhVaXqGGR_T7DLgiOrX9fdc=@ciel.dev> <Y3MAglSvf3L6O56g@mdounin.ru>
Message-ID: <b_h_560a_BERSQxNqny82K7ZHDki9JgvqAZoqg9HaP0q3HJGw8DIFS2Zkx6PjsLXv9UWKKfSQ7Q6l6zDNyIf-BttEJFp7jTarR1pSYEqcK4=@ciel.dev>

Hello!

Great insights. This version seems way more clear for me, and works perfectly on
my server. So this pronounced like the final fix for this ticket. (Sorry about
forgetting adding the reference again)

Since it may take significant time for the patch to rollout, I'd reachout to the
Debian package maintainers for the patching of Debian 11 bullseye, if this patch
is now accepted by NGINX.

Thanks again for your attention and patience.

Ciel


From pluknet at nginx.com  Tue Nov 15 12:28:30 2022
From: pluknet at nginx.com (Sergey Kandaurov)
Date: Tue, 15 Nov 2022 16:28:30 +0400
Subject: [PATCH 3 of 4] QUIC: support for setting QUIC methods with
 LibreSSL
In-Reply-To: <Y1HjW9CFekYwNSzp@mdounin.ru>
References: <patchbomb.1665484549@enoparse.local>
 <c0165ddcb1c6981f8e52.1665484552@enoparse.local>
 <Y1HjW9CFekYwNSzp@mdounin.ru>
Message-ID: <A1EC1C9A-F6BF-4F5C-9E6F-A44A189BC50B@nginx.com>


> On 21 Oct 2022, at 04:10, Maxim Dounin <mdounin at mdounin.ru> wrote:
> 
> Hello!
> 
> On Tue, Oct 11, 2022 at 02:35:52PM +0400, Sergey Kandaurov wrote:
> 
>> # HG changeset patch
>> # User Sergey Kandaurov <pluknet at nginx.com>
>> # Date 1665484414 -14400
>> #      Tue Oct 11 14:33:34 2022 +0400
>> # Branch quic
>> # Node ID c0165ddcb1c6981f8e5230081f03a277f62d20c3
>> # Parent  caced81ce0a9cb218ae8cdd6176c12e0614acee9
>> QUIC: support for setting QUIC methods with LibreSSL.
>> 
>> Setting QUIC methods is converted to use C99 designated initializers
>> for simplicity, as LibreSSL 3.6.0 has different SSL_QUIC_METHOD layout.
> 
> I'm somewhat sceptical about C99 designated initializers.  These 
> aren't supported by MSVC till 2012: in particular, this includes 
> all MSVC versions available via wineticks, as well as MSVC 
> versions currently used to build nginx win32 binaries.
> 
> A more portable solution might be to use run-time initialization 
> instead.

While I agree in principle, fixing build with MSVC would require
a larger work to implement support for UDP connections on win32.
Specifically, nginx-quic uses c->udp->buffer and sockets API that
needs porting to win32, see ngx_quic_send() / ngx_quic_recvmsg().

Other than that, currently the build fails in other places
as seen with MSVC 2010.  Below the patches to address this.

# HG changeset patch
# User Sergey Kandaurov <pluknet at nginx.com>
# Date 1668509694 -14400
#      Tue Nov 15 14:54:54 2022 +0400
# Branch quic
# Node ID 322f11b1272fd3a0b67fc4cc29b12b6c09e64c6f
# Parent  572a9a4a0642514fa4ef1d8c8121dfb14cadf7c0
QUIC: fixed C4706 warnings with MSVC 2010.

The fix is to avoid assignments within conditional expression.

diff --git a/src/event/quic/ngx_event_quic_transport.c b/src/event/quic/ngx_event_quic_transport.c
--- a/src/event/quic/ngx_event_quic_transport.c
+++ b/src/event/quic/ngx_event_quic_transport.c
@@ -804,11 +804,23 @@ ngx_quic_parse_frame(ngx_quic_header_t *
     case NGX_QUIC_FT_ACK:
     case NGX_QUIC_FT_ACK_ECN:
 
-        if (!((p = ngx_quic_parse_int(p, end, &f->u.ack.largest))
-              && (p = ngx_quic_parse_int(p, end, &f->u.ack.delay))
-              && (p = ngx_quic_parse_int(p, end, &f->u.ack.range_count))
-              && (p = ngx_quic_parse_int(p, end, &f->u.ack.first_range))))
-        {
+        p = ngx_quic_parse_int(p, end, &f->u.ack.largest);
+        if (p == NULL) {
+            goto error;
+        }
+
+        p = ngx_quic_parse_int(p, end, &f->u.ack.delay);
+        if (p == NULL) {
+            goto error;
+        }
+
+        p = ngx_quic_parse_int(p, end, &f->u.ack.range_count);
+        if (p == NULL) {
+            goto error;
+        }
+
+        p = ngx_quic_parse_int(p, end, &f->u.ack.first_range);
+        if (p == NULL) {
             goto error;
         }
 
@@ -818,10 +830,11 @@ ngx_quic_parse_frame(ngx_quic_header_t *
         for (i = 0; i < f->u.ack.range_count; i++) {
 
             p = ngx_quic_parse_int(p, end, &varint);
-            if (p) {
-                p = ngx_quic_parse_int(p, end, &varint);
+            if (p == NULL) {
+                goto error;
             }
 
+            p = ngx_quic_parse_int(p, end, &varint);
             if (p == NULL) {
                 goto error;
             }
@@ -833,10 +846,18 @@ ngx_quic_parse_frame(ngx_quic_header_t *
 
         if (f->type == NGX_QUIC_FT_ACK_ECN) {
 
-            if (!((p = ngx_quic_parse_int(p, end, &f->u.ack.ect0))
-                  && (p = ngx_quic_parse_int(p, end, &f->u.ack.ect1))
-                  && (p = ngx_quic_parse_int(p, end, &f->u.ack.ce))))
-            {
+            p = ngx_quic_parse_int(p, end, &f->u.ack.ect0);
+            if (p == NULL) {
+                goto error;
+            }
+
+            p = ngx_quic_parse_int(p, end, &f->u.ack.ect1);
+            if (p == NULL) {
+                goto error;
+            }
+
+            p = ngx_quic_parse_int(p, end, &f->u.ack.ce);
+            if (p == NULL) {
                 goto error;
             }
 
@@ -989,11 +1010,18 @@ ngx_quic_parse_frame(ngx_quic_header_t *
 
     case NGX_QUIC_FT_RESET_STREAM:
 
-        if (!((p = ngx_quic_parse_int(p, end, &f->u.reset_stream.id))
-              && (p = ngx_quic_parse_int(p, end, &f->u.reset_stream.error_code))
-              && (p = ngx_quic_parse_int(p, end,
-                                         &f->u.reset_stream.final_size))))
-        {
+        p = ngx_quic_parse_int(p, end, &f->u.reset_stream.id);
+        if (p == NULL) {
+            goto error;
+        }
+
+        p = ngx_quic_parse_int(p, end, &f->u.reset_stream.error_code);
+        if (p == NULL) {
+            goto error;
+        }
+
+        p = ngx_quic_parse_int(p, end, &f->u.reset_stream.final_size);
+        if (p == NULL) {
             goto error;
         }
 
# HG changeset patch
# User Sergey Kandaurov <pluknet at nginx.com>
# Date 1668510614 -14400
#      Tue Nov 15 15:10:14 2022 +0400
# Branch quic
# Node ID eff714f6d07aa595b0f4f552b76a228b5dd5758f
# Parent  322f11b1272fd3a0b67fc4cc29b12b6c09e64c6f
QUIC: moved variable declaration to fix build with MSVC 2010.

Previously, ngx_quic_hkdf_t variables used declaration with assignment
in the middle of a function, which is not supported by MSVC 2010.
Fixing this also required to rewrite the ngx_quic_hkdf_set macro
and to switch to an explicit array size.

diff --git a/src/event/quic/ngx_event_quic_protection.c b/src/event/quic/ngx_event_quic_protection.c
--- a/src/event/quic/ngx_event_quic_protection.c
+++ b/src/event/quic/ngx_event_quic_protection.c
@@ -48,12 +48,10 @@ typedef struct {
     const u_char             *label;
 } ngx_quic_hkdf_t;
 
-#define ngx_quic_hkdf_set(label, out, prk)                                    \
-    {                                                                         \
-        (out)->len, (out)->data,                                              \
-        (prk)->len, (prk)->data,                                              \
-        (sizeof(label) - 1), (u_char *)(label),                               \
-    }
+#define ngx_quic_hkdf_set(seq, _label, _out, _prk)                            \
+    (seq)->out_len = (_out)->len; (seq)->out = (_out)->data;                  \
+    (seq)->prk_len = (_prk)->len, (seq)->prk = (_prk)->data,                  \
+    (seq)->label_len = (sizeof(_label) - 1); (seq)->label = (u_char *)(_label);
 
 
 static ngx_int_t ngx_hkdf_expand(u_char *out_key, size_t out_len,
@@ -151,6 +149,7 @@ ngx_quic_keys_set_initial_secret(ngx_qui
     uint8_t             is[SHA256_DIGEST_LENGTH];
     ngx_uint_t          i;
     const EVP_MD       *digest;
+    ngx_quic_hkdf_t     seq[8];
     ngx_quic_secret_t  *client, *server;
 
     static const uint8_t salt[20] =
@@ -203,17 +202,15 @@ ngx_quic_keys_set_initial_secret(ngx_qui
     client->iv.len = NGX_QUIC_IV_LEN;
     server->iv.len = NGX_QUIC_IV_LEN;
 
-    ngx_quic_hkdf_t seq[] = {
-        /* labels per RFC 9001, 5.1. Packet Protection Keys */
-        ngx_quic_hkdf_set("tls13 client in", &client->secret, &iss),
-        ngx_quic_hkdf_set("tls13 quic key",  &client->key,    &client->secret),
-        ngx_quic_hkdf_set("tls13 quic iv",   &client->iv,     &client->secret),
-        ngx_quic_hkdf_set("tls13 quic hp",   &client->hp,     &client->secret),
-        ngx_quic_hkdf_set("tls13 server in", &server->secret, &iss),
-        ngx_quic_hkdf_set("tls13 quic key",  &server->key,    &server->secret),
-        ngx_quic_hkdf_set("tls13 quic iv",   &server->iv,     &server->secret),
-        ngx_quic_hkdf_set("tls13 quic hp",   &server->hp,     &server->secret),
-    };
+    /* labels per RFC 9001, 5.1. Packet Protection Keys */
+    ngx_quic_hkdf_set(&seq[0], "tls13 client in", &client->secret, &iss);
+    ngx_quic_hkdf_set(&seq[1], "tls13 quic key", &client->key, &client->secret);
+    ngx_quic_hkdf_set(&seq[2], "tls13 quic iv", &client->iv, &client->secret);
+    ngx_quic_hkdf_set(&seq[3], "tls13 quic hp", &client->hp, &client->secret);
+    ngx_quic_hkdf_set(&seq[4], "tls13 server in", &server->secret, &iss);
+    ngx_quic_hkdf_set(&seq[5], "tls13 quic key", &server->key, &server->secret);
+    ngx_quic_hkdf_set(&seq[6], "tls13 quic iv", &server->iv, &server->secret);
+    ngx_quic_hkdf_set(&seq[7], "tls13 quic hp", &server->hp, &server->secret);
 
     for (i = 0; i < (sizeof(seq) / sizeof(seq[0])); i++) {
         if (ngx_quic_hkdf_expand(&seq[i], digest, log) != NGX_OK) {
@@ -639,6 +636,7 @@ ngx_quic_keys_set_encryption_secret(ngx_
     ngx_int_t            key_len;
     ngx_str_t            secret_str;
     ngx_uint_t           i;
+    ngx_quic_hkdf_t      seq[3];
     ngx_quic_secret_t   *peer_secret;
     ngx_quic_ciphers_t   ciphers;
 
@@ -670,11 +668,10 @@ ngx_quic_keys_set_encryption_secret(ngx_
     secret_str.len = secret_len;
     secret_str.data = (u_char *) secret;
 
-    ngx_quic_hkdf_t seq[] = {
-        ngx_quic_hkdf_set("tls13 quic key", &peer_secret->key, &secret_str),
-        ngx_quic_hkdf_set("tls13 quic iv", &peer_secret->iv, &secret_str),
-        ngx_quic_hkdf_set("tls13 quic hp", &peer_secret->hp, &secret_str),
-    };
+    ngx_quic_hkdf_set(&seq[0], "tls13 quic key",
+                      &peer_secret->key, &secret_str);
+    ngx_quic_hkdf_set(&seq[1], "tls13 quic iv", &peer_secret->iv, &secret_str);
+    ngx_quic_hkdf_set(&seq[2], "tls13 quic hp", &peer_secret->hp, &secret_str);
 
     for (i = 0; i < (sizeof(seq) / sizeof(seq[0])); i++) {
         if (ngx_quic_hkdf_expand(&seq[i], ciphers.d, log) != NGX_OK) {
@@ -720,6 +717,7 @@ ngx_int_t
 ngx_quic_keys_update(ngx_connection_t *c, ngx_quic_keys_t *keys)
 {
     ngx_uint_t           i;
+    ngx_quic_hkdf_t      seq[6];
     ngx_quic_ciphers_t   ciphers;
     ngx_quic_secrets_t  *current, *next;
 
@@ -744,20 +742,18 @@ ngx_quic_keys_update(ngx_connection_t *c
     next->server.iv.len = NGX_QUIC_IV_LEN;
     next->server.hp = current->server.hp;
 
-    ngx_quic_hkdf_t seq[] = {
-        ngx_quic_hkdf_set("tls13 quic ku",
-                          &next->client.secret, &current->client.secret),
-        ngx_quic_hkdf_set("tls13 quic key",
-                          &next->client.key, &next->client.secret),
-        ngx_quic_hkdf_set("tls13 quic iv",
-                          &next->client.iv, &next->client.secret),
-        ngx_quic_hkdf_set("tls13 quic ku",
-                          &next->server.secret, &current->server.secret),
-        ngx_quic_hkdf_set("tls13 quic key",
-                          &next->server.key, &next->server.secret),
-        ngx_quic_hkdf_set("tls13 quic iv",
-                          &next->server.iv, &next->server.secret),
-    };
+    ngx_quic_hkdf_set(&seq[0], "tls13 quic ku",
+                      &next->client.secret, &current->client.secret);
+    ngx_quic_hkdf_set(&seq[1], "tls13 quic key",
+                      &next->client.key, &next->client.secret);
+    ngx_quic_hkdf_set(&seq[2], "tls13 quic iv",
+                      &next->client.iv, &next->client.secret);
+    ngx_quic_hkdf_set(&seq[3], "tls13 quic ku",
+                      &next->server.secret, &current->server.secret);
+    ngx_quic_hkdf_set(&seq[4], "tls13 quic key",
+                      &next->server.key, &next->server.secret);
+    ngx_quic_hkdf_set(&seq[5], "tls13 quic iv",
+                      &next->server.iv, &next->server.secret);
 
     for (i = 0; i < (sizeof(seq) / sizeof(seq[0])); i++) {
         if (ngx_quic_hkdf_expand(&seq[i], ciphers.d, c->log) != NGX_OK) {
# HG changeset patch
# User Sergey Kandaurov <pluknet at nginx.com>
# Date 1668515080 -14400
#      Tue Nov 15 16:24:40 2022 +0400
# Branch quic
# Node ID bec3d3cdc80388fb71204df2cf778c8d9f2a0321
# Parent  eff714f6d07aa595b0f4f552b76a228b5dd5758f
QUIC: avoid using C99 designated initializers.

They are not supported by MSVC till 2012.

SSL_QUIC_METHOD initialization is moved to run-time to preserve portability
among SSL library implementations, which allows to reduce its visibility.
Note using of a static storage to keep SSL_set_quic_method() reference valid.

diff --git a/src/event/quic/ngx_event_quic_protection.c b/src/event/quic/ngx_event_quic_protection.c
--- a/src/event/quic/ngx_event_quic_protection.c
+++ b/src/event/quic/ngx_event_quic_protection.c
@@ -147,6 +147,7 @@ ngx_quic_keys_set_initial_secret(ngx_qui
 {
     size_t              is_len;
     uint8_t             is[SHA256_DIGEST_LENGTH];
+    ngx_str_t           iss;
     ngx_uint_t          i;
     const EVP_MD       *digest;
     ngx_quic_hkdf_t     seq[8];
@@ -176,10 +177,8 @@ ngx_quic_keys_set_initial_secret(ngx_qui
         return NGX_ERROR;
     }
 
-    ngx_str_t iss = {
-        .data = is,
-        .len = is_len
-    };
+    iss.len = is_len;
+    iss.data = is;
 
     ngx_log_debug0(NGX_LOG_DEBUG_EVENT, log, 0,
                    "quic ngx_quic_set_initial_secret");
diff --git a/src/event/quic/ngx_event_quic_ssl.c b/src/event/quic/ngx_event_quic_ssl.c
--- a/src/event/quic/ngx_event_quic_ssl.c
+++ b/src/event/quic/ngx_event_quic_ssl.c
@@ -39,19 +39,6 @@ static int ngx_quic_send_alert(ngx_ssl_c
 static ngx_int_t ngx_quic_crypto_input(ngx_connection_t *c, ngx_chain_t *data);
 
 
-static SSL_QUIC_METHOD quic_method = {
-#if defined OPENSSL_IS_BORINGSSL || defined LIBRESSL_VERSION_NUMBER
-    .set_read_secret = ngx_quic_set_read_secret,
-    .set_write_secret = ngx_quic_set_write_secret,
-#else
-    .set_encryption_secrets = ngx_quic_set_encryption_secrets,
-#endif
-    .add_handshake_data = ngx_quic_add_handshake_data,
-    .flush_flight = ngx_quic_flush_flight,
-    .send_alert = ngx_quic_send_alert,
-};
-
-
 #if defined OPENSSL_IS_BORINGSSL || defined LIBRESSL_VERSION_NUMBER
 
 static int
@@ -533,13 +520,14 @@ ngx_quic_crypto_input(ngx_connection_t *
 ngx_int_t
 ngx_quic_init_connection(ngx_connection_t *c)
 {
-    u_char                 *p;
-    size_t                  clen;
-    ssize_t                 len;
-    ngx_str_t               dcid;
-    ngx_ssl_conn_t         *ssl_conn;
-    ngx_quic_socket_t      *qsock;
-    ngx_quic_connection_t  *qc;
+    u_char                  *p;
+    size_t                   clen;
+    ssize_t                  len;
+    ngx_str_t                dcid;
+    ngx_ssl_conn_t          *ssl_conn;
+    ngx_quic_socket_t       *qsock;
+    ngx_quic_connection_t   *qc;
+    static SSL_QUIC_METHOD   quic_method;
 
     qc = ngx_quic_get_connection(c);
 
@@ -551,6 +539,18 @@ ngx_quic_init_connection(ngx_connection_
 
     ssl_conn = c->ssl->connection;
 
+    if (!quic_method.send_alert) {
+#if defined OPENSSL_IS_BORINGSSL || defined LIBRESSL_VERSION_NUMBER
+        quic_method.set_read_secret = ngx_quic_set_read_secret;
+        quic_method.set_write_secret = ngx_quic_set_write_secret;
+#else
+        quic_method.set_encryption_secrets = ngx_quic_set_encryption_secrets;
+#endif
+        quic_method.add_handshake_data = ngx_quic_add_handshake_data;
+        quic_method.flush_flight = ngx_quic_flush_flight;
+        quic_method.send_alert = ngx_quic_send_alert;
+    }
+
     if (SSL_set_quic_method(ssl_conn, &quic_method) == 0) {
         ngx_log_error(NGX_LOG_INFO, c->log, 0,
                       "quic SSL_set_quic_method() failed");
# HG changeset patch
# User Sergey Kandaurov <pluknet at nginx.com>
# Date 1668513723 -14400
#      Tue Nov 15 16:02:03 2022 +0400
# Branch quic
# Node ID 42a9a02b402a44a48cef2b35e09e7fbec7ee8e1e
# Parent  004338233abf98def885a35ea465349d190eb19b
QUIC: fixed C4389 MSVC warning about signed/unsigned mismatch.

diff --git a/src/event/quic/ngx_event_quic_protection.c b/src/event/quic/ngx_event_quic_protection.c
--- a/src/event/quic/ngx_event_quic_protection.c
+++ b/src/event/quic/ngx_event_quic_protection.c
@@ -988,8 +988,9 @@ ngx_quic_decrypt(ngx_quic_header_t *pkt,
     u_char              *p, *sample;
     size_t               len;
     uint64_t             pn, lpn;
-    ngx_int_t            pnl, rc, key_phase;
+    ngx_int_t            pnl, rc;
     ngx_str_t            in, ad;
+    ngx_uint_t           key_phase;
     ngx_quic_secret_t   *secret;
     ngx_quic_ciphers_t   ciphers;
     uint8_t              nonce[NGX_QUIC_IV_LEN], mask[NGX_QUIC_HP_LEN];
# HG changeset patch
# User Sergey Kandaurov <pluknet at nginx.com>
# Date 1668513724 -14400
#      Tue Nov 15 16:02:04 2022 +0400
# Branch quic
# Node ID d4f3464f71f03a28d16484f861057be03ffdd72a
# Parent  42a9a02b402a44a48cef2b35e09e7fbec7ee8e1e
Added shutdown macros for win32 required for QUIC.

diff --git a/src/os/win32/ngx_socket.h b/src/os/win32/ngx_socket.h
--- a/src/os/win32/ngx_socket.h
+++ b/src/os/win32/ngx_socket.h
@@ -14,6 +14,8 @@
 
 
 #define NGX_WRITE_SHUTDOWN SD_SEND
+#define NGX_READ_SHUTDOWN  SD_RECEIVE
+#define NGX_RDWR_SHUTDOWN  SD_BOTH
 
 
 typedef SOCKET  ngx_socket_t;
# HG changeset patch
# User Sergey Kandaurov <pluknet at nginx.com>
# Date 1668513728 -14400
#      Tue Nov 15 16:02:08 2022 +0400
# Branch quic
# Node ID d760c2e49d4c4647621765fef3404c34d1aef81b
# Parent  d4f3464f71f03a28d16484f861057be03ffdd72a
QUIC: plug MSVC warning about potentially uninitialized variable.

diff --git a/src/event/quic/ngx_event_quic_tokens.c b/src/event/quic/ngx_event_quic_tokens.c
--- a/src/event/quic/ngx_event_quic_tokens.c
+++ b/src/event/quic/ngx_event_quic_tokens.c
@@ -178,6 +178,10 @@ ngx_quic_validate_token(ngx_connection_t
     u_char             addr_hash[20];
     u_char             tdec[NGX_QUIC_MAX_TOKEN_SIZE];
 
+#if NGX_SUPPRESS_WARN
+    ngx_str_null(&odcid);
+#endif
+
     /* Retry token or NEW_TOKEN in a previous connection */
 
     cipher = EVP_aes_256_cbc();
# HG changeset patch
# User Sergey Kandaurov <pluknet at nginx.com>
# Date 1668513998 -14400
#      Tue Nov 15 16:06:38 2022 +0400
# Branch quic
# Node ID 1ba7992c7bb0f801e069adb15b3378b5211a85a8
# Parent  d760c2e49d4c4647621765fef3404c34d1aef81b
QUIC: silenced C4334 MSVC warning about 32 to 64 bits convertion.

diff --git a/src/event/quic/ngx_event_quic_ack.c b/src/event/quic/ngx_event_quic_ack.c
--- a/src/event/quic/ngx_event_quic_ack.c
+++ b/src/event/quic/ngx_event_quic_ack.c
@@ -195,7 +195,7 @@ ngx_quic_rtt_sample(ngx_connection_t *c,
     } else {
         qc->min_rtt = ngx_min(qc->min_rtt, latest_rtt);
 
-        ack_delay = ack->delay * (1 << qc->ctp.ack_delay_exponent) / 1000;
+        ack_delay = ack->delay * (1ULL << qc->ctp.ack_delay_exponent) / 1000;
 
         if (c->ssl->handshaked) {
             ack_delay = ngx_min(ack_delay, qc->ctp.max_ack_delay);

-- 
Sergey Kandaurov


From xeioex at nginx.com  Wed Nov 16 02:41:57 2022
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Wed, 16 Nov 2022 02:41:57 +0000
Subject: [njs] Modules: fixed Fetch Response prototype reinitialization.
Message-ID: <hg.046a493850bc.1668566517.5965299922797593991@dev.nginx>

details:   https://hg.nginx.org/njs/rev/046a493850bc
branches:  
changeset: 1999:046a493850bc
user:      Dmitry Volyntsev <xeioex at nginx.com>
date:      Tue Nov 15 18:38:49 2022 -0800
description:
Modules: fixed Fetch Response prototype reinitialization.

Previously, since 446a1cb64a6a (0.7.7), when at least one js_import directive
was declared in both HTTP and Stream, ngx.fetch() returned inapproriate
response in Stream.

The prototype for Response object was created two times for HTTP and STREAM,
but the second initialization of global variable with the index of the
Response() prototype overwrites the first value with a different value.  This
caused a problem in Stream code which manifested itself as a `Stream flags`
object returned as a result of ngx.fetch() call instead of a Response instance.

The fix is to ensure that shared prototypes like a Response prototype
have indentical index value in all the modules.

This fixes #596 issue on Github.

diffstat:

 nginx/ngx_js.c |  13 +++++++++----
 1 files changed, 9 insertions(+), 4 deletions(-)

diffs (26 lines):

diff -r 993f19791a71 -r 046a493850bc nginx/ngx_js.c
--- a/nginx/ngx_js.c	Mon Nov 14 17:31:03 2022 -0800
+++ b/nginx/ngx_js.c	Tue Nov 15 18:38:49 2022 -0800
@@ -953,13 +953,18 @@ ngx_js_init_conf_vm(ngx_conf_t *cf, ngx_
         }
     }
 
-    rc = externals_init(cf, conf);
-    if (rc != NGX_OK) {
+    /*
+     * Core prototypes must be inited before externals_init() because
+     * the core prototype ids have to be identical in all the modules.
+     */
+
+    rc = ngx_js_core_init(conf->vm, cf->log);
+    if (njs_slow_path(rc != NJS_OK)) {
         return NGX_ERROR;
     }
 
-    rc = ngx_js_core_init(conf->vm, cf->log);
-    if (njs_slow_path(rc != NJS_OK)) {
+    rc = externals_init(cf, conf);
+    if (rc != NGX_OK) {
         return NGX_ERROR;
     }
 


From e.grebenshchikov at f5.com  Wed Nov 16 05:55:40 2022
From: e.grebenshchikov at f5.com (Eugene Grebenschikov)
Date: Tue, 15 Nov 2022 21:55:40 -0800
Subject: [PATCH] Tests: clearing of pre-existing Content-Range headers
In-Reply-To: <Y1MZVuRtH8tND0sD@mdounin.ru>
References: <20ef9719316bdee06a6f.1666364134@DHNVMN3.localdomain>
 <Y1MZVuRtH8tND0sD@mdounin.ru>
Message-ID: <20221116055540.mfxwkoldlgtvfhxj@JLM82Z2.localdomain>

Hello,

After discussion with Sergey Kandaurov, it was decided to seperate these tests
into a separate file.

On Sat, Oct 22, 2022 at 01:12:38AM +0300, Maxim Dounin wrote:
> EXTERNAL MAIL: nginx-devel-bounces at nginx.org
> 
> Hello!
> 
> On Fri, Oct 21, 2022 at 07:55:34AM -0700, Eugene Grebenschikov via nginx-devel wrote:
> 
> > # HG changeset patch
> > # User Eugene Grebenschikov <e.grebenshchikov at f5.com>
> > # Date 1666307094 25200
> > #      Thu Oct 20 16:04:54 2022 -0700
> > # Node ID 20ef9719316bdee06a6faf731bf76c1777bc98ba
> > # Parent  4f282294d8aec6fc20d6d68690bdf800629ad606
> > Tests: clearing of pre-existing Content-Range headers
> > 
> > diff -r 4f282294d8ae -r 20ef9719316b range.t
> > --- a/range.t	Wed Oct 19 10:27:19 2022 -0700
> > +++ b/range.t	Thu Oct 20 16:04:54 2022 -0700
> > @@ -21,7 +21,7 @@
> >  select STDERR; $| = 1;
> >  select STDOUT; $| = 1;
> >  
> > -my $t = Test::Nginx->new()->has(qw/http charset/)->plan(41);
> > +my $t = Test::Nginx->new()->has(qw/http charset proxy cache/)->plan(44);
> >  
> >  $t->write_file_expand('nginx.conf', <<'EOF');
> >  
> > @@ -39,6 +39,9 @@
> >          58 59; # X -> Y
> >      }
> >  
> > +    proxy_cache_path   %%TESTDIR%%/cache  levels=1:2
> > +                       keys_zone=NAME:1m;
> > +
> >      server {
> >          listen       127.0.0.1:8080;
> >          server_name  localhost;
> > @@ -55,6 +58,18 @@
> >          location /t4.html {
> >              max_ranges 0;
> >          }
> > +
> > +        location /t5.html {
> > +            proxy_pass https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1%3A8080%2Fstub&amp;data=05%7C01%7CE.Grebenshchikov%40F5.com%7C75b12c2462eb4c66e65a08dab3b17cfc%7Cdd3dfd2f6a3b40d19be0bf8327d81c50%7C0%7C0%7C638019872151212686%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=BREK%2FOZHJwWoZwZvFBV4hvZVgh1tK9Fi0QwkeiO5QB4%3D&amp;reserved=0;
> > +            proxy_cache   NAME;
> > +            proxy_cache_valid 200 1m;
> > +        }
> > +
> > +        location /stub {
> > +            add_header Content-Range stub;
> > +            add_header Accept-Ranges bytes;
> > +            return 200 "SEE-THIS";
> > +        }
> >      }
> >  }
> >  
> > @@ -154,6 +169,20 @@
> >  unlike(http_get_range('/t4.html', 'Range: bytes=0-9'), qr/ 206 /,
> >  	'max_ranges zero');
> >  
> > +# clearing of pre-existing Content-Range headers
> > +
> > +TODO: {
> > +local $TODO = 'not yet' unless $t->has_version('1.23.1');
> > +
> > +like(http_get_range('/t5.html', 'Range: bytes=0-4'),
> > +    qr/ 206 (?!.*stub)/s, 'content range cleared - range request');
> > +like(http_get_range('/t5.html', 'Range: bytes=0-2,4-'),
> > +    qr/ 206 (?!.*stub)/s, 'content range cleared - multipart');
> > +like(http_get_range('/t5.html', 'Range: bytes=1000-'),
> > +    qr/ 416 (?!.*stub)/s, 'content range cleared - not satisfable');
> > +
> > +}
> > +
> >  ###############################################################################
> >  
> >  sub http_get_range {
> 
> It might be a good idea to keep this separate from the basic range 
> filter tests.  E.g., in a separate file, similarly to 
> range_charset.t tests.  Or in a generic test file for tests with 
> proxying, that is, combined with range_charset.t.
> 
> This approach ensures that even a stripped-down nginx version, 
> such as one compiled with "--without-http_proxy_module", can be 
> properly tested by the test suite.
> 
> -- 
> Maxim Dounin
> https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmdounin.ru%2F&amp;data=05%7C01%7CE.Grebenshchikov%40F5.com%7C75b12c2462eb4c66e65a08dab3b17cfc%7Cdd3dfd2f6a3b40d19be0bf8327d81c50%7C0%7C0%7C638019872151369224%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=fLPhy9hfvrkq9KOGYN8hEKo7015dkHE1jxZ2gb6pq3k%3D&amp;reserved=0
> _______________________________________________
> nginx-devel mailing list -- nginx-devel at nginx.org
> To unsubscribe send an email to nginx-devel-leave at nginx.org
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: nginx-tests.patch
Type: text/x-diff
Size: 2483 bytes
Desc: not available
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20221115/dfe5e59e/attachment.bin>

From e.grebenshchikov at f5.com  Wed Nov 16 06:04:03 2022
From: e.grebenshchikov at f5.com (=?iso-8859-1?q?Eugene_Grebenschikov?=)
Date: Tue, 15 Nov 2022 22:04:03 -0800
Subject: [PATCH] Tests: proxy protocol v2 TLVs tests for stream module
Message-ID: <6040bfd6acbd8492730d.1668578643@DHNVMN3.localdomain>

# HG changeset patch
# User Eugene Grebenschikov <e.grebenshchikov at f5.com>
# Date 1668578494 28800
#      Tue Nov 15 22:01:34 2022 -0800
# Node ID 6040bfd6acbd8492730dee6ffe1bc9e89f70a3cb
# Parent  01fcc82a435aedd41ed3c23dbf7252371652cc7a
Tests: proxy protocol v2 TLVs tests for stream module.

diff -r 01fcc82a435a -r 6040bfd6acbd stream_proxy_protocol2_tlv.t
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/stream_proxy_protocol2_tlv.t	Tue Nov 15 22:01:34 2022 -0800
@@ -0,0 +1,154 @@
+#!/usr/bin/perl
+
+# (C) Roman Arutyunyan
+# (C) Eugene Grebenschikov
+# (C) Nginx, Inc.
+
+# Tests for variables for proxy protocol v2 TLVs.
+
+###############################################################################
+
+use warnings;
+use strict;
+
+use Test::More;
+
+BEGIN { use FindBin; chdir($FindBin::Bin); }
+
+use lib 'lib';
+use Test::Nginx;
+use Test::Nginx::Stream qw/ stream /;
+
+###############################################################################
+
+select STDERR; $| = 1;
+select STDOUT; $| = 1;
+
+my $t = Test::Nginx->new()->has(qw/stream stream_return map/)
+	->write_file_expand('nginx.conf', <<'EOF');
+
+%%TEST_GLOBALS%%
+
+daemon off;
+
+events {
+}
+
+stream {
+    %%TEST_GLOBALS_STREAM%%
+
+    map $proxy_protocol_tlv_ssl $binary_present {
+        "~\\x00" "true";
+    }
+
+    server {
+        listen 127.0.0.1:8080 proxy_protocol;
+        return "
+            alpn:$proxy_protocol_tlv_alpn
+            authority:$proxy_protocol_tlv_authority
+            crc32c:$proxy_protocol_tlv_0x3
+            unique-id:$proxy_protocol_tlv_unique_id
+            netns:$proxy_protocol_tlv_netns
+            ssl-verify:$proxy_protocol_tlv_ssl_verify
+            ssl-version:$proxy_protocol_tlv_ssl_version
+            ssl-cn:$proxy_protocol_tlv_ssl_cn
+            ssl-cipher:$proxy_protocol_tlv_ssl_cipher
+            ssl-sig-alg:$proxy_protocol_tlv_ssl_sig_alg
+            ssl-key-alg:$proxy_protocol_tlv_ssl_key_alg
+            custom:$proxy_protocol_tlv_0x000ae
+            x:$proxy_protocol_tlv_0x000e
+            ssl-binary:$binary_present";
+    }
+}
+
+EOF
+
+$t->try_run('no proxy_protocol tlv')->plan(14);
+
+###############################################################################
+
+my $tlv = pp2_create_tlv(0x1, "ALPN1");
+$tlv .= pp2_create_tlv(0x2, "localhost");
+$tlv .= pp2_create_tlv(0x3, "4321");
+$tlv .= pp2_create_tlv(0x5, "UNIQQ");
+$tlv .= pp2_create_tlv(0x30, "NETNS");
+$tlv .= pp2_create_tlv(0xae, "12345");
+my $p = pp2_create($tlv);
+
+my $r = pp_get(8080, $p);
+like($r, qr/alpn:ALPN1\x0d?$/m, 'ALPN');
+like($r, qr/authority:localhost\x0d?$/m, 'AUTHORITY');
+like($r, qr/crc32c:4321\x0d?$/m, 'CRC32C');
+like($r, qr/unique-id:UNIQQ\x0d?$/m, 'UNIQUE_ID');
+like($r, qr/netns:NETNS\x0d?$/m, 'NETNS');
+like($r, qr/custom:12345\x0d?$/m, 'custom');
+like($r, qr/x:\x0d?$/m, 'non-existent');
+
+# big proxy protocol header with TLVs
+
+TODO: {
+local $TODO = 'not yet' unless $t->has_version('1.23.3');
+
+my $sub = pp2_create_tlv(0x21, "TLSv1.2");
+$sub .= pp2_create_tlv(0x22, "example.com");
+$sub .= pp2_create_tlv(0x23, "AES256-SHA");
+$sub .= pp2_create_tlv(0x24, "SHA1");
+$sub .= pp2_create_tlv(0x25, "RSA512");
+my $ssl = pp2_create_ssl(0x01, 255, $sub);
+$tlv .= pp2_create_tlv(0x20, $ssl);
+$p = pp2_create($tlv);
+
+$r = pp_get(8080, $p);
+like($r, qr/ssl-verify:255\x0d?$/m, 'SSL_VERIFY');
+like($r, qr/ssl-version:TLSv1.2\x0d?$/m, 'SSL_VERSION');
+like($r, qr/ssl-cn:example.com\x0d?$/m, 'SSL_CN');
+like($r, qr/ssl-cipher:AES256-SHA\x0d?$/m, 'SSL_CIPHER');
+like($r, qr/ssl-sig-alg:SHA1\x0d?$/m, 'SSL_SIG_ALG');
+like($r, qr/ssl-key-alg:RSA512\x0d?$/m, 'SSL_KEY_ALG');
+like($r, qr/ssl-binary:true/, 'SSL_BINARY');
+
+}
+
+###############################################################################
+
+sub pp_get {
+	my ($port, $proxy) = @_;
+	stream(PeerPort => port($port))->io($proxy);
+}
+
+sub pp2_create {
+	my ($tlv) = @_;
+
+	my $pp2_sig = pack("N3", 0x0D0A0D0A, 0x000D0A51, 0x5549540A);
+	my $ver_cmd = pack('C', 0x21);
+	my $family = pack('C', 0x11);
+	my $packet = $pp2_sig . $ver_cmd . $family;
+
+	my $ip1 = pack('N', 0xc0000201); # 192.0.2.1
+	my $ip2 = pack('N', 0xc0000202); # 192.0.2.2
+	my $port1 = pack('n', 123);
+	my $port2 = pack('n', 5678);
+	my $addrs = $ip1 . $ip2 . $port1 . $port2;
+
+	my $len = length($addrs) + length($tlv);
+
+	$packet .= pack('n', $len) . $addrs . $tlv;
+
+	return $packet;
+}
+
+sub pp2_create_tlv {
+	my ($type, $content) = @_;
+
+	my $len = length($content);
+
+	return pack("CnA*", $type, $len, $content);
+}
+
+sub pp2_create_ssl {
+	my ($client, $verify, $content) = @_;
+
+	return pack("CNA*", $client, $verify, $content);
+}
+
+###############################################################################


From v.zhestikov at f5.com  Wed Nov 16 17:18:03 2022
From: v.zhestikov at f5.com (Vadim Zhestikov)
Date: Wed, 16 Nov 2022 17:18:03 +0000
Subject: [njs] Fixed for-in rhs expression parsing error handling.
Message-ID: <hg.0d0b16ecf796.1668619083.5965299922797593991@dev.nginx>

details:   https://hg.nginx.org/njs/rev/0d0b16ecf796
branches:  
changeset: 2000:0d0b16ecf796
user:      Vadim Zhestikov <v.zhestikov at f5.com>
date:      Wed Nov 16 09:15:39 2022 -0800
description:
Fixed for-in rhs expression parsing error handling.

This fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=53426.

diffstat:

 src/njs_parser.c         |  2 +-
 src/test/njs_unit_test.c |  3 +++
 2 files changed, 4 insertions(+), 1 deletions(-)

diffs (25 lines):

diff -r 046a493850bc -r 0d0b16ecf796 src/njs_parser.c
--- a/src/njs_parser.c	Tue Nov 15 18:38:49 2022 -0800
+++ b/src/njs_parser.c	Wed Nov 16 09:15:39 2022 -0800
@@ -6054,7 +6054,7 @@ njs_parser_for_in_statement_statement(nj
 {
     njs_parser_node_t  *forin;
 
-    if (token->type != NJS_TOKEN_CLOSE_PARENTHESIS) {
+    if (parser->ret != NJS_OK || token->type != NJS_TOKEN_CLOSE_PARENTHESIS) {
         return njs_parser_failed(parser);
     }
 
diff -r 046a493850bc -r 0d0b16ecf796 src/test/njs_unit_test.c
--- a/src/test/njs_unit_test.c	Tue Nov 15 18:38:49 2022 -0800
+++ b/src/test/njs_unit_test.c	Wed Nov 16 09:15:39 2022 -0800
@@ -2962,6 +2962,9 @@ static njs_unit_test_t  njs_test[] =
     { njs_str("for ((a,b,c) => {};;) {break}"),
       njs_str("undefined") },
 
+    { njs_str("for(I in``[)8"),
+      njs_str("SyntaxError: Unexpected token \")\" in 1") },
+
     /* switch. */
 
     { njs_str("switch"),


From v.zhestikov at f5.com  Thu Nov 17 00:11:06 2022
From: v.zhestikov at f5.com (Vadim Zhestikov)
Date: Thu, 17 Nov 2022 00:11:06 +0000
Subject: [njs] Fixed for-in unwanted detection of arrow function.
Message-ID: <hg.487780764501.1668643866.5965299922797593991@dev.nginx>

details:   https://hg.nginx.org/njs/rev/487780764501
branches:  
changeset: 2001:487780764501
user:      Vadim Zhestikov <v.zhestikov at f5.com>
date:      Wed Nov 16 16:09:47 2022 -0800
description:
Fixed for-in unwanted detection of arrow function.

diffstat:

 src/njs_parser.c         |  18 ++++++++++--------
 src/test/njs_unit_test.c |   3 +++
 2 files changed, 13 insertions(+), 8 deletions(-)

diffs (41 lines):

diff -r 0d0b16ecf796 -r 487780764501 src/njs_parser.c
--- a/src/njs_parser.c	Wed Nov 16 09:15:39 2022 -0800
+++ b/src/njs_parser.c	Wed Nov 16 16:09:47 2022 -0800
@@ -4291,14 +4291,16 @@ njs_parser_assignment_expression(njs_par
 {
     njs_int_t  ret;
 
-    ret = njs_parser_match_arrow_expression(parser, token);
-    if (ret == NJS_OK) {
-        njs_parser_next(parser, njs_parser_arrow_function);
-
-        return NJS_OK;
-
-    } else if (ret == NJS_ERROR) {
-        return NJS_ERROR;
+    if (!parser->use_lhs) {
+        ret = njs_parser_match_arrow_expression(parser, token);
+        if (ret == NJS_OK) {
+            njs_parser_next(parser, njs_parser_arrow_function);
+
+            return NJS_OK;
+
+        } else if (ret == NJS_ERROR) {
+            return NJS_ERROR;
+        }
     }
 
     njs_parser_next(parser, njs_parser_conditional_expression);
diff -r 0d0b16ecf796 -r 487780764501 src/test/njs_unit_test.c
--- a/src/test/njs_unit_test.c	Wed Nov 16 09:15:39 2022 -0800
+++ b/src/test/njs_unit_test.c	Wed Nov 16 16:09:47 2022 -0800
@@ -2965,6 +2965,9 @@ static njs_unit_test_t  njs_test[] =
     { njs_str("for(I in``[)8"),
       njs_str("SyntaxError: Unexpected token \")\" in 1") },
 
+    { njs_str("for(9A=>>"),
+      njs_str("SyntaxError: Unexpected token \"A\" in 1") },
+
     /* switch. */
 
     { njs_str("switch"),


From xeioex at nginx.com  Thu Nov 17 00:24:02 2022
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Thu, 17 Nov 2022 00:24:02 +0000
Subject: [njs] Added a target to generate pkg-config file.
Message-ID: <hg.47b4fd12767e.1668644642.5965299922797593991@dev.nginx>

details:   https://hg.nginx.org/njs/rev/47b4fd12767e
branches:  
changeset: 2002:47b4fd12767e
user:      Konstantin Pavlov <thresh at nginx.com>
date:      Mon Oct 17 16:45:53 2022 +0400
description:
Added a target to generate pkg-config file.

diffstat:

 auto/make     |  15 ++++++++++++++-
 src/njs.pc.in |   8 ++++++++
 2 files changed, 22 insertions(+), 1 deletions(-)

diffs (44 lines):

diff -r 487780764501 -r 47b4fd12767e auto/make
--- a/auto/make	Wed Nov 16 16:09:47 2022 -0800
+++ b/auto/make	Mon Oct 17 16:45:53 2022 +0400
@@ -64,7 +64,7 @@ default: njs
 NJS_LIB_INCS = $njs_incs
 NJS_LIB_OBJS = $njs_objs
 
-libnjs: $NJS_BUILD_DIR/libnjs.a
+libnjs: $NJS_BUILD_DIR/libnjs.a pc
 
 $NJS_BUILD_DIR/libnjs.a:   \\
 	$NJS_BUILD_DIR/njs_auto_config.h \\
@@ -314,6 +314,19 @@ ts_clean:
 	rm -rf $NJS_BUILD_DIR/ts
 END
 
+# pkg-config file
+cat << END >> $NJS_MAKEFILE
+
+pc: $NJS_BUILD_DIR/njs.pc
+
+$NJS_BUILD_DIR/njs.pc: $NJS_BUILD_DIR/njs_auto_config.h
+	sed -e "s, at PREFIX@,$(pwd)/$NJS_BUILD_DIR," \\
+		-e "s, at LIBDIR@,$(pwd)/$NJS_BUILD_DIR," \\
+		-e "s, at CFLAGS@,-I$(pwd)/$NJS_BUILD_DIR -I$(pwd)/src," \\
+		-e "s, at VERSION@,\$(NJS_VER)," \\
+		-e "s, at EXTRA_LIBS@,-lm $NJS_LIBS $NJS_LIB_AUX_LIBS," \\
+		src/njs.pc.in > \$@
+END
 
 # Makefile.
 
diff -r 487780764501 -r 47b4fd12767e src/njs.pc.in
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/src/njs.pc.in	Mon Oct 17 16:45:53 2022 +0400
@@ -0,0 +1,8 @@
+prefix=@PREFIX@
+libdir=@LIBDIR@
+
+Name: njs
+Description: library to embed njs scripting language
+Version: @VERSION@
+Libs: -L${libdir} -lnjs @EXTRA_LIBS@
+Cflags: @CFLAGS@


From xeioex at nginx.com  Thu Nov 17 01:48:23 2022
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Thu, 17 Nov 2022 01:48:23 +0000
Subject: [njs] Version 0.7.9.
Message-ID: <hg.5f705230a62c.1668649703.5965299922797593991@dev.nginx>

details:   https://hg.nginx.org/njs/rev/5f705230a62c
branches:  
changeset: 2003:5f705230a62c
user:      Dmitry Volyntsev <xeioex at nginx.com>
date:      Wed Nov 16 17:16:36 2022 -0800
description:
Version 0.7.9.

diffstat:

 CHANGES |  27 +++++++++++++++++++++++++++
 1 files changed, 27 insertions(+), 0 deletions(-)

diffs (34 lines):

diff -r 47b4fd12767e -r 5f705230a62c CHANGES
--- a/CHANGES	Mon Oct 17 16:45:53 2022 +0400
+++ b/CHANGES	Wed Nov 16 17:16:36 2022 -0800
@@ -1,3 +1,30 @@
+Changes with njs 0.7.9                                        17 Nov 2022
+    nginx modules:
+
+    *) Bugfix: fixed Fetch Response prototype reinitialization.
+       When at least one js_import directive was declared in both HTTP
+       and Stream, ngx.fetch() returned inapproriate response in Stream.
+       The bug was introduced in 0.7.7.
+
+    Core:
+
+    *) Bugfix: fixed String.prototype.replace(re) if re.exec() returns
+       non-flat array.
+
+    *) Bugfix: fixed Array.prototype.fill() when start object changes
+       "this".
+
+    *) Bugfix: fixed description for fs.mkdir() and fs.rmdir() methods.
+
+    *) Bugfix: fixed %TypedArray%.prototype.set(s) when s element changes
+       "this".
+
+    *) Bugfix: fixed Array.prototype.splice(s, d) when d resizes "this"
+       during evaluation.
+
+    *) Bugfix: fixed for-in loop with left and right hand side
+       expressions.
+
 Changes with njs 0.7.8                                        25 Oct 2022
 
     nginx modules:


From xeioex at nginx.com  Thu Nov 17 01:48:25 2022
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Thu, 17 Nov 2022 01:48:25 +0000
Subject: [njs] Added tag 0.7.9 for changeset 5f705230a62c
Message-ID: <hg.51bf94dc5043.1668649705.5965299922797593991@dev.nginx>

details:   https://hg.nginx.org/njs/rev/51bf94dc5043
branches:  
changeset: 2004:51bf94dc5043
user:      Dmitry Volyntsev <xeioex at nginx.com>
date:      Wed Nov 16 17:47:33 2022 -0800
description:
Added tag 0.7.9 for changeset 5f705230a62c

diffstat:

 .hgtags |  1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diffs (8 lines):

diff -r 5f705230a62c -r 51bf94dc5043 .hgtags
--- a/.hgtags	Wed Nov 16 17:16:36 2022 -0800
+++ b/.hgtags	Wed Nov 16 17:47:33 2022 -0800
@@ -54,3 +54,4 @@ 63c258c456ca018385b13f352faefdf25c7bd3bb
 461dfb0bb60e531d361319f30993f29860c19f55 0.7.6
 1592d46d9076aa832b2d37d50b90f5edfca67030 0.7.7
 3308415d7de83c3c0c7c65405bec4836685a71de 0.7.8
+5f705230a62ccae8718ef9b7a85b29dc569d8b5e 0.7.9


From pluknet at nginx.com  Thu Nov 17 17:33:13 2022
From: pluknet at nginx.com (Sergey Kandaurov)
Date: Thu, 17 Nov 2022 21:33:13 +0400
Subject: [PATCH] Tests: proxy protocol v2 TLVs tests for stream module
In-Reply-To: <6040bfd6acbd8492730d.1668578643@DHNVMN3.localdomain>
References: <6040bfd6acbd8492730d.1668578643@DHNVMN3.localdomain>
Message-ID: <A6EE6A7A-3298-4DF0-BABF-C515D2790282@nginx.com>


> On 16 Nov 2022, at 10:04, Eugene Grebenschikov via nginx-devel <nginx-devel at nginx.org> wrote:
> 
> # HG changeset patch
> # User Eugene Grebenschikov <e.grebenshchikov at f5.com>
> # Date 1668578494 28800
> #      Tue Nov 15 22:01:34 2022 -0800
> # Node ID 6040bfd6acbd8492730dee6ffe1bc9e89f70a3cb
> # Parent  01fcc82a435aedd41ed3c23dbf7252371652cc7a
> Tests: proxy protocol v2 TLVs tests for stream module.
> 
> diff -r 01fcc82a435a -r 6040bfd6acbd stream_proxy_protocol2_tlv.t
> --- /dev/null	Thu Jan 01 00:00:00 1970 +0000
> +++ b/stream_proxy_protocol2_tlv.t	Tue Nov 15 22:01:34 2022 -0800
> @@ -0,0 +1,154 @@
> +#!/usr/bin/perl
> +
> +# (C) Roman Arutyunyan
> +# (C) Eugene Grebenschikov
> +# (C) Nginx, Inc.
> +
> +# Tests for variables for proxy protocol v2 TLVs.
> +
> +###############################################################################
> +
> +use warnings;
> +use strict;
> +
> +use Test::More;
> +
> +BEGIN { use FindBin; chdir($FindBin::Bin); }
> +
> +use lib 'lib';
> +use Test::Nginx;
> +use Test::Nginx::Stream qw/ stream /;
> +
> +###############################################################################
> +
> +select STDERR; $| = 1;
> +select STDOUT; $| = 1;
> +
> +my $t = Test::Nginx->new()->has(qw/stream stream_return map/)
> +	->write_file_expand('nginx.conf', <<'EOF');
> +
> +%%TEST_GLOBALS%%
> +
> +daemon off;
> +
> +events {
> +}
> +
> +stream {
> +    %%TEST_GLOBALS_STREAM%%
> +
> +    map $proxy_protocol_tlv_ssl $binary_present {
> +        "~\\x00" "true";
> +    }
> +
> +    server {
> +        listen 127.0.0.1:8080 proxy_protocol;
> +        return "
> +            alpn:$proxy_protocol_tlv_alpn
> +            authority:$proxy_protocol_tlv_authority
> +            crc32c:$proxy_protocol_tlv_0x3
> +            unique-id:$proxy_protocol_tlv_unique_id
> +            netns:$proxy_protocol_tlv_netns
> +            ssl-verify:$proxy_protocol_tlv_ssl_verify
> +            ssl-version:$proxy_protocol_tlv_ssl_version
> +            ssl-cn:$proxy_protocol_tlv_ssl_cn
> +            ssl-cipher:$proxy_protocol_tlv_ssl_cipher
> +            ssl-sig-alg:$proxy_protocol_tlv_ssl_sig_alg
> +            ssl-key-alg:$proxy_protocol_tlv_ssl_key_alg
> +            custom:$proxy_protocol_tlv_0x000ae
> +            x:$proxy_protocol_tlv_0x000e
> +            ssl-binary:$binary_present";
> +    }
> +}
> +
> +EOF
> +
> +$t->try_run('no proxy_protocol tlv')->plan(14);
> +
> +###############################################################################
> +
> +my $tlv = pp2_create_tlv(0x1, "ALPN1");
> +$tlv .= pp2_create_tlv(0x2, "localhost");
> +$tlv .= pp2_create_tlv(0x3, "4321");
> +$tlv .= pp2_create_tlv(0x5, "UNIQQ");
> +$tlv .= pp2_create_tlv(0x30, "NETNS");
> +$tlv .= pp2_create_tlv(0xae, "12345");
> +my $p = pp2_create($tlv);
> +
> +my $r = pp_get(8080, $p);
> +like($r, qr/alpn:ALPN1\x0d?$/m, 'ALPN');
> +like($r, qr/authority:localhost\x0d?$/m, 'AUTHORITY');
> +like($r, qr/crc32c:4321\x0d?$/m, 'CRC32C');
> +like($r, qr/unique-id:UNIQQ\x0d?$/m, 'UNIQUE_ID');
> +like($r, qr/netns:NETNS\x0d?$/m, 'NETNS');
> +like($r, qr/custom:12345\x0d?$/m, 'custom');
> +like($r, qr/x:\x0d?$/m, 'non-existent');
> +
> +# big proxy protocol header with TLVs
> +
> +TODO: {
> +local $TODO = 'not yet' unless $t->has_version('1.23.3');
> +
> +my $sub = pp2_create_tlv(0x21, "TLSv1.2");
> +$sub .= pp2_create_tlv(0x22, "example.com");
> +$sub .= pp2_create_tlv(0x23, "AES256-SHA");
> +$sub .= pp2_create_tlv(0x24, "SHA1");
> +$sub .= pp2_create_tlv(0x25, "RSA512");
> +my $ssl = pp2_create_ssl(0x01, 255, $sub);
> +$tlv .= pp2_create_tlv(0x20, $ssl);
> +$p = pp2_create($tlv);
> +
> +$r = pp_get(8080, $p);
> +like($r, qr/ssl-verify:255\x0d?$/m, 'SSL_VERIFY');
> +like($r, qr/ssl-version:TLSv1.2\x0d?$/m, 'SSL_VERSION');
> +like($r, qr/ssl-cn:example.com\x0d?$/m, 'SSL_CN');
> +like($r, qr/ssl-cipher:AES256-SHA\x0d?$/m, 'SSL_CIPHER');
> +like($r, qr/ssl-sig-alg:SHA1\x0d?$/m, 'SSL_SIG_ALG');
> +like($r, qr/ssl-key-alg:RSA512\x0d?$/m, 'SSL_KEY_ALG');
> +like($r, qr/ssl-binary:true/, 'SSL_BINARY');
> +
> +}
> +
> +###############################################################################
> +
> +sub pp_get {
> +	my ($port, $proxy) = @_;
> +	stream(PeerPort => port($port))->io($proxy);
> +}
> +
> +sub pp2_create {
> +	my ($tlv) = @_;
> +
> +	my $pp2_sig = pack("N3", 0x0D0A0D0A, 0x000D0A51, 0x5549540A);
> +	my $ver_cmd = pack('C', 0x21);
> +	my $family = pack('C', 0x11);
> +	my $packet = $pp2_sig . $ver_cmd . $family;
> +
> +	my $ip1 = pack('N', 0xc0000201); # 192.0.2.1
> +	my $ip2 = pack('N', 0xc0000202); # 192.0.2.2
> +	my $port1 = pack('n', 123);
> +	my $port2 = pack('n', 5678);
> +	my $addrs = $ip1 . $ip2 . $port1 . $port2;
> +
> +	my $len = length($addrs) + length($tlv);
> +
> +	$packet .= pack('n', $len) . $addrs . $tlv;
> +
> +	return $packet;
> +}
> +
> +sub pp2_create_tlv {
> +	my ($type, $content) = @_;
> +
> +	my $len = length($content);
> +
> +	return pack("CnA*", $type, $len, $content);
> +}
> +
> +sub pp2_create_ssl {
> +	my ($client, $verify, $content) = @_;
> +
> +	return pack("CNA*", $client, $verify, $content);
> +}
> +
> +###############################################################################
> 

Looks good.

-- 
Sergey Kandaurov


From pluknet at nginx.com  Thu Nov 17 17:35:15 2022
From: pluknet at nginx.com (Sergey Kandaurov)
Date: Thu, 17 Nov 2022 21:35:15 +0400
Subject: [PATCH] Tests: clearing of pre-existing Content-Range headers
In-Reply-To: <20221116055540.mfxwkoldlgtvfhxj@JLM82Z2.localdomain>
References: <20ef9719316bdee06a6f.1666364134@DHNVMN3.localdomain>
 <Y1MZVuRtH8tND0sD@mdounin.ru>
 <20221116055540.mfxwkoldlgtvfhxj@JLM82Z2.localdomain>
Message-ID: <20221117173515.7eyiptadx2knj62a@Y9MQ9X2QVV>

On Tue, Nov 15, 2022 at 09:55:40PM -0800, Eugene Grebenschikov via nginx-devel wrote:
> Hello,
> 
> After discussion with Sergey Kandaurov, it was decided to seperate these tests
> into a separate file.
> 
[..]

> > 
> > It might be a good idea to keep this separate from the basic range 
> > filter tests.  E.g., in a separate file, similarly to 
> > range_charset.t tests.  Or in a generic test file for tests with 
> > proxying, that is, combined with range_charset.t.
> > 
> > This approach ensures that even a stripped-down nginx version, 
> > such as one compiled with "--without-http_proxy_module", can be 
> > properly tested by the test suite.
> > 
> > -- 
> > Maxim Dounin

> # HG changeset patch
> # User Eugene Grebenschikov <e.grebenshchikov at f5.com>
> # Date 1668577517 28800
> #      Tue Nov 15 21:45:17 2022 -0800
> # Node ID f9be09ba3198568642afd5107e0ab40faa3b3c11
> # Parent  01fcc82a435aedd41ed3c23dbf7252371652cc7a
> Tests: clearing of pre-existing Content-Range headers.
> 
> diff -r 01fcc82a435a -r f9be09ba3198 range_clearing.t
> --- /dev/null	Thu Jan 01 00:00:00 1970 +0000
> +++ b/range_clearing.t	Tue Nov 15 21:45:17 2022 -0800
> @@ -0,0 +1,87 @@
> +#!/usr/bin/perl
> +
> +# (C) Eugene Grebenschikov
> +# (C) Sergey Kandaurov
> +# (C) Nginx, Inc.
> +
> +# Tests for clearing of pre-existing Content-Range headers.
> +
> +###############################################################################
> +
> +use warnings;
> +use strict;
> +
> +use Test::More;
> +
> +BEGIN { use FindBin; chdir($FindBin::Bin); }
> +
> +use lib 'lib';
> +use Test::Nginx;
> +
> +###############################################################################
> +
> +select STDERR; $| = 1;
> +select STDOUT; $| = 1;
> +
> +my $t = Test::Nginx->new()->has(qw/http rewrite proxy cache/)
> +	->write_file_expand('nginx.conf', <<'EOF');
> +
> +%%TEST_GLOBALS%%
> +
> +daemon off;
> +
> +events {
> +}
> +
> +http {
> +    %%TEST_GLOBALS_HTTP%%
> +
> +    proxy_cache_path %%TESTDIR%%/cache levels=1:2 keys_zone=NAME:1m;
> +
> +    server {
> +        listen       127.0.0.1:8080;
> +        server_name  localhost;
> +
> +        location / {
> +            proxy_pass http://127.0.0.1:8080/stub;
> +            proxy_cache NAME;
> +            proxy_cache_valid 200 1m;
> +        }
> +
> +        location /stub {
> +            add_header Content-Range stub;
> +            add_header Accept-Ranges bytes;
> +            return 200 "SEE-THIS";
> +        }
> +    }
> +}
> +
> +EOF
> +
> +$t->run()->plan(3);
> +
> +###############################################################################
> +
> +local $TODO = 'not yet' unless $t->has_version('1.23.1');
> +
> +like(http_get_range('/', 'Range: bytes=0-4'),
> +	qr/ 206 (?!.*stub)/s, 'content range cleared - range request');
> +like(http_get_range('/', 'Range: bytes=0-2,4-'),
> +	qr/ 206 (?!.*stub)/s, 'content range cleared - multipart');
> +like(http_get_range('/', 'Range: bytes=1000-'),
> +	qr/ 416 (?!.*stub)/s, 'content range cleared - not satisfable');
> +
> +###############################################################################
> +
> +sub http_get_range {
> +	my ($url, $extra) = @_;
> +	return http(<<EOF);
> +GET $url HTTP/1.1
> +Host: localhost
> +Connection: close
> +$extra
> +
> +EOF
> +}
> +
> +###############################################################################

Good for me.


From claireecf at gmail.com  Thu Nov 17 18:30:46 2022
From: claireecf at gmail.com (claire liu)
Date: Thu, 17 Nov 2022 12:30:46 -0600
Subject: Nginx - quic issue
Message-ID: <CAPWKkO_OBLxx8JHPXbP_dtiQg4hNQthMyvFS+SMTz-4U6U7mFg@mail.gmail.com>

Hello,

I am using Nginx-quic in my research and having trouble getting traces from
Nginx-quic using the opentelemetry-Nginx module provided by opentelemetry,
which enables tracing of incoming requests to the server by injecting
instrumentation into the Nginx server at runtime. These are the steps
I followed
https://github.com/open-telemetry/opentelemetry-cpp-contrib/tree/main/instrumentation/otel-webserver-module.
Nginx mainline was also tested, and I got traces. Please confirm whether
Nginx-quic also supports opentelemetry. If it does support it, could you
let me know how to do it? The version I am using is 1.23.1. Thank you for
the help in advance.

Best regards,
Claire
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20221117/88b800d2/attachment.htm>

From osa at freebsd.org.ru  Thu Nov 17 18:48:12 2022
From: osa at freebsd.org.ru (Sergey A. Osokin)
Date: Thu, 17 Nov 2022 21:48:12 +0300
Subject: Nginx - quic issue
In-Reply-To: <CAPWKkO_OBLxx8JHPXbP_dtiQg4hNQthMyvFS+SMTz-4U6U7mFg@mail.gmail.com>
References: <CAPWKkO_OBLxx8JHPXbP_dtiQg4hNQthMyvFS+SMTz-4U6U7mFg@mail.gmail.com>
Message-ID: <Y3aB7ClhpB0n6oB8@FreeBSD.org.ru>

Hi Claire,

thanks for the question.

On Thu, Nov 17, 2022 at 12:30:46PM -0600, claire liu wrote:
> I am using Nginx-quic in my research and having trouble getting traces from
> Nginx-quic using the opentelemetry-Nginx module provided by opentelemetry,
> which enables tracing of incoming requests to the server by injecting
> instrumentation into the Nginx server at runtime. These are the steps
> I followed
> https://github.com/open-telemetry/opentelemetry-cpp-contrib/tree/main/instrumentation/otel-webserver-module.
> Nginx mainline was also tested, and I got traces. Please confirm whether
> Nginx-quic also supports opentelemetry. If it does support it, could you
> let me know how to do it? The version I am using is 1.23.1. Thank you for
> the help in advance.

It's better to use nginx mailling list for this case, because primary gold
of the nginx-devel mailing list is nginx development.

The opentelemery module is a third-party product, follow that, would you
mind to ask the vendor to update the functionality of the module to support
a modern protocol.

Thank you.

-- 
Sergey A. Osokin


From claireecf at gmail.com  Thu Nov 17 19:05:22 2022
From: claireecf at gmail.com (claire liu)
Date: Thu, 17 Nov 2022 13:05:22 -0600
Subject: Nginx - quic issue
In-Reply-To: <Y3aB7ClhpB0n6oB8@FreeBSD.org.ru>
References: <CAPWKkO_OBLxx8JHPXbP_dtiQg4hNQthMyvFS+SMTz-4U6U7mFg@mail.gmail.com>
 <Y3aB7ClhpB0n6oB8@FreeBSD.org.ru>
Message-ID: <CAPWKkO-nb_t24ocT8jj6R092DZevkGQqiFpY1q0ho1aKRUi0Sw@mail.gmail.com>

Hello Sergey,

Thanks for your reply. I will update this to opentelemetry. But I still
want to ensure that I only need to load the opentelemetry-webserver module
in nginx.conf to get the traces, no other steps need to be done. I am
afraid I missed something. I checked the source code of the mainline and
quic version but didn't see any code related to opentelemetry. Please let
me know if you know any material about getting nginx-quic traces, if there
is any person I can reach out to, or if there is an nginx-quic community.
All of the official articles I saw are about the mainline version, not quic
version. I want to use nginx-quic in my research, but I have
already been suck at here for several weeks. Thanks for your help.

Best regards,
Claire

On Thu, Nov 17, 2022 at 12:48 PM Sergey A. Osokin <osa at freebsd.org.ru>
wrote:

> Hi Claire,
>
> thanks for the question.
>
> On Thu, Nov 17, 2022 at 12:30:46PM -0600, claire liu wrote:
> > I am using Nginx-quic in my research and having trouble getting traces
> from
> > Nginx-quic using the opentelemetry-Nginx module provided by
> opentelemetry,
> > which enables tracing of incoming requests to the server by injecting
> > instrumentation into the Nginx server at runtime. These are the steps
> > I followed
> >
> https://github.com/open-telemetry/opentelemetry-cpp-contrib/tree/main/instrumentation/otel-webserver-module
> .
> > Nginx mainline was also tested, and I got traces. Please confirm whether
> > Nginx-quic also supports opentelemetry. If it does support it, could you
> > let me know how to do it? The version I am using is 1.23.1. Thank you for
> > the help in advance.
>
> It's better to use nginx mailling list for this case, because primary gold
> of the nginx-devel mailing list is nginx development.
>
> The opentelemery module is a third-party product, follow that, would you
> mind to ask the vendor to update the functionality of the module to support
> a modern protocol.
>
> Thank you.
>
> --
> Sergey A. Osokin
> _______________________________________________
> nginx-devel mailing list -- nginx-devel at nginx.org
> To unsubscribe send an email to nginx-devel-leave at nginx.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20221117/2dd1b539/attachment.htm>

From claireecf at gmail.com  Thu Nov 17 21:52:24 2022
From: claireecf at gmail.com (claire liu)
Date: Thu, 17 Nov 2022 15:52:24 -0600
Subject: Couldn't download old release of nginx-quic
Message-ID: <CAPWKkO822SKd2rZi5X33EGx+4ABq5B9mMJYH8mRFRKNCz_tR2w@mail.gmail.com>

Hello,
I can download the latest version of nginx-quic from
https://hg.nginx.org/nginx-quic, which is 1.23.2. I tried to download old
versions, such as 1.23.1 and 1.23.0. Although the name is nginx-quic, the
source code lacks v3, which should be maline not quic. Please let me know
how to download the old version of nginx-quic. Thanks.
[image: image.png]

Best Regards,
Claire
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20221117/ec52b851/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 101296 bytes
Desc: not available
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20221117/ec52b851/attachment.png>

From A.Bavshin at F5.com  Thu Nov 17 23:25:19 2022
From: A.Bavshin at F5.com (Aleksei Bavshin)
Date: Thu, 17 Nov 2022 23:25:19 +0000
Subject: [PATCH] HTTP: add internal_redirect directive.
Message-ID:  <CO6PR01MB7482906A6CA9DC1847971291C0069@CO6PR01MB7482.prod.exchangelabs.com>

The directive in question provides direct access to the ngx_http_internal_redirect/ngx_http_named_location APIs, allowing to simplify and optimize several real-life configurations where the modules should be executed multiple times/in a different order/with a different configuration/etc, sometimes based on a condition evaluated during the request processing.
Common examples include passing artifacts of one access module to another (auth_request -> auth_jwt) or deferring the enforcement of pre-access checks (limit_req/limit_conn) after the request is authorized.

Most of that was already achievable with the existing tools, so here is a list of reasons for implementing a dedicated internal_redirect directive:

 * It offers a small, single purpose utility which is easier to document or find in the documentation.
  We often see configurations using double proxying instead of a redirect, just because the authors weren't aware of the simpler solution and weren't able to find it in the reference; it's not obvious that `try_files` can be used this way, and that's not the main purpose of the directive. The necessary clues are documented, but without highlighting the scenarios in question.
   It also may look like an abuse of the directive with some more workarounds on top.

 * There is a certain overhead in comparable configurations with the try_files directive: blocking filesystem access, which may introduce noticeable delays[^1], a few allocations and more code in general.

 * An ability of conditional execution[^2]. It is common enough with other configuration directives that an empty value means no operation. We can use that to implement the conditions that allow skipping the redirect and proceeding to the content phase handlers.

 * We already offer an access to the internal redirect through the njs or perl APIs. The directive allows expressing similar logic as a static configuration.

 * The existing directive has some unexpected configuration pitfalls. Case in point, `alias` directive in the same location may result in a prefix being stripped from the redirect target, but only if the parameter contains variables. The behavior may catch some users unaware (#97 and all the duplicates).

[^1]: Which could be partially mitigated with `open_file_cache`. You'll need to read the source or have above average knowledge of the configuration to know that.
[^2]: This can be added to the `try_files` as well, but it will further complicate the code and make already overloaded documentation even worse.

With best regards,
Aleksei

-------------- next part --------------
A non-text attachment was scrubbed...
Name: nginx.patch
Type: application/octet-stream
Size: 6697 bytes
Desc: nginx.patch
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20221117/f16c3a6f/attachment.obj>

From mdounin at mdounin.ru  Fri Nov 18 03:58:26 2022
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Fri, 18 Nov 2022 06:58:26 +0300
Subject: [PATCH 3 of 4] QUIC: support for setting QUIC methods with
 LibreSSL
In-Reply-To: <A1EC1C9A-F6BF-4F5C-9E6F-A44A189BC50B@nginx.com>
References: <patchbomb.1665484549@enoparse.local>
 <c0165ddcb1c6981f8e52.1665484552@enoparse.local>
 <Y1HjW9CFekYwNSzp@mdounin.ru>
 <A1EC1C9A-F6BF-4F5C-9E6F-A44A189BC50B@nginx.com>
Message-ID: <Y3cC4v3bWkaUwB48@mdounin.ru>

Hello!

On Tue, Nov 15, 2022 at 04:28:30PM +0400, Sergey Kandaurov wrote:

> 
> > On 21 Oct 2022, at 04:10, Maxim Dounin <mdounin at mdounin.ru> wrote:
> > 
> > Hello!
> > 
> > On Tue, Oct 11, 2022 at 02:35:52PM +0400, Sergey Kandaurov wrote:
> > 
> >> # HG changeset patch
> >> # User Sergey Kandaurov <pluknet at nginx.com>
> >> # Date 1665484414 -14400
> >> #      Tue Oct 11 14:33:34 2022 +0400
> >> # Branch quic
> >> # Node ID c0165ddcb1c6981f8e5230081f03a277f62d20c3
> >> # Parent  caced81ce0a9cb218ae8cdd6176c12e0614acee9
> >> QUIC: support for setting QUIC methods with LibreSSL.
> >> 
> >> Setting QUIC methods is converted to use C99 designated initializers
> >> for simplicity, as LibreSSL 3.6.0 has different SSL_QUIC_METHOD layout.
> > 
> > I'm somewhat sceptical about C99 designated initializers.  These 
> > aren't supported by MSVC till 2012: in particular, this includes 
> > all MSVC versions available via wineticks, as well as MSVC 
> > versions currently used to build nginx win32 binaries.
> > 
> > A more portable solution might be to use run-time initialization 
> > instead.
> 
> While I agree in principle, fixing build with MSVC would require
> a larger work to implement support for UDP connections on win32.
> Specifically, nginx-quic uses c->udp->buffer and sockets API that
> needs porting to win32, see ngx_quic_send() / ngx_quic_recvmsg().

This shouldn't be a major problem, given the relevant APIs are 
available on Windows.

But the point is: there are no reasons to introduce additional 
issues in advance, we'll have to fix this sooner or later.

> Other than that, currently the build fails in other places
> as seen with MSVC 2010.  Below the patches to address this.
> 
> # HG changeset patch
> # User Sergey Kandaurov <pluknet at nginx.com>
> # Date 1668509694 -14400
> #      Tue Nov 15 14:54:54 2022 +0400
> # Branch quic
> # Node ID 322f11b1272fd3a0b67fc4cc29b12b6c09e64c6f
> # Parent  572a9a4a0642514fa4ef1d8c8121dfb14cadf7c0
> QUIC: fixed C4706 warnings with MSVC 2010.
> 
> The fix is to avoid assignments within conditional expression.
> 
> diff --git a/src/event/quic/ngx_event_quic_transport.c b/src/event/quic/ngx_event_quic_transport.c
> --- a/src/event/quic/ngx_event_quic_transport.c
> +++ b/src/event/quic/ngx_event_quic_transport.c
> @@ -804,11 +804,23 @@ ngx_quic_parse_frame(ngx_quic_header_t *
>      case NGX_QUIC_FT_ACK:
>      case NGX_QUIC_FT_ACK_ECN:
>  
> -        if (!((p = ngx_quic_parse_int(p, end, &f->u.ack.largest))
> -              && (p = ngx_quic_parse_int(p, end, &f->u.ack.delay))
> -              && (p = ngx_quic_parse_int(p, end, &f->u.ack.range_count))
> -              && (p = ngx_quic_parse_int(p, end, &f->u.ack.first_range))))
> -        {
> +        p = ngx_quic_parse_int(p, end, &f->u.ack.largest);
> +        if (p == NULL) {
> +            goto error;
> +        }
> +
> +        p = ngx_quic_parse_int(p, end, &f->u.ack.delay);
> +        if (p == NULL) {
> +            goto error;
> +        }
> +
> +        p = ngx_quic_parse_int(p, end, &f->u.ack.range_count);
> +        if (p == NULL) {
> +            goto error;
> +        }
> +
> +        p = ngx_quic_parse_int(p, end, &f->u.ack.first_range);
> +        if (p == NULL) {
>              goto error;
>          }
>  
> @@ -818,10 +830,11 @@ ngx_quic_parse_frame(ngx_quic_header_t *
>          for (i = 0; i < f->u.ack.range_count; i++) {
>  
>              p = ngx_quic_parse_int(p, end, &varint);
> -            if (p) {
> -                p = ngx_quic_parse_int(p, end, &varint);
> +            if (p == NULL) {
> +                goto error;
>              }
>  
> +            p = ngx_quic_parse_int(p, end, &varint);
>              if (p == NULL) {
>                  goto error;
>              }
> @@ -833,10 +846,18 @@ ngx_quic_parse_frame(ngx_quic_header_t *
>  
>          if (f->type == NGX_QUIC_FT_ACK_ECN) {
>  
> -            if (!((p = ngx_quic_parse_int(p, end, &f->u.ack.ect0))
> -                  && (p = ngx_quic_parse_int(p, end, &f->u.ack.ect1))
> -                  && (p = ngx_quic_parse_int(p, end, &f->u.ack.ce))))
> -            {
> +            p = ngx_quic_parse_int(p, end, &f->u.ack.ect0);
> +            if (p == NULL) {
> +                goto error;
> +            }
> +
> +            p = ngx_quic_parse_int(p, end, &f->u.ack.ect1);
> +            if (p == NULL) {
> +                goto error;
> +            }
> +
> +            p = ngx_quic_parse_int(p, end, &f->u.ack.ce);
> +            if (p == NULL) {
>                  goto error;
>              }
>  
> @@ -989,11 +1010,18 @@ ngx_quic_parse_frame(ngx_quic_header_t *
>  
>      case NGX_QUIC_FT_RESET_STREAM:
>  
> -        if (!((p = ngx_quic_parse_int(p, end, &f->u.reset_stream.id))
> -              && (p = ngx_quic_parse_int(p, end, &f->u.reset_stream.error_code))
> -              && (p = ngx_quic_parse_int(p, end,
> -                                         &f->u.reset_stream.final_size))))
> -        {
> +        p = ngx_quic_parse_int(p, end, &f->u.reset_stream.id);
> +        if (p == NULL) {
> +            goto error;
> +        }
> +
> +        p = ngx_quic_parse_int(p, end, &f->u.reset_stream.error_code);
> +        if (p == NULL) {
> +            goto error;
> +        }
> +
> +        p = ngx_quic_parse_int(p, end, &f->u.reset_stream.final_size);
> +        if (p == NULL) {
>              goto error;
>          }
>  

Looks good.

> # HG changeset patch
> # User Sergey Kandaurov <pluknet at nginx.com>
> # Date 1668510614 -14400
> #      Tue Nov 15 15:10:14 2022 +0400
> # Branch quic
> # Node ID eff714f6d07aa595b0f4f552b76a228b5dd5758f
> # Parent  322f11b1272fd3a0b67fc4cc29b12b6c09e64c6f
> QUIC: moved variable declaration to fix build with MSVC 2010.
> 
> Previously, ngx_quic_hkdf_t variables used declaration with assignment
> in the middle of a function, which is not supported by MSVC 2010.
> Fixing this also required to rewrite the ngx_quic_hkdf_set macro
> and to switch to an explicit array size.
> 
> diff --git a/src/event/quic/ngx_event_quic_protection.c b/src/event/quic/ngx_event_quic_protection.c
> --- a/src/event/quic/ngx_event_quic_protection.c
> +++ b/src/event/quic/ngx_event_quic_protection.c
> @@ -48,12 +48,10 @@ typedef struct {
>      const u_char             *label;
>  } ngx_quic_hkdf_t;
>  
> -#define ngx_quic_hkdf_set(label, out, prk)                                    \
> -    {                                                                         \
> -        (out)->len, (out)->data,                                              \
> -        (prk)->len, (prk)->data,                                              \
> -        (sizeof(label) - 1), (u_char *)(label),                               \
> -    }
> +#define ngx_quic_hkdf_set(seq, _label, _out, _prk)                            \
> +    (seq)->out_len = (_out)->len; (seq)->out = (_out)->data;                  \
> +    (seq)->prk_len = (_prk)->len, (seq)->prk = (_prk)->data,                  \
> +    (seq)->label_len = (sizeof(_label) - 1); (seq)->label = (u_char *)(_label);
>  
>  
>  static ngx_int_t ngx_hkdf_expand(u_char *out_key, size_t out_len,
> @@ -151,6 +149,7 @@ ngx_quic_keys_set_initial_secret(ngx_qui
>      uint8_t             is[SHA256_DIGEST_LENGTH];
>      ngx_uint_t          i;
>      const EVP_MD       *digest;
> +    ngx_quic_hkdf_t     seq[8];
>      ngx_quic_secret_t  *client, *server;
>  
>      static const uint8_t salt[20] =
> @@ -203,17 +202,15 @@ ngx_quic_keys_set_initial_secret(ngx_qui
>      client->iv.len = NGX_QUIC_IV_LEN;
>      server->iv.len = NGX_QUIC_IV_LEN;
>  
> -    ngx_quic_hkdf_t seq[] = {
> -        /* labels per RFC 9001, 5.1. Packet Protection Keys */
> -        ngx_quic_hkdf_set("tls13 client in", &client->secret, &iss),
> -        ngx_quic_hkdf_set("tls13 quic key",  &client->key,    &client->secret),
> -        ngx_quic_hkdf_set("tls13 quic iv",   &client->iv,     &client->secret),
> -        ngx_quic_hkdf_set("tls13 quic hp",   &client->hp,     &client->secret),
> -        ngx_quic_hkdf_set("tls13 server in", &server->secret, &iss),
> -        ngx_quic_hkdf_set("tls13 quic key",  &server->key,    &server->secret),
> -        ngx_quic_hkdf_set("tls13 quic iv",   &server->iv,     &server->secret),
> -        ngx_quic_hkdf_set("tls13 quic hp",   &server->hp,     &server->secret),
> -    };
> +    /* labels per RFC 9001, 5.1. Packet Protection Keys */
> +    ngx_quic_hkdf_set(&seq[0], "tls13 client in", &client->secret, &iss);
> +    ngx_quic_hkdf_set(&seq[1], "tls13 quic key", &client->key, &client->secret);
> +    ngx_quic_hkdf_set(&seq[2], "tls13 quic iv", &client->iv, &client->secret);
> +    ngx_quic_hkdf_set(&seq[3], "tls13 quic hp", &client->hp, &client->secret);
> +    ngx_quic_hkdf_set(&seq[4], "tls13 server in", &server->secret, &iss);
> +    ngx_quic_hkdf_set(&seq[5], "tls13 quic key", &server->key, &server->secret);
> +    ngx_quic_hkdf_set(&seq[6], "tls13 quic iv", &server->iv, &server->secret);
> +    ngx_quic_hkdf_set(&seq[7], "tls13 quic hp", &server->hp, &server->secret);
>  
>      for (i = 0; i < (sizeof(seq) / sizeof(seq[0])); i++) {
>          if (ngx_quic_hkdf_expand(&seq[i], digest, log) != NGX_OK) {
> @@ -639,6 +636,7 @@ ngx_quic_keys_set_encryption_secret(ngx_
>      ngx_int_t            key_len;
>      ngx_str_t            secret_str;
>      ngx_uint_t           i;
> +    ngx_quic_hkdf_t      seq[3];
>      ngx_quic_secret_t   *peer_secret;
>      ngx_quic_ciphers_t   ciphers;
>  
> @@ -670,11 +668,10 @@ ngx_quic_keys_set_encryption_secret(ngx_
>      secret_str.len = secret_len;
>      secret_str.data = (u_char *) secret;
>  
> -    ngx_quic_hkdf_t seq[] = {
> -        ngx_quic_hkdf_set("tls13 quic key", &peer_secret->key, &secret_str),
> -        ngx_quic_hkdf_set("tls13 quic iv", &peer_secret->iv, &secret_str),
> -        ngx_quic_hkdf_set("tls13 quic hp", &peer_secret->hp, &secret_str),
> -    };
> +    ngx_quic_hkdf_set(&seq[0], "tls13 quic key",
> +                      &peer_secret->key, &secret_str);
> +    ngx_quic_hkdf_set(&seq[1], "tls13 quic iv", &peer_secret->iv, &secret_str);
> +    ngx_quic_hkdf_set(&seq[2], "tls13 quic hp", &peer_secret->hp, &secret_str);
>  
>      for (i = 0; i < (sizeof(seq) / sizeof(seq[0])); i++) {
>          if (ngx_quic_hkdf_expand(&seq[i], ciphers.d, log) != NGX_OK) {
> @@ -720,6 +717,7 @@ ngx_int_t
>  ngx_quic_keys_update(ngx_connection_t *c, ngx_quic_keys_t *keys)
>  {
>      ngx_uint_t           i;
> +    ngx_quic_hkdf_t      seq[6];
>      ngx_quic_ciphers_t   ciphers;
>      ngx_quic_secrets_t  *current, *next;
>  
> @@ -744,20 +742,18 @@ ngx_quic_keys_update(ngx_connection_t *c
>      next->server.iv.len = NGX_QUIC_IV_LEN;
>      next->server.hp = current->server.hp;
>  
> -    ngx_quic_hkdf_t seq[] = {
> -        ngx_quic_hkdf_set("tls13 quic ku",
> -                          &next->client.secret, &current->client.secret),
> -        ngx_quic_hkdf_set("tls13 quic key",
> -                          &next->client.key, &next->client.secret),
> -        ngx_quic_hkdf_set("tls13 quic iv",
> -                          &next->client.iv, &next->client.secret),
> -        ngx_quic_hkdf_set("tls13 quic ku",
> -                          &next->server.secret, &current->server.secret),
> -        ngx_quic_hkdf_set("tls13 quic key",
> -                          &next->server.key, &next->server.secret),
> -        ngx_quic_hkdf_set("tls13 quic iv",
> -                          &next->server.iv, &next->server.secret),
> -    };
> +    ngx_quic_hkdf_set(&seq[0], "tls13 quic ku",
> +                      &next->client.secret, &current->client.secret);
> +    ngx_quic_hkdf_set(&seq[1], "tls13 quic key",
> +                      &next->client.key, &next->client.secret);
> +    ngx_quic_hkdf_set(&seq[2], "tls13 quic iv",
> +                      &next->client.iv, &next->client.secret);
> +    ngx_quic_hkdf_set(&seq[3], "tls13 quic ku",
> +                      &next->server.secret, &current->server.secret);
> +    ngx_quic_hkdf_set(&seq[4], "tls13 quic key",
> +                      &next->server.key, &next->server.secret);
> +    ngx_quic_hkdf_set(&seq[5], "tls13 quic iv",
> +                      &next->server.iv, &next->server.secret);
>  
>      for (i = 0; i < (sizeof(seq) / sizeof(seq[0])); i++) {
>          if (ngx_quic_hkdf_expand(&seq[i], ciphers.d, c->log) != NGX_OK) {

Looks good.

> # HG changeset patch
> # User Sergey Kandaurov <pluknet at nginx.com>
> # Date 1668515080 -14400
> #      Tue Nov 15 16:24:40 2022 +0400
> # Branch quic
> # Node ID bec3d3cdc80388fb71204df2cf778c8d9f2a0321
> # Parent  eff714f6d07aa595b0f4f552b76a228b5dd5758f
> QUIC: avoid using C99 designated initializers.
> 
> They are not supported by MSVC till 2012.
> 
> SSL_QUIC_METHOD initialization is moved to run-time to preserve portability
> among SSL library implementations, which allows to reduce its visibility.
> Note using of a static storage to keep SSL_set_quic_method() reference valid.
> 
> diff --git a/src/event/quic/ngx_event_quic_protection.c b/src/event/quic/ngx_event_quic_protection.c
> --- a/src/event/quic/ngx_event_quic_protection.c
> +++ b/src/event/quic/ngx_event_quic_protection.c
> @@ -147,6 +147,7 @@ ngx_quic_keys_set_initial_secret(ngx_qui
>  {
>      size_t              is_len;
>      uint8_t             is[SHA256_DIGEST_LENGTH];
> +    ngx_str_t           iss;
>      ngx_uint_t          i;
>      const EVP_MD       *digest;
>      ngx_quic_hkdf_t     seq[8];
> @@ -176,10 +177,8 @@ ngx_quic_keys_set_initial_secret(ngx_qui
>          return NGX_ERROR;
>      }
>  
> -    ngx_str_t iss = {
> -        .data = is,
> -        .len = is_len
> -    };
> +    iss.len = is_len;
> +    iss.data = is;
>  
>      ngx_log_debug0(NGX_LOG_DEBUG_EVENT, log, 0,
>                     "quic ngx_quic_set_initial_secret");
> diff --git a/src/event/quic/ngx_event_quic_ssl.c b/src/event/quic/ngx_event_quic_ssl.c
> --- a/src/event/quic/ngx_event_quic_ssl.c
> +++ b/src/event/quic/ngx_event_quic_ssl.c
> @@ -39,19 +39,6 @@ static int ngx_quic_send_alert(ngx_ssl_c
>  static ngx_int_t ngx_quic_crypto_input(ngx_connection_t *c, ngx_chain_t *data);
>  
>  
> -static SSL_QUIC_METHOD quic_method = {
> -#if defined OPENSSL_IS_BORINGSSL || defined LIBRESSL_VERSION_NUMBER
> -    .set_read_secret = ngx_quic_set_read_secret,
> -    .set_write_secret = ngx_quic_set_write_secret,
> -#else
> -    .set_encryption_secrets = ngx_quic_set_encryption_secrets,
> -#endif
> -    .add_handshake_data = ngx_quic_add_handshake_data,
> -    .flush_flight = ngx_quic_flush_flight,
> -    .send_alert = ngx_quic_send_alert,
> -};
> -
> -
>  #if defined OPENSSL_IS_BORINGSSL || defined LIBRESSL_VERSION_NUMBER
>  
>  static int
> @@ -533,13 +520,14 @@ ngx_quic_crypto_input(ngx_connection_t *
>  ngx_int_t
>  ngx_quic_init_connection(ngx_connection_t *c)
>  {
> -    u_char                 *p;
> -    size_t                  clen;
> -    ssize_t                 len;
> -    ngx_str_t               dcid;
> -    ngx_ssl_conn_t         *ssl_conn;
> -    ngx_quic_socket_t      *qsock;
> -    ngx_quic_connection_t  *qc;
> +    u_char                  *p;
> +    size_t                   clen;
> +    ssize_t                  len;
> +    ngx_str_t                dcid;
> +    ngx_ssl_conn_t          *ssl_conn;
> +    ngx_quic_socket_t       *qsock;
> +    ngx_quic_connection_t   *qc;
> +    static SSL_QUIC_METHOD   quic_method;
>  
>      qc = ngx_quic_get_connection(c);
>  
> @@ -551,6 +539,18 @@ ngx_quic_init_connection(ngx_connection_
>  
>      ssl_conn = c->ssl->connection;
>  
> +    if (!quic_method.send_alert) {
> +#if defined OPENSSL_IS_BORINGSSL || defined LIBRESSL_VERSION_NUMBER
> +        quic_method.set_read_secret = ngx_quic_set_read_secret;
> +        quic_method.set_write_secret = ngx_quic_set_write_secret;
> +#else
> +        quic_method.set_encryption_secrets = ngx_quic_set_encryption_secrets;
> +#endif
> +        quic_method.add_handshake_data = ngx_quic_add_handshake_data;
> +        quic_method.flush_flight = ngx_quic_flush_flight;
> +        quic_method.send_alert = ngx_quic_send_alert;
> +    }
> +
>      if (SSL_set_quic_method(ssl_conn, &quic_method) == 0) {
>          ngx_log_error(NGX_LOG_INFO, c->log, 0,
>                        "quic SSL_set_quic_method() failed");

Looks good.

> # HG changeset patch
> # User Sergey Kandaurov <pluknet at nginx.com>
> # Date 1668513723 -14400
> #      Tue Nov 15 16:02:03 2022 +0400
> # Branch quic
> # Node ID 42a9a02b402a44a48cef2b35e09e7fbec7ee8e1e
> # Parent  004338233abf98def885a35ea465349d190eb19b
> QUIC: fixed C4389 MSVC warning about signed/unsigned mismatch.
> 
> diff --git a/src/event/quic/ngx_event_quic_protection.c b/src/event/quic/ngx_event_quic_protection.c
> --- a/src/event/quic/ngx_event_quic_protection.c
> +++ b/src/event/quic/ngx_event_quic_protection.c
> @@ -988,8 +988,9 @@ ngx_quic_decrypt(ngx_quic_header_t *pkt,
>      u_char              *p, *sample;
>      size_t               len;
>      uint64_t             pn, lpn;
> -    ngx_int_t            pnl, rc, key_phase;
> +    ngx_int_t            pnl, rc;
>      ngx_str_t            in, ad;
> +    ngx_uint_t           key_phase;
>      ngx_quic_secret_t   *secret;
>      ngx_quic_ciphers_t   ciphers;
>      uint8_t              nonce[NGX_QUIC_IV_LEN], mask[NGX_QUIC_HP_LEN];

Looks good.

> # HG changeset patch
> # User Sergey Kandaurov <pluknet at nginx.com>
> # Date 1668513724 -14400
> #      Tue Nov 15 16:02:04 2022 +0400
> # Branch quic
> # Node ID d4f3464f71f03a28d16484f861057be03ffdd72a
> # Parent  42a9a02b402a44a48cef2b35e09e7fbec7ee8e1e
> Added shutdown macros for win32 required for QUIC.
> 
> diff --git a/src/os/win32/ngx_socket.h b/src/os/win32/ngx_socket.h
> --- a/src/os/win32/ngx_socket.h
> +++ b/src/os/win32/ngx_socket.h
> @@ -14,6 +14,8 @@
>  
>  
>  #define NGX_WRITE_SHUTDOWN SD_SEND
> +#define NGX_READ_SHUTDOWN  SD_RECEIVE
> +#define NGX_RDWR_SHUTDOWN  SD_BOTH
>  
>  
>  typedef SOCKET  ngx_socket_t;

Looks good.

> # HG changeset patch
> # User Sergey Kandaurov <pluknet at nginx.com>
> # Date 1668513728 -14400
> #      Tue Nov 15 16:02:08 2022 +0400
> # Branch quic
> # Node ID d760c2e49d4c4647621765fef3404c34d1aef81b
> # Parent  d4f3464f71f03a28d16484f861057be03ffdd72a
> QUIC: plug MSVC warning about potentially uninitialized variable.
> 
> diff --git a/src/event/quic/ngx_event_quic_tokens.c b/src/event/quic/ngx_event_quic_tokens.c
> --- a/src/event/quic/ngx_event_quic_tokens.c
> +++ b/src/event/quic/ngx_event_quic_tokens.c
> @@ -178,6 +178,10 @@ ngx_quic_validate_token(ngx_connection_t
>      u_char             addr_hash[20];
>      u_char             tdec[NGX_QUIC_MAX_TOKEN_SIZE];
>  
> +#if NGX_SUPPRESS_WARN
> +    ngx_str_null(&odcid);
> +#endif
> +
>      /* Retry token or NEW_TOKEN in a previous connection */
>  
>      cipher = EVP_aes_256_cbc();

Looks good.

> # HG changeset patch
> # User Sergey Kandaurov <pluknet at nginx.com>
> # Date 1668513998 -14400
> #      Tue Nov 15 16:06:38 2022 +0400
> # Branch quic
> # Node ID 1ba7992c7bb0f801e069adb15b3378b5211a85a8
> # Parent  d760c2e49d4c4647621765fef3404c34d1aef81b
> QUIC: silenced C4334 MSVC warning about 32 to 64 bits convertion.

converSion?

> 
> diff --git a/src/event/quic/ngx_event_quic_ack.c b/src/event/quic/ngx_event_quic_ack.c
> --- a/src/event/quic/ngx_event_quic_ack.c
> +++ b/src/event/quic/ngx_event_quic_ack.c
> @@ -195,7 +195,7 @@ ngx_quic_rtt_sample(ngx_connection_t *c,
>      } else {
>          qc->min_rtt = ngx_min(qc->min_rtt, latest_rtt);
>  
> -        ack_delay = ack->delay * (1 << qc->ctp.ack_delay_exponent) / 1000;
> +        ack_delay = ack->delay * (1ULL << qc->ctp.ack_delay_exponent) / 1000;
>  
>          if (c->ssl->handshaked) {
>              ack_delay = ngx_min(ack_delay, qc->ctp.max_ack_delay);

Why "1 << ..." at all?
Shouldn't it be

diff -r d9ef59e283e3 src/event/quic/ngx_event_quic_ack.c
--- a/src/event/quic/ngx_event_quic_ack.c	Tue Nov 15 16:02:08 2022 +0400
+++ b/src/event/quic/ngx_event_quic_ack.c	Fri Nov 18 06:55:05 2022 +0300
@@ -195,7 +195,7 @@ ngx_quic_rtt_sample(ngx_connection_t *c,
     } else {
         qc->min_rtt = ngx_min(qc->min_rtt, latest_rtt);
 
-        ack_delay = ack->delay * (1 << qc->ctp.ack_delay_exponent) / 1000;
+        ack_delay = (ack->delay << qc->ctp.ack_delay_exponent) / 1000;
 
         if (c->ssl->handshaked) {
             ack_delay = ngx_min(ack_delay, qc->ctp.max_ack_delay);


instead?

-- 
Maxim Dounin
http://mdounin.ru/


From mdounin at mdounin.ru  Fri Nov 18 04:10:43 2022
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Fri, 18 Nov 2022 07:10:43 +0300
Subject: Couldn't download old release of nginx-quic
In-Reply-To: <CAPWKkO822SKd2rZi5X33EGx+4ABq5B9mMJYH8mRFRKNCz_tR2w@mail.gmail.com>
References: <CAPWKkO822SKd2rZi5X33EGx+4ABq5B9mMJYH8mRFRKNCz_tR2w@mail.gmail.com>
Message-ID: <Y3cFw++Sr9+40h7Z@mdounin.ru>

Hello!

On Thu, Nov 17, 2022 at 03:52:24PM -0600, claire liu wrote:

> Hello,
> I can download the latest version of nginx-quic from
> https://hg.nginx.org/nginx-quic, which is 1.23.2. I tried to download old
> versions, such as 1.23.1 and 1.23.0. Although the name is nginx-quic, the
> source code lacks v3, which should be maline not quic. Please let me know
> how to download the old version of nginx-quic. Thanks.

You may want to learn how to work with branches in Mercurial, see 
https://www.mercurial-scm.org/ for Mercurial docs.  In particular, 
the graph log, which is also available on the web here:

https://hg.nginx.org/nginx-quic/graph/tip

might be helpful to better understand repository structure and 
available branches.

Note well that there are no releases on the quic branch.  It's 
experimental branch without any releases, all releases happen on 
the mainline and stable branches.  The best you can get are some 
revisions on the quic branch after merging with the mainline 
branch following a release.

-- 
Maxim Dounin
http://mdounin.ru/


From mdounin at mdounin.ru  Fri Nov 18 04:22:29 2022
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Fri, 18 Nov 2022 07:22:29 +0300
Subject: [PATCH] HTTP: add internal_redirect directive.
In-Reply-To: <CO6PR01MB7482906A6CA9DC1847971291C0069@CO6PR01MB7482.prod.exchangelabs.com>
References: <CO6PR01MB7482906A6CA9DC1847971291C0069@CO6PR01MB7482.prod.exchangelabs.com>
Message-ID: <Y3cIhd0li6k6Bzdh@mdounin.ru>

Hello!

On Thu, Nov 17, 2022 at 11:25:19PM +0000, Aleksei Bavshin via nginx-devel wrote:

> The directive in question provides direct access to the 
> ngx_http_internal_redirect/ngx_http_named_location APIs, 
> allowing to simplify and optimize several real-life 
> configurations where the modules should be executed multiple 
> times/in a different order/with a different configuration/etc, 
> sometimes based on a condition evaluated during the request 
> processing.

No, thanks.

Discussions about adding the "goto" directive date back to at 
least 2009.  Igor's and my position on this are summarized here:

https://mailman.nginx.org/pipermail/nginx-ru/2017-April/059736.html

Hope this helps.

-- 
Maxim Dounin
http://mdounin.ru/


From xeioex at nginx.com  Fri Nov 18 05:04:32 2022
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Fri, 18 Nov 2022 05:04:32 +0000
Subject: [njs] Version bump.
Message-ID: <hg.b3eeac9ee9f4.1668747872.5965299922797593991@dev.nginx>

details:   https://hg.nginx.org/njs/rev/b3eeac9ee9f4
branches:  
changeset: 2005:b3eeac9ee9f4
user:      Dmitry Volyntsev <xeioex at nginx.com>
date:      Wed Nov 16 18:47:36 2022 -0800
description:
Version bump.

diffstat:

 src/njs.h |  4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diffs (14 lines):

diff -r 51bf94dc5043 -r b3eeac9ee9f4 src/njs.h
--- a/src/njs.h	Wed Nov 16 17:47:33 2022 -0800
+++ b/src/njs.h	Wed Nov 16 18:47:36 2022 -0800
@@ -11,8 +11,8 @@
 
 #include <njs_auto_config.h>
 
-#define NJS_VERSION                 "0.7.9"
-#define NJS_VERSION_NUMBER          0x000709
+#define NJS_VERSION                 "0.7.10"
+#define NJS_VERSION_NUMBER          0x00070a
 
 
 #include <unistd.h>                 /* STDOUT_FILENO, STDERR_FILENO */


From xeioex at nginx.com  Fri Nov 18 05:04:34 2022
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Fri, 18 Nov 2022 05:04:34 +0000
Subject: [njs] Shell: added signal support in CLI.
Message-ID: <hg.8163612eb950.1668747874.5965299922797593991@dev.nginx>

details:   https://hg.nginx.org/njs/rev/8163612eb950
branches:  
changeset: 2006:8163612eb950
user:      Dmitry Volyntsev <xeioex at nginx.com>
date:      Thu Nov 17 20:38:23 2022 -0800
description:
Shell: added signal support in CLI.

diffstat:

 src/njs_shell.c     |  184 +++++++++++++++++++++++++++++++++++++++++++--------
 test/shell_test.exp |   32 ++++----
 2 files changed, 169 insertions(+), 47 deletions(-)

diffs (414 lines):

diff -r b3eeac9ee9f4 -r 8163612eb950 src/njs_shell.c
--- a/src/njs_shell.c	Wed Nov 16 18:47:36 2022 -0800
+++ b/src/njs_shell.c	Thu Nov 17 20:38:23 2022 -0800
@@ -10,6 +10,8 @@
 #if (!defined NJS_FUZZER_TARGET && defined NJS_HAVE_READLINE)
 
 #include <locale.h>
+#include <signal.h>
+#include <sys/select.h>
 #if (NJS_HAVE_EDITLINE)
 #include <editline/readline.h>
 #elif (NJS_HAVE_EDIT_READLINE)
@@ -24,13 +26,15 @@
 #endif
 
 
+typedef void (*njs_console_output_pt)(njs_vm_t *vm, njs_int_t ret);
+
+
 typedef struct {
     uint8_t                 disassemble;
     uint8_t                 denormals;
     uint8_t                 interactive;
     uint8_t                 module;
     uint8_t                 quiet;
-    uint8_t                 silent;
     uint8_t                 sandbox;
     uint8_t                 safe;
     uint8_t                 version;
@@ -90,10 +94,12 @@ typedef struct {
 
 
 static njs_int_t njs_console_init(njs_vm_t *vm, njs_console_t *console);
+static void njs_console_output(njs_vm_t *vm, njs_int_t ret);
 static njs_int_t njs_externals_init(njs_vm_t *vm);
 static njs_vm_t *njs_create_vm(njs_opts_t *opts, njs_vm_opt_t *vm_options);
-static njs_int_t njs_process_script(njs_vm_t *vm, njs_opts_t *opts,
-    void *runtime, const njs_str_t *script);
+static void njs_process_output(njs_vm_t *vm, njs_int_t ret);
+static njs_int_t njs_process_script(njs_vm_t *vm, void *runtime,
+    const njs_str_t *script);
 
 #ifndef NJS_FUZZER_TARGET
 
@@ -243,6 +249,15 @@ main(int argc, char **argv)
     njs_str_t     command;
     njs_vm_opt_t  vm_options;
 
+    static uintptr_t uptr[] = {
+        (uintptr_t) njs_console_output,
+    };
+
+    static njs_vm_meta_t metas = {
+        .size = njs_nitems(uptr),
+        .values = uptr
+    };
+
     njs_memzero(&opts, sizeof(njs_opts_t));
     opts.interactive = 1;
 
@@ -287,6 +302,7 @@ main(int argc, char **argv)
 
     vm_options.ops = &njs_console_ops;
     vm_options.addons = njs_console_addon_modules;
+    vm_options.metas = &metas;
     vm_options.external = &njs_console;
     vm_options.argv = opts.argv;
     vm_options.argc = opts.argc;
@@ -311,7 +327,7 @@ main(int argc, char **argv)
         if (vm != NULL) {
             command.start = (u_char *) opts.command;
             command.length = njs_strlen(opts.command);
-            ret = njs_process_script(vm, &opts, vm_options.external, &command);
+            ret = njs_process_script(vm, vm->external, &command);
             njs_vm_destroy(vm);
         }
 
@@ -659,7 +675,7 @@ njs_process_file(njs_opts_t *opts, njs_v
         }
     }
 
-    ret = njs_process_script(vm, opts, vm_options->external, &script);
+    ret = njs_process_script(vm, vm_options->external, &script);
     if (ret != NJS_OK) {
         ret = NJS_ERROR;
         goto done;
@@ -696,18 +712,26 @@ LLVMFuzzerTestOneInput(const uint8_t* da
     njs_str_t     script;
     njs_vm_opt_t  vm_options;
 
+    static uintptr_t uptr[] = {
+        (uintptr_t) NULL,
+    };
+
+    static njs_vm_meta_t metas = {
+        .size = njs_nitems(uptr),
+        .values = uptr
+    };
+
     if (size == 0) {
         return 0;
     }
 
     njs_memzero(&opts, sizeof(njs_opts_t));
 
-    opts.silent = 1;
-
     njs_vm_opt_init(&vm_options);
 
     vm_options.init = 1;
     vm_options.backtrace = 0;
+    vm_options.metas = &metas;
     vm_options.ops = &njs_console_ops;
 
     vm = njs_create_vm(&opts, &vm_options);
@@ -716,7 +740,7 @@ LLVMFuzzerTestOneInput(const uint8_t* da
         script.length = size;
         script.start = (u_char *) data;
 
-        (void) njs_process_script(vm, &opts, NULL, &script);
+        (void) njs_process_script(vm, NULL, &script);
         njs_vm_destroy(vm);
     }
 
@@ -853,14 +877,10 @@ njs_create_vm(njs_opts_t *opts, njs_vm_o
 
 
 static void
-njs_output(njs_opts_t *opts, njs_vm_t *vm, njs_int_t ret)
+njs_console_output(njs_vm_t *vm, njs_int_t ret)
 {
     njs_str_t  out;
 
-    if (opts->silent) {
-        return;
-    }
-
     if (ret == NJS_OK) {
         if (njs_vm_retval_dump(vm, &out, 1) != NJS_OK) {
             njs_stderror("Shell:failed to get retval from VM\n");
@@ -917,8 +937,7 @@ njs_process_events(void *runtime)
 
 
 static njs_int_t
-njs_process_script(njs_vm_t *vm, njs_opts_t *opts, void *runtime,
-    const njs_str_t *script)
+njs_process_script(njs_vm_t *vm, void *runtime, const njs_str_t *script)
 {
     u_char     *start, *end;
     njs_int_t  ret;
@@ -938,14 +957,15 @@ njs_process_script(njs_vm_t *vm, njs_opt
         }
     }
 
-    njs_output(opts, vm, ret);
+    njs_process_output(vm, ret);
 
-    if (!opts->interactive && ret == NJS_ERROR) {
+    if (!vm->options.interactive && ret == NJS_ERROR) {
         return NJS_ERROR;
     }
 
     for ( ;; ) {
         if (!njs_vm_pending(vm) && !njs_vm_unhandled_rejection(vm)) {
+            ret = NJS_OK;
             break;
         }
 
@@ -967,9 +987,9 @@ njs_process_script(njs_vm_t *vm, njs_opt
         ret = njs_vm_run(vm);
 
         if (ret == NJS_ERROR) {
-            njs_output(opts, vm, ret);
+            njs_process_output(vm, ret);
 
-            if (!opts->interactive) {
+            if (!vm->options.interactive) {
                 return NJS_ERROR;
             }
         }
@@ -979,13 +999,64 @@ njs_process_script(njs_vm_t *vm, njs_opt
 }
 
 
+static void
+njs_process_output(njs_vm_t *vm, njs_int_t ret)
+{
+    njs_console_output_pt  pt;
+
+    pt = (njs_console_output_pt) njs_vm_meta(vm, 0);
+
+    if (pt != NULL) {
+        pt(vm, ret);
+    }
+}
+
+
 #if (!defined NJS_FUZZER_TARGET && defined NJS_HAVE_READLINE)
 
+
+volatile sig_atomic_t njs_running;
+volatile sig_atomic_t njs_sigint_count;
+volatile sig_atomic_t njs_sigint_received;
+
+
+static void
+njs_cb_line_handler(char *line_in)
+{
+    njs_int_t  ret;
+    njs_str_t  line;
+
+    if (line_in == NULL || strcmp(line_in, ".exit") == 0) {
+        njs_running = NJS_DONE;
+        return;
+    }
+
+    njs_sigint_count = 0;
+
+    line.start = (u_char *) line_in;
+    line.length = njs_strlen(line.start);
+
+    if (line.length == 0) {
+        return;
+    }
+
+    add_history((char *) line.start);
+
+    ret = njs_process_script(njs_console.vm, &njs_console, &line);
+    if (ret == NJS_ERROR) {
+        njs_running = NJS_ERROR;
+    }
+
+    free(line.start);
+}
+
+
 static njs_int_t
 njs_interactive_shell(njs_opts_t *opts, njs_vm_opt_t *vm_options)
 {
+    fd_set     fds;
     njs_vm_t   *vm;
-    njs_str_t  line;
+    njs_int_t  ret;
 
     if (njs_editline_init() != NJS_OK) {
         njs_stderror("failed to init completions\n");
@@ -1003,27 +1074,64 @@ njs_interactive_shell(njs_opts_t *opts, 
         njs_printf("v.<Tab> -> the properties and prototype methods of v.\n\n");
     }
 
-    for ( ;; ) {
-        line.start = (u_char *) readline(">> ");
-        if (line.start == NULL) {
+    rl_callback_handler_install(">> ", njs_cb_line_handler);
+
+    njs_running = NJS_OK;
+
+    while (njs_running == NJS_OK) {
+        FD_ZERO(&fds);
+        FD_SET(fileno(rl_instream), &fds);
+
+        ret = select(FD_SETSIZE, &fds, NULL, NULL, NULL);
+        if (ret < 0 && errno != EINTR) {
+            njs_stderror("select() failed\n");
+            njs_running = NJS_ERROR;
             break;
         }
 
-        line.length = njs_strlen(line.start);
+        if (njs_sigint_received) {
+            if (njs_sigint_count > 1) {
+                njs_running = NJS_DONE;
+                break;
+            }
+
+            if (rl_end != 0) {
+                njs_printf("\n");
+
+                njs_sigint_count = 0;
 
-        if (line.length != 0) {
-            add_history((char *) line.start);
+            } else {
+                njs_printf("(To exit, press Ctrl+C again or Ctrl+D "
+                           "or type .exit)\n");
 
-            njs_process_script(vm, opts, vm_options->external, &line);
+                njs_sigint_count = 1;
+            }
+
+            rl_point = rl_end = 0;
+            rl_on_new_line();
+            rl_redisplay();
+
+            njs_sigint_received = 0;
         }
 
-        /* editline allocs a new buffer every time. */
-        free(line.start);
+        if (ret < 0) {
+            continue;
+        }
+
+        if (FD_ISSET(fileno(rl_instream), &fds)) {
+            rl_callback_read_char();
+        }
+    }
+
+    rl_callback_handler_remove();
+
+    if (njs_running == NJS_DONE) {
+        njs_printf("exiting\n");
     }
 
     njs_vm_destroy(vm);
 
-    return NJS_OK;
+    return njs_running == NJS_DONE ? NJS_OK : njs_running;
 }
 
 
@@ -1036,6 +1144,20 @@ njs_completion_handler(const char *text,
 }
 
 
+static void
+njs_signal_handler(int signal)
+{
+    switch (signal) {
+    case SIGINT:
+        njs_sigint_received = 1;
+        njs_sigint_count += 1;
+        break;
+    default:
+        break;
+    }
+}
+
+
 static njs_int_t
 njs_editline_init(void)
 {
@@ -1045,6 +1167,8 @@ njs_editline_init(void)
 
     setlocale(LC_ALL, "");
 
+    signal(SIGINT, njs_signal_handler);
+
     return NJS_OK;
 }
 
diff -r b3eeac9ee9f4 -r 8163612eb950 test/shell_test.exp
--- a/test/shell_test.exp	Wed Nov 16 18:47:36 2022 -0800
+++ b/test/shell_test.exp	Thu Nov 17 20:38:23 2022 -0800
@@ -36,8 +36,8 @@ proc njs_test {body {opts ""}} {
         expect [lindex $pair 1]
     }
 
-    # Ctrl-C
-    send \x03
+    send "\n"
+    send ".exit\r\n"
     expect eof
 }
 
@@ -62,10 +62,11 @@ njs_test {
 }
 
 # Global completions, no
-njs_test {
-    {"\t\tn"
-     "\a\r\nDisplay all*possibilities? (y or n)*>> "}
-}
+# Disabled: readline does not support it in callback mode
+# njs_test {
+#    {"\t\tn"
+#     "\a\r\nDisplay all*possibilities? (y or n)*>> "}
+# }
 
 # Global completions, yes
 njs_test {
@@ -87,13 +88,10 @@ njs_test {
      "Ma\a*th"}
 }
 
-# FIXME: completions for external objects
-# are not supported
-
-# njs_test {
-#     {"conso\t"
-#     "conso\a*le"}
-# }
+ njs_test {
+     {"conso\t"
+     "conso\a*le"}
+ }
 
 # Global completions, multiple partial match
 njs_test {
@@ -145,10 +143,10 @@ njs_test {
 
 # Global completions, global vars
 njs_test {
-    {"var a = 1; var aa = 2\r\n"
-     "var a = 1; var aa = 2\r\nundefined\r\n>> "}
-    {"a\t\t"
-     "a*aa*arguments*await"}
+    {"var AA = 1; var AAA = 2\r\n"
+     "var AA = 1; var AAA = 2\r\nundefined\r\n>> "}
+    {"AA\t\t"
+     "AA*AAA*"}
 }
 
 # z*z is WORKAROUND for libedit-20170329.3.1-r3


From mdounin at mdounin.ru  Fri Nov 18 13:53:56 2022
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Fri, 18 Nov 2022 16:53:56 +0300
Subject: [PATCH] Added logging to PROXY protocol write buffer check
In-Reply-To: <Y2t+i8h0Ic32fWRT@mdounin.ru>
References: <22c65e5f1c372f251e2c.1667900802@vm-bsd.mdounin.ru>
 <20221108104510.stbjwbq4zqo2si6v@N00W24XTQX>
 <Y2t+i8h0Ic32fWRT@mdounin.ru>
Message-ID: <Y3eOdIOQAWm1RCHC@mdounin.ru>

Hello!

On Wed, Nov 09, 2022 at 01:18:51PM +0300, Maxim Dounin wrote:

> On Tue, Nov 08, 2022 at 02:45:10PM +0400, Roman Arutyunyan wrote:
> 
> > On Tue, Nov 08, 2022 at 12:46:42PM +0300, Maxim Dounin wrote:
> > > # HG changeset patch
> > > # User Maxim Dounin <mdounin at mdounin.ru>
> > > # Date 1667891773 -10800
> > > #      Tue Nov 08 10:16:13 2022 +0300
> > > # Node ID 22c65e5f1c372f251e2cefdd7aae743794ecfa9e
> > > # Parent  17d6a537fb1bb587e4de22961bf5be5f0c648fa8
> > > Added logging to PROXY protocol write buffer check.
> > > 
> > > The check is not expected to fail unless there is a bug in the calling
> > > code.  But given the check is here, it should log an alert if it fails
> > > instead of silently closing the connection.
> > > 
> > > diff --git a/src/core/ngx_proxy_protocol.c b/src/core/ngx_proxy_protocol.c
> > > --- a/src/core/ngx_proxy_protocol.c
> > > +++ b/src/core/ngx_proxy_protocol.c
> > > @@ -282,6 +282,8 @@ ngx_proxy_protocol_write(ngx_connection_
> > >      ngx_uint_t  port, lport;
> > >  
> > >      if (last - buf < NGX_PROXY_PROTOCOL_V1_MAX_HEADER) {
> > > +        ngx_log_error(NGX_LOG_ALERT, c->log, 0,
> > > +                      "too small buffer for PROXY protocol");
> > >          return NULL;
> > >      }
> > >  
> > 
> > Looks fine
> 
> Pushed to http://mdounin.ru/hg/nginx along with the second patch, 
> thnx.

Ping.  That's still not pulled into http://hg.nginx.org/nginx.

-- 
Maxim Dounin
http://mdounin.ru/


From mdounin at mdounin.ru  Fri Nov 18 13:54:44 2022
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Fri, 18 Nov 2022 16:54:44 +0300
Subject: [PATCH] Filtering duplicate addresses in listen (ticket #2400)
In-Reply-To: <55bcf8dc4ee35ccf40f5.1667097660@vm-bsd.mdounin.ru>
References: <55bcf8dc4ee35ccf40f5.1667097660@vm-bsd.mdounin.ru>
Message-ID: <Y3eOpM9SDtitCXpU@mdounin.ru>

Hello!

Ping.

On Sun, Oct 30, 2022 at 05:41:00AM +0300, Maxim Dounin wrote:

> # HG changeset patch
> # User Maxim Dounin <mdounin at mdounin.ru>
> # Date 1667097653 -10800
> #      Sun Oct 30 05:40:53 2022 +0300
> # Node ID 55bcf8dc4ee35ccf40f5b8a7cffde63e7edb9494
> # Parent  1ae25660c0c76edef14121ca64362f28b9d57a70
> Filtering duplicate addresses in listen (ticket #2400).
> 
> Due to the glibc bug[1], getaddrinfo("localhost") with AI_ADDRCONFIG
> on a typical host with glibc and without IPv6 returns two 127.0.0.1
> addresses, and therefore "listen localhost:80;" used to result the
> "duplicate ... address and port pair" after 4f9b72a229c1.
> 
> Fix is to explicitly filter out duplicate addresses returned during
> resolution of a name.
> 
> [1] https://sourceware.org/bugzilla/show_bug.cgi?id=14969
> 
> diff --git a/src/http/ngx_http_core_module.c b/src/http/ngx_http_core_module.c
> --- a/src/http/ngx_http_core_module.c
> +++ b/src/http/ngx_http_core_module.c
> @@ -3963,7 +3963,7 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx
>  
>      ngx_str_t              *value, size;
>      ngx_url_t               u;
> -    ngx_uint_t              n;
> +    ngx_uint_t              n, i;
>      ngx_http_listen_opt_t   lsopt;
>  
>      cscf->listen = 1;
> @@ -4289,6 +4289,16 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx
>      }
>  
>      for (n = 0; n < u.naddrs; n++) {
> +
> +        for (i = 0; i < n; i++) {
> +            if (ngx_cmp_sockaddr(u.addrs[n].sockaddr, u.addrs[n].socklen,
> +                                 u.addrs[i].sockaddr, u.addrs[i].socklen, 0)
> +                == NGX_OK)
> +            {
> +                goto next;
> +            }
> +        }
> +
>          lsopt.sockaddr = u.addrs[n].sockaddr;
>          lsopt.socklen = u.addrs[n].socklen;
>          lsopt.addr_text = u.addrs[n].name;
> @@ -4297,6 +4307,9 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx
>          if (ngx_http_add_listen(cf, cscf, &lsopt) != NGX_OK) {
>              return NGX_CONF_ERROR;
>          }
> +
> +    next:
> +        continue;
>      }
>  
>      return NGX_CONF_OK;
> diff --git a/src/mail/ngx_mail_core_module.c b/src/mail/ngx_mail_core_module.c
> --- a/src/mail/ngx_mail_core_module.c
> +++ b/src/mail/ngx_mail_core_module.c
> @@ -308,7 +308,7 @@ ngx_mail_core_listen(ngx_conf_t *cf, ngx
>      ngx_str_t                  *value, size;
>      ngx_url_t                   u;
>      ngx_uint_t                  i, n, m;
> -    ngx_mail_listen_t          *ls, *als;
> +    ngx_mail_listen_t          *ls, *als, *nls;
>      ngx_mail_module_t          *module;
>      ngx_mail_core_main_conf_t  *cmcf;
>  
> @@ -333,7 +333,7 @@ ngx_mail_core_listen(ngx_conf_t *cf, ngx
>  
>      cmcf = ngx_mail_conf_get_module_main_conf(cf, ngx_mail_core_module);
>  
> -    ls = ngx_array_push_n(&cmcf->listen, u.naddrs);
> +    ls = ngx_array_push(&cmcf->listen);
>      if (ls == NULL) {
>          return NGX_CONF_ERROR;
>      }
> @@ -571,17 +571,37 @@ ngx_mail_core_listen(ngx_conf_t *cf, ngx
>      als = cmcf->listen.elts;
>  
>      for (n = 0; n < u.naddrs; n++) {
> -        ls[n] = ls[0];
> +
> +        for (i = 0; i < n; i++) {
> +            if (ngx_cmp_sockaddr(u.addrs[n].sockaddr, u.addrs[n].socklen,
> +                                 u.addrs[i].sockaddr, u.addrs[i].socklen, 0)
> +                == NGX_OK)
> +            {
> +                goto next;
> +            }
> +        }
>  
> -        ls[n].sockaddr = u.addrs[n].sockaddr;
> -        ls[n].socklen = u.addrs[n].socklen;
> -        ls[n].addr_text = u.addrs[n].name;
> -        ls[n].wildcard = ngx_inet_wildcard(ls[n].sockaddr);
> +        if (n != 0) {
> +            nls = ngx_array_push(&cmcf->listen);
> +            if (nls == NULL) {
> +                return NGX_CONF_ERROR;
> +            }
> +
> +            *nls = *ls;
>  
> -        for (i = 0; i < cmcf->listen.nelts - u.naddrs + n; i++) {
> +        } else {
> +            nls = ls;
> +        }
> +
> +        nls->sockaddr = u.addrs[n].sockaddr;
> +        nls->socklen = u.addrs[n].socklen;
> +        nls->addr_text = u.addrs[n].name;
> +        nls->wildcard = ngx_inet_wildcard(nls->sockaddr);
> +
> +        for (i = 0; i < cmcf->listen.nelts - 1; i++) {
>  
>              if (ngx_cmp_sockaddr(als[i].sockaddr, als[i].socklen,
> -                                 ls[n].sockaddr, ls[n].socklen, 1)
> +                                 nls->sockaddr, nls->socklen, 1)
>                  != NGX_OK)
>              {
>                  continue;
> @@ -589,9 +609,12 @@ ngx_mail_core_listen(ngx_conf_t *cf, ngx
>  
>              ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
>                                 "duplicate \"%V\" address and port pair",
> -                               &ls[n].addr_text);
> +                               &nls->addr_text);
>              return NGX_CONF_ERROR;
>          }
> +
> +    next:
> +        continue;
>      }
>  
>      return NGX_CONF_OK;
> diff --git a/src/stream/ngx_stream_core_module.c b/src/stream/ngx_stream_core_module.c
> --- a/src/stream/ngx_stream_core_module.c
> +++ b/src/stream/ngx_stream_core_module.c
> @@ -578,7 +578,7 @@ ngx_stream_core_listen(ngx_conf_t *cf, n
>      ngx_str_t                    *value, size;
>      ngx_url_t                     u;
>      ngx_uint_t                    i, n, backlog;
> -    ngx_stream_listen_t          *ls, *als;
> +    ngx_stream_listen_t          *ls, *als, *nls;
>      ngx_stream_core_main_conf_t  *cmcf;
>  
>      cscf->listen = 1;
> @@ -602,7 +602,7 @@ ngx_stream_core_listen(ngx_conf_t *cf, n
>  
>      cmcf = ngx_stream_conf_get_module_main_conf(cf, ngx_stream_core_module);
>  
> -    ls = ngx_array_push_n(&cmcf->listen, u.naddrs);
> +    ls = ngx_array_push(&cmcf->listen);
>      if (ls == NULL) {
>          return NGX_CONF_ERROR;
>      }
> @@ -889,20 +889,40 @@ ngx_stream_core_listen(ngx_conf_t *cf, n
>      als = cmcf->listen.elts;
>  
>      for (n = 0; n < u.naddrs; n++) {
> -        ls[n] = ls[0];
> +
> +        for (i = 0; i < n; i++) {
> +            if (ngx_cmp_sockaddr(u.addrs[n].sockaddr, u.addrs[n].socklen,
> +                                 u.addrs[i].sockaddr, u.addrs[i].socklen, 0)
> +                == NGX_OK)
> +            {
> +                goto next;
> +            }
> +        }
>  
> -        ls[n].sockaddr = u.addrs[n].sockaddr;
> -        ls[n].socklen = u.addrs[n].socklen;
> -        ls[n].addr_text = u.addrs[n].name;
> -        ls[n].wildcard = ngx_inet_wildcard(ls[n].sockaddr);
> +        if (n != 0) {
> +            nls = ngx_array_push(&cmcf->listen);
> +            if (nls == NULL) {
> +                return NGX_CONF_ERROR;
> +            }
> +
> +            *nls = *ls;
>  
> -        for (i = 0; i < cmcf->listen.nelts - u.naddrs + n; i++) {
> -            if (ls[n].type != als[i].type) {
> +        } else {
> +            nls = ls;
> +        }
> +
> +        nls->sockaddr = u.addrs[n].sockaddr;
> +        nls->socklen = u.addrs[n].socklen;
> +        nls->addr_text = u.addrs[n].name;
> +        nls->wildcard = ngx_inet_wildcard(nls->sockaddr);
> +
> +        for (i = 0; i < cmcf->listen.nelts - 1; i++) {
> +            if (nls->type != als[i].type) {
>                  continue;
>              }
>  
>              if (ngx_cmp_sockaddr(als[i].sockaddr, als[i].socklen,
> -                                 ls[n].sockaddr, ls[n].socklen, 1)
> +                                 nls->sockaddr, nls->socklen, 1)
>                  != NGX_OK)
>              {
>                  continue;
> @@ -910,9 +930,12 @@ ngx_stream_core_listen(ngx_conf_t *cf, n
>  
>              ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
>                                 "duplicate \"%V\" address and port pair",
> -                               &ls[n].addr_text);
> +                               &nls->addr_text);
>              return NGX_CONF_ERROR;
>          }
> +
> +    next:
> +        continue;
>      }
>  
>      return NGX_CONF_OK;
> 
> _______________________________________________
> nginx-devel mailing list -- nginx-devel at nginx.org
> To unsubscribe send an email to nginx-devel-leave at nginx.org

-- 
Maxim Dounin
http://mdounin.ru/


From mdounin at mdounin.ru  Fri Nov 18 13:55:15 2022
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Fri, 18 Nov 2022 16:55:15 +0300
Subject: [PATCH] Disabled cloning of sockets without master process
 (ticket #2403)
In-Reply-To: <b73d95226c84b93e51f2.1667097698@vm-bsd.mdounin.ru>
References: <b73d95226c84b93e51f2.1667097698@vm-bsd.mdounin.ru>
Message-ID: <Y3eOw/ESJlxhp2p+@mdounin.ru>

Hello!

Ping.

On Sun, Oct 30, 2022 at 05:41:38AM +0300, Maxim Dounin wrote:

> # HG changeset patch
> # User Maxim Dounin <mdounin at mdounin.ru>
> # Date 1667097682 -10800
> #      Sun Oct 30 05:41:22 2022 +0300
> # Node ID b73d95226c84b93e51f23f7b35782d98d3b516b9
> # Parent  55bcf8dc4ee35ccf40f5b8a7cffde63e7edb9494
> Disabled cloning of sockets without master process (ticket #2403).
> 
> Cloning of listening sockets for each worker process does not make sense
> when working without master process, and causes some of the connections
> not to be accepted if worker_processes is set to more than one and there
> are listening sockets configured with the reuseport flag.  Fix is to
> disable cloning when master process is disabled.
> 
> diff --git a/src/event/ngx_event.c b/src/event/ngx_event.c
> --- a/src/event/ngx_event.c
> +++ b/src/event/ngx_event.c
> @@ -416,6 +416,7 @@ ngx_event_init_conf(ngx_cycle_t *cycle, 
>  {
>  #if (NGX_HAVE_REUSEPORT)
>      ngx_uint_t        i;
> +    ngx_core_conf_t  *ccf;
>      ngx_listening_t  *ls;
>  #endif
>  
> @@ -442,7 +443,9 @@ ngx_event_init_conf(ngx_cycle_t *cycle, 
>  
>  #if (NGX_HAVE_REUSEPORT)
>  
> -    if (!ngx_test_config) {
> +    ccf = (ngx_core_conf_t *) ngx_get_conf(cycle->conf_ctx, ngx_core_module);
> +
> +    if (!ngx_test_config && ccf->master) {
>  
>          ls = cycle->listening.elts;
>          for (i = 0; i < cycle->listening.nelts; i++) {
> 
> _______________________________________________
> nginx-devel mailing list -- nginx-devel at nginx.org
> To unsubscribe send an email to nginx-devel-leave at nginx.org

-- 
Maxim Dounin
http://mdounin.ru/


From mdounin at mdounin.ru  Fri Nov 18 13:55:49 2022
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Fri, 18 Nov 2022 16:55:49 +0300
Subject: [PATCH] Fixed segfault when switching off master process during
 upgrade
In-Reply-To: <ef9c94be7fe4685f0eee.1667097753@vm-bsd.mdounin.ru>
References: <ef9c94be7fe4685f0eee.1667097753@vm-bsd.mdounin.ru>
Message-ID: <Y3eO5e7MgrL3l902@mdounin.ru>

Hello!

Ping.

On Sun, Oct 30, 2022 at 05:42:33AM +0300, Maxim Dounin wrote:

> # HG changeset patch
> # User Maxim Dounin <mdounin at mdounin.ru>
> # Date 1667097733 -10800
> #      Sun Oct 30 05:42:13 2022 +0300
> # Node ID ef9c94be7fe4685f0eeee41f76b964ea252f519f
> # Parent  b73d95226c84b93e51f23f7b35782d98d3b516b9
> Fixed segfault when switching off master process during upgrade.
> 
> Binary upgrades are not supported without master process, but it is,
> however, possible, that nginx running with master process is asked
> to upgrade binary, and the configuration file as available on disk
> at this time includes "master_process off;".
> 
> If this happens, listening sockets inherited from the previous binary
> will have ls[i].previous set.  But the old cycle on initial process
> startup, including startup after binary upgrade, is destroyed by
> ngx_init_cycle() once configuration parsing is complete.  As a result,
> an attempt to dereference ls[i].previous in ngx_event_process_init()
> accesses already freed memory.
> 
> Fix is to avoid looking into ls[i].previous if the old cycle is already
> freed.
> 
> diff --git a/src/event/ngx_event.c b/src/event/ngx_event.c
> --- a/src/event/ngx_event.c
> +++ b/src/event/ngx_event.c
> @@ -813,7 +813,9 @@ ngx_event_process_init(ngx_cycle_t *cycl
>          rev->deferred_accept = ls[i].deferred_accept;
>  #endif
>  
> -        if (!(ngx_event_flags & NGX_USE_IOCP_EVENT)) {
> +        if (!(ngx_event_flags & NGX_USE_IOCP_EVENT)
> +            && cycle->old_cycle)
> +        {
>              if (ls[i].previous) {
>  
>                  /*
> 
> _______________________________________________
> nginx-devel mailing list -- nginx-devel at nginx.org
> To unsubscribe send an email to nginx-devel-leave at nginx.org

-- 
Maxim Dounin
http://mdounin.ru/


From arut at nginx.com  Fri Nov 18 15:33:02 2022
From: arut at nginx.com (=?iso-8859-1?q?Roman_Arutyunyan?=)
Date: Fri, 18 Nov 2022 19:33:02 +0400
Subject: [PATCH] Process events posted by ngx_close_idle_connections()
 immediately
Message-ID: <c185d439cbaecfe8ec5d.1668785582@arut-laptop>

# HG changeset patch
# User Roman Arutyunyan <arut at nginx.com>
# Date 1668785498 -14400
#      Fri Nov 18 19:31:38 2022 +0400
# Node ID c185d439cbaecfe8ec5deb0fadd1a6e9a769990b
# Parent  17d6a537fb1bb587e4de22961bf5be5f0c648fa8
Process events posted by ngx_close_idle_connections() immediately.

Previously, if an event was posted by a read event handler, called by
ngx_close_idle_connections(), that event was not processed until the next
event loop iteration, which could happen after a timeout.

diff --git a/src/os/unix/ngx_process_cycle.c b/src/os/unix/ngx_process_cycle.c
--- a/src/os/unix/ngx_process_cycle.c
+++ b/src/os/unix/ngx_process_cycle.c
@@ -736,6 +736,7 @@ ngx_worker_process_cycle(ngx_cycle_t *cy
                 ngx_set_shutdown_timer(cycle);
                 ngx_close_listening_sockets(cycle);
                 ngx_close_idle_connections(cycle);
+                ngx_event_process_posted(cycle, &ngx_posted_events);
             }
         }
 
diff --git a/src/os/win32/ngx_process_cycle.c b/src/os/win32/ngx_process_cycle.c
--- a/src/os/win32/ngx_process_cycle.c
+++ b/src/os/win32/ngx_process_cycle.c
@@ -804,6 +804,7 @@ ngx_worker_thread(void *data)
                 ngx_set_shutdown_timer(cycle);
                 ngx_close_listening_sockets(cycle);
                 ngx_close_idle_connections(cycle);
+                ngx_event_process_posted(cycle, &ngx_posted_events);
             }
         }
 


From yolkking at 126.com  Fri Nov 18 16:48:46 2022
From: yolkking at 126.com (Yu Zhu)
Date: Sat, 19 Nov 2022 00:48:46 +0800 (CST)
Subject: QUIC: fixed computation of nonce
Message-ID: <6e54119a.44a7.1848ba488b5.Coremail.yolkking@126.com>

# HG changeset patch

# User Yu Zhu <lishu.zy at alibaba-inc.com>

# Date 1668789115 -28800

#      Sat Nov 19 00:31:55 2022 +0800

# Branch quic

# Node ID 1a320805265db14904ca9deaae8330f4979619ce

# Parent  6cf8ed15fd00668b7efa0226c06f47d7238f26e8

QUIC: fixed computation of nonce




RFC 9001, 5.3. AEAD Usage

The nonce, N, is formed by combining the packet protection IV with the packet number. The 62 bits of the reconstructed QUIC packet number in network byte order are left-padded with zeros to the size of the IV. The exclusive OR of the padded packet number and the IV forms the AEAD nonce.




diff -r 6cf8ed15fd00 -r 1a320805265d src/event/quic/ngx_event_quic_protection.c

--- a/src/event/quic/ngx_event_quic_protection.c        Tue Nov 01 17:00:35 2022 +0400

+++ b/src/event/quic/ngx_event_quic_protection.c        Sat Nov 19 00:31:55 2022 +0800

@@ -969,10 +969,11 @@

 static void

 ngx_quic_compute_nonce(u_char *nonce, size_t len, uint64_t pn)

 {

-    nonce[len - 4] ^= (pn & 0xff000000) >> 24;

-    nonce[len - 3] ^= (pn & 0x00ff0000) >> 16;

-    nonce[len - 2] ^= (pn & 0x0000ff00) >> 8;

-    nonce[len - 1] ^= (pn & 0x000000ff);

+    size_t  i;

+

+    for (i = 0; i < 8; i++) {

+        nonce[len - 8 + i] ^= (pn >> (8 - i - 1) * 8) & 0xff;

+    }

 }
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20221119/95bcd0b7/attachment.htm>

From v.zhestikov at f5.com  Fri Nov 18 22:12:04 2022
From: v.zhestikov at f5.com (Vadim Zhestikov)
Date: Fri, 18 Nov 2022 22:12:04 +0000
Subject: [njs] Fixed for(expr1; conditional syntax error handling.
Message-ID: <hg.c51adee54dfe.1668809524.5965299922797593991@dev.nginx>

details:   https://hg.nginx.org/njs/rev/c51adee54dfe
branches:  
changeset: 2007:c51adee54dfe
user:      Vadim Zhestikov <v.zhestikov at f5.com>
date:      Fri Nov 18 14:10:25 2022 -0800
description:
Fixed for(expr1; conditional syntax error handling.

diffstat:

 src/njs_parser.c         |  4 ++++
 src/test/njs_unit_test.c |  3 +++
 2 files changed, 7 insertions(+), 0 deletions(-)

diffs (27 lines):

diff -r 8163612eb950 -r c51adee54dfe src/njs_parser.c
--- a/src/njs_parser.c	Thu Nov 17 20:38:23 2022 -0800
+++ b/src/njs_parser.c	Fri Nov 18 14:10:25 2022 -0800
@@ -5652,6 +5652,10 @@ static njs_int_t
 njs_parser_expression_continue_assign_comma(njs_parser_t *parser,
     njs_lexer_token_t *token, njs_queue_link_t *current)
 {
+    if (parser->ret != NJS_OK) {
+        return njs_parser_failed(parser);
+    }
+
     njs_parser_next(parser, njs_parser_assignment_expression_after);
 
     return njs_parser_after(parser, current, NULL, 1,
diff -r 8163612eb950 -r c51adee54dfe src/test/njs_unit_test.c
--- a/src/test/njs_unit_test.c	Thu Nov 17 20:38:23 2022 -0800
+++ b/src/test/njs_unit_test.c	Fri Nov 18 14:10:25 2022 -0800
@@ -2968,6 +2968,9 @@ static njs_unit_test_t  njs_test[] =
     { njs_str("for(9A=>>"),
       njs_str("SyntaxError: Unexpected token \"A\" in 1") },
 
+    { njs_str("for(A?{,"),
+      njs_str("SyntaxError: Unexpected token \",\" in 1") },
+
     /* switch. */
 
     { njs_str("switch"),


From pluknet at nginx.com  Mon Nov 21 09:28:20 2022
From: pluknet at nginx.com (Sergey Kandaurov)
Date: Mon, 21 Nov 2022 13:28:20 +0400
Subject: [PATCH] SSI: ensure context of main request exists for subrequest
 using SSI
In-Reply-To: <Y3MAglSvf3L6O56g@mdounin.ru>
References: <-SpypjU_KG4ByLBWOaBV_3rln7SC_v-4sAF5Z9nLtOr3EL7bvilrVTfbPXmVCw1XetJSevc6xGYxdGuqLfTfvnAYR2G1ZKcyZqTurHo0vRc=@ciel.dev>
 <Y2P1JK9cD+nMyRfh@mdounin.ru>
 <-gUKUM8ZX15I_C0Ar8PG2FqZyzZg4agx-52A6kAx8_zLiAc-eXbg5903hwRPZxT_6wrpIvVyxTUqNlx9-2JmiTtjVlCoQwp2KM6igGnoddo=@ciel.dev>
 <Y2n8edVi8dKvTsV1@mdounin.ru>
 <4MwuNFVwHxE7Xfc57rVEktH4gLKCciPjR6yGjd5zowzkBnEmz9Q-8THWnUDmsbtljEbclcg_Vwp0MC36DZuPhhVaXqGGR_T7DLgiOrX9fdc=@ciel.dev>
 <Y3MAglSvf3L6O56g@mdounin.ru>
Message-ID: <51506F21-C497-44DF-9DE8-D785F48A2D67@nginx.com>


> On 15 Nov 2022, at 06:59, Maxim Dounin <mdounin at mdounin.ru> wrote:
> 
> Hello!
> 
> [..]
> Below is slightly cleaned up version of this patch, please take a 
> look:
> 
> # HG changeset patch
> # User Ciel Zhao <i at ciel.dev>
> # Date 1667928380 -28800
> #      Wed Nov 09 01:26:20 2022 +0800
> # Node ID 41c67de61c7a916c01f0a1f5054d11821991ffce
> # Parent  42bc158a47ecb3c2bd0396c723c307c757f2770e
> SSI: handling of subrequests from other modules (ticket #1263).
> 
> As the SSI parser always uses the context from the main request for storing
> variables and blocks, that context should always exist for subrequests using
> SSI, even though the main request does not necessarily have SSI enabled.
> 
> However, `ngx_http_get_module_ctx(r->main, ...)` is getting NULL in such cases,
> resulting in the worker crashing SIGSEGV when accessing its attributes.
> 
> This patch links the first initialized context to the main request, and
> upgrades it only when main context is initialized.
> 
> diff --git a/src/http/modules/ngx_http_ssi_filter_module.c b/src/http/modules/ngx_http_ssi_filter_module.c
> --- a/src/http/modules/ngx_http_ssi_filter_module.c
> +++ b/src/http/modules/ngx_http_ssi_filter_module.c
> @@ -329,7 +329,7 @@ static ngx_http_variable_t  ngx_http_ssi
> static ngx_int_t
> ngx_http_ssi_header_filter(ngx_http_request_t *r)
> {
> -    ngx_http_ssi_ctx_t       *ctx;
> +    ngx_http_ssi_ctx_t       *ctx, *mctx;
>     ngx_http_ssi_loc_conf_t  *slcf;
> 
>     slcf = ngx_http_get_module_loc_conf(r, ngx_http_ssi_filter_module);
> @@ -341,6 +341,8 @@ ngx_http_ssi_header_filter(ngx_http_requ
>         return ngx_http_next_header_filter(r);
>     }
> 
> +    mctx = ngx_http_get_module_ctx(r->main, ngx_http_ssi_filter_module);
> +
>     ctx = ngx_pcalloc(r->pool, sizeof(ngx_http_ssi_ctx_t));
>     if (ctx == NULL) {
>         return NGX_ERROR;
> @@ -367,6 +369,20 @@ ngx_http_ssi_header_filter(ngx_http_requ
>     r->filter_need_in_memory = 1;
> 
>     if (r == r->main) {
> +
> +        if (mctx) {
> +
> +            /*
> +             * if there was a shared context previously used as main,
> +             * copy variables and blocks
> +             */
> +
> +            ctx->variables = mctx->variables;
> +            ctx->blocks = mctx->blocks;
> +
> +            mctx->shared = 0;
> +        }
> +
>         ngx_http_clear_content_length(r);
>         ngx_http_clear_accept_ranges(r);
> 
> @@ -379,6 +395,10 @@ ngx_http_ssi_header_filter(ngx_http_requ
>         } else {
>             ngx_http_weak_etag(r);
>         }
> +
> +    } else if (mctx == NULL) {
> +        ngx_http_set_ctx(r->main, ctx, ngx_http_ssi_filter_module);
> +        ctx->shared = 1;
>     }
> 
>     return ngx_http_next_header_filter(r);
> @@ -405,6 +425,7 @@ ngx_http_ssi_body_filter(ngx_http_reques
>     ctx = ngx_http_get_module_ctx(r, ngx_http_ssi_filter_module);
> 
>     if (ctx == NULL
> +        || (ctx->shared && r == r->main)
>         || (in == NULL
>             && ctx->buf == NULL
>             && ctx->in == NULL
> diff --git a/src/http/modules/ngx_http_ssi_filter_module.h b/src/http/modules/ngx_http_ssi_filter_module.h
> --- a/src/http/modules/ngx_http_ssi_filter_module.h
> +++ b/src/http/modules/ngx_http_ssi_filter_module.h
> @@ -71,6 +71,7 @@ typedef struct {
>     u_char                   *captures_data;
> #endif
> 
> +    unsigned                  shared:1;
>     unsigned                  conditional:2;
>     unsigned                  encoding:2;
>     unsigned                  block:1;
> 

The patch doesn't cover regex positional captures in the "if" command.
They are also used to be stored in the main request in ctx->*captures*.
Upgrading main context makes them unavailable, so the fields need to be
copied under mctx condition as well if we want to care about that.

Otherwise, looks good.

-- 
Sergey Kandaurov


From pluknet at nginx.com  Mon Nov 21 11:25:51 2022
From: pluknet at nginx.com (Sergey Kandaurov)
Date: Mon, 21 Nov 2022 15:25:51 +0400
Subject: [PATCH 3 of 4] QUIC: support for setting QUIC methods with
 LibreSSL
In-Reply-To: <Y3cC4v3bWkaUwB48@mdounin.ru>
References: <patchbomb.1665484549@enoparse.local>
 <c0165ddcb1c6981f8e52.1665484552@enoparse.local>
 <Y1HjW9CFekYwNSzp@mdounin.ru>
 <A1EC1C9A-F6BF-4F5C-9E6F-A44A189BC50B@nginx.com>
 <Y3cC4v3bWkaUwB48@mdounin.ru>
Message-ID: <CA597600-AA85-403F-A560-DAD5541E21BC@nginx.com>


> On 18 Nov 2022, at 07:58, Maxim Dounin <mdounin at mdounin.ru> wrote:
> 
> Hello!
> 
> On Tue, Nov 15, 2022 at 04:28:30PM +0400, Sergey Kandaurov wrote:
> 
>> 
>>> On 21 Oct 2022, at 04:10, Maxim Dounin <mdounin at mdounin.ru> wrote:
>>> 
>>> Hello!
>>> 
>>> On Tue, Oct 11, 2022 at 02:35:52PM +0400, Sergey Kandaurov wrote:
>>> 
>>>> # HG changeset patch
>>>> # User Sergey Kandaurov <pluknet at nginx.com>
>>>> # Date 1665484414 -14400
>>>> #      Tue Oct 11 14:33:34 2022 +0400
>>>> # Branch quic
>>>> # Node ID c0165ddcb1c6981f8e5230081f03a277f62d20c3
>>>> # Parent  caced81ce0a9cb218ae8cdd6176c12e0614acee9
>>>> QUIC: support for setting QUIC methods with LibreSSL.
>>>> 
>>>> Setting QUIC methods is converted to use C99 designated initializers
>>>> for simplicity, as LibreSSL 3.6.0 has different SSL_QUIC_METHOD layout.
>>> 
>>> I'm somewhat sceptical about C99 designated initializers.  These 
>>> aren't supported by MSVC till 2012: in particular, this includes 
>>> all MSVC versions available via wineticks, as well as MSVC 
>>> versions currently used to build nginx win32 binaries.
>>> 
>>> A more portable solution might be to use run-time initialization 
>>> instead.
>> 
>> While I agree in principle, fixing build with MSVC would require
>> a larger work to implement support for UDP connections on win32.
>> Specifically, nginx-quic uses c->udp->buffer and sockets API that
>> needs porting to win32, see ngx_quic_send() / ngx_quic_recvmsg().
> 
> This shouldn't be a major problem, given the relevant APIs are 
> available on Windows.
> 
> But the point is: there are no reasons to introduce additional 
> issues in advance, we'll have to fix this sooner or later.
> 
>> Other than that, currently the build fails in other places
>> as seen with MSVC 2010.  Below the patches to address this.

[..]

> 
> Looks good.
> 
>> # HG changeset patch
>> # User Sergey Kandaurov <pluknet at nginx.com>
>> # Date 1668513998 -14400
>> #      Tue Nov 15 16:06:38 2022 +0400
>> # Branch quic
>> # Node ID 1ba7992c7bb0f801e069adb15b3378b5211a85a8
>> # Parent  d760c2e49d4c4647621765fef3404c34d1aef81b
>> QUIC: silenced C4334 MSVC warning about 32 to 64 bits convertion.
> 
> converSion?

Sure, tnx.

> 
>> 
>> diff --git a/src/event/quic/ngx_event_quic_ack.c b/src/event/quic/ngx_event_quic_ack.c
>> --- a/src/event/quic/ngx_event_quic_ack.c
>> +++ b/src/event/quic/ngx_event_quic_ack.c
>> @@ -195,7 +195,7 @@ ngx_quic_rtt_sample(ngx_connection_t *c,
>>     } else {
>>         qc->min_rtt = ngx_min(qc->min_rtt, latest_rtt);
>> 
>> -        ack_delay = ack->delay * (1 << qc->ctp.ack_delay_exponent) / 1000;
>> +        ack_delay = ack->delay * (1ULL << qc->ctp.ack_delay_exponent) / 1000;
>> 
>>         if (c->ssl->handshaked) {
>>             ack_delay = ngx_min(ack_delay, qc->ctp.max_ack_delay);
> 
> Why "1 << ..." at all?
> Shouldn't it be
> 
> diff -r d9ef59e283e3 src/event/quic/ngx_event_quic_ack.c
> --- a/src/event/quic/ngx_event_quic_ack.c	Tue Nov 15 16:02:08 2022 +0400
> +++ b/src/event/quic/ngx_event_quic_ack.c	Fri Nov 18 06:55:05 2022 +0300
> @@ -195,7 +195,7 @@ ngx_quic_rtt_sample(ngx_connection_t *c,
>     } else {
>         qc->min_rtt = ngx_min(qc->min_rtt, latest_rtt);
> 
> -        ack_delay = ack->delay * (1 << qc->ctp.ack_delay_exponent) / 1000;
> +        ack_delay = (ack->delay << qc->ctp.ack_delay_exponent) / 1000;
> 
>         if (c->ssl->handshaked) {
>             ack_delay = ngx_min(ack_delay, qc->ctp.max_ack_delay);
> 
> 
> instead?

Yes, indeed.

BTW, this prevented to emit efficient code of just shift left.
Previously, it used to store and sign extend an intermediate result
and apply the multiply instruction even on high optimization level.

-- 
Sergey Kandaurov


From i at ciel.dev  Mon Nov 21 11:33:42 2022
From: i at ciel.dev (Ciel)
Date: Mon, 21 Nov 2022 11:33:42 +0000
Subject: [PATCH] SSI: ensure context of main request exists for subrequest using SSI
In-Reply-To: <51506F21-C497-44DF-9DE8-D785F48A2D67@nginx.com>
References: <-SpypjU_KG4ByLBWOaBV_3rln7SC_v-4sAF5Z9nLtOr3EL7bvilrVTfbPXmVCw1XetJSevc6xGYxdGuqLfTfvnAYR2G1ZKcyZqTurHo0vRc=@ciel.dev> <Y2P1JK9cD+nMyRfh@mdounin.ru> <-gUKUM8ZX15I_C0Ar8PG2FqZyzZg4agx-52A6kAx8_zLiAc-eXbg5903hwRPZxT_6wrpIvVyxTUqNlx9-2JmiTtjVlCoQwp2KM6igGnoddo=@ciel.dev> <Y2n8edVi8dKvTsV1@mdounin.ru> <4MwuNFVwHxE7Xfc57rVEktH4gLKCciPjR6yGjd5zowzkBnEmz9Q-8THWnUDmsbtljEbclcg_Vwp0MC36DZuPhhVaXqGGR_T7DLgiOrX9fdc=@ciel.dev> <Y3MAglSvf3L6O56g@mdounin.ru> <51506F21-C497-44DF-9DE8-D785F48A2D67@nginx.com>
Message-ID: <e7f6HUbKMV0GtSI6OAhBYA9eyS2Uu7VOc6q25C5-2gMmKgezhGDMQOnQE-zjMf2oKxvK-bTcX3ZCOtn2gB3v4It_yGkuooFpO0wSwJPGBYE=@ciel.dev>

Hello!

On Monday, November 21st, 2022 at 17:28, Sergey Kandaurov <pluknet at nginx.com> wrote:

> The patch doesn't cover regex positional captures in the "if" command.
> They are also used to be stored in the main request in ctx->captures.
> 
> Upgrading main context makes them unavailable, so the fields need to be
> copied under mctx condition as well if we want to care about that.

Thanks for pointing out that. I've been scanning this code finding `mctx`s, but
the one in `ngx_http_ssi_get_variable` used `ctx` instead.

Now I've changed that variable to `mctx` to align with other references, and
also made sure that every context of `r->main` gets correct name. If that's not
appropriate, feel free to trim that part.

I'm not familiar with nginx's roll-out strategy, could anyone kindly tell me when
shall I expect to see this usable in a stable/mainline release?

Updated patch below.


# HG changeset patch
# User Ciel Zhao <i at ciel.dev>
# Date 1669029675 -28800
#      Mon Nov 21 19:21:15 2022 +0800
# Node ID a42cc9c1b1fd6c07798c7ad6c6a2a3da24e9cc21
# Parent  17d6a537fb1bb587e4de22961bf5be5f0c648fa8
SSI: handling of subrequests from other modules (ticket #1263).

As the SSI parser always uses the context from the main request for storing
variables and blocks, that context should always exist for subrequests using
SSI, even though the main request does not necessarily have SSI enabled.

However, `ngx_http_get_module_ctx(r->main, ...)` is getting NULL in such cases,
resulting in the worker crashing SIGSEGV when accessing its attributes.

This patch links the first initialized context to the main request, and
upgrades it only when main context is initialized.

diff -r 17d6a537fb1b -r a42cc9c1b1fd src/http/modules/ngx_http_ssi_filter_module.c
--- a/src/http/modules/ngx_http_ssi_filter_module.c     Wed Nov 02 13:46:16 2022 +0400
+++ b/src/http/modules/ngx_http_ssi_filter_module.c     Mon Nov 21 19:21:15 2022 +0800
@@ -329,7 +329,7 @@
 static ngx_int_t
 ngx_http_ssi_header_filter(ngx_http_request_t *r)
 {
-    ngx_http_ssi_ctx_t       *ctx;
+    ngx_http_ssi_ctx_t       *ctx, *mctx;
     ngx_http_ssi_loc_conf_t  *slcf;
 
     slcf = ngx_http_get_module_loc_conf(r, ngx_http_ssi_filter_module);
@@ -341,6 +341,8 @@
         return ngx_http_next_header_filter(r);
     }
 
+    mctx = ngx_http_get_module_ctx(r->main, ngx_http_ssi_filter_module);
+
     ctx = ngx_pcalloc(r->pool, sizeof(ngx_http_ssi_ctx_t));
     if (ctx == NULL) {
         return NGX_ERROR;
@@ -367,6 +369,26 @@
     r->filter_need_in_memory = 1;
 
     if (r == r->main) {
+
+        if (mctx) {
+
+            /*
+             * if there was a shared context previously used as main,
+             * copy variables and blocks
+             */
+
+            ctx->variables = mctx->variables;
+            ctx->blocks = mctx->blocks;
+
+#if (NGX_PCRE)
+            ctx->ncaptures = mctx->ncaptures;
+            ctx->captures = mctx->captures;
+            ctx->captures_data = mctx->captures_data;
+#endif
+
+            mctx->shared = 0;
+        }
+
         ngx_http_clear_content_length(r);
         ngx_http_clear_accept_ranges(r);
 
@@ -379,6 +401,10 @@
         } else {
             ngx_http_weak_etag(r);
         }
+
+    } else if (mctx == NULL) {
+        ngx_http_set_ctx(r->main, ctx, ngx_http_ssi_filter_module);
+        ctx->shared = 1;
     }
 
     return ngx_http_next_header_filter(r);
@@ -405,6 +431,7 @@
     ctx = ngx_http_get_module_ctx(r, ngx_http_ssi_filter_module);
 
     if (ctx == NULL
+        || (ctx->shared && r == r->main)
         || (in == NULL
             && ctx->buf == NULL
             && ctx->in == NULL
@@ -1559,9 +1586,9 @@
     ngx_uint_t           i;
     ngx_list_part_t     *part;
     ngx_http_ssi_var_t  *var;
-    ngx_http_ssi_ctx_t  *ctx;
-
-    ctx = ngx_http_get_module_ctx(r->main, ngx_http_ssi_filter_module);
+    ngx_http_ssi_ctx_t  *mctx;
+
+    mctx = ngx_http_get_module_ctx(r->main, ngx_http_ssi_filter_module);
 
 #if (NGX_PCRE)
     {
@@ -1570,7 +1597,7 @@
     if (key >= '0' && key <= '9') {
         i = key - '0';
 
-        if (i < ctx->ncaptures) {
+        if (i < mctx->ncaptures) {
             value = ngx_palloc(r->pool, sizeof(ngx_str_t));
             if (value == NULL) {
                 return NULL;
@@ -1578,8 +1605,8 @@
 
             i *= 2;
 
-            value->data = ctx->captures_data + ctx->captures[i];
-            value->len = ctx->captures[i + 1] - ctx->captures[i];
+            value->data = mctx->captures_data + mctx->captures[i];
+            value->len = mctx->captures[i + 1] - mctx->captures[i];
 
             return value;
         }
@@ -1587,11 +1614,11 @@
     }
 #endif
 
-    if (ctx->variables == NULL) {
+    if (mctx->variables == NULL) {
         return NULL;
     }
 
-    part = &ctx->variables->part;
+    part = &mctx->variables->part;
     var = part->elts;
 
     for (i = 0; /* void */ ; i++) {
diff -r 17d6a537fb1b -r a42cc9c1b1fd src/http/modules/ngx_http_ssi_filter_module.h
--- a/src/http/modules/ngx_http_ssi_filter_module.h     Wed Nov 02 13:46:16 2022 +0400
+++ b/src/http/modules/ngx_http_ssi_filter_module.h     Mon Nov 21 19:21:15 2022 +0800
@@ -71,6 +71,7 @@
     u_char                   *captures_data;
 #endif
 
+    unsigned                  shared:1;
     unsigned                  conditional:2;
     unsigned                  encoding:2;
     unsigned                  block:1;


From mdounin at mdounin.ru  Mon Nov 21 14:04:17 2022
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Mon, 21 Nov 2022 17:04:17 +0300
Subject: [PATCH] SSI: ensure context of main request exists for
 subrequest using SSI
In-Reply-To: <e7f6HUbKMV0GtSI6OAhBYA9eyS2Uu7VOc6q25C5-2gMmKgezhGDMQOnQE-zjMf2oKxvK-bTcX3ZCOtn2gB3v4It_yGkuooFpO0wSwJPGBYE=@ciel.dev>
References: <-SpypjU_KG4ByLBWOaBV_3rln7SC_v-4sAF5Z9nLtOr3EL7bvilrVTfbPXmVCw1XetJSevc6xGYxdGuqLfTfvnAYR2G1ZKcyZqTurHo0vRc=@ciel.dev>
 <Y2P1JK9cD+nMyRfh@mdounin.ru>
 <-gUKUM8ZX15I_C0Ar8PG2FqZyzZg4agx-52A6kAx8_zLiAc-eXbg5903hwRPZxT_6wrpIvVyxTUqNlx9-2JmiTtjVlCoQwp2KM6igGnoddo=@ciel.dev>
 <Y2n8edVi8dKvTsV1@mdounin.ru>
 <4MwuNFVwHxE7Xfc57rVEktH4gLKCciPjR6yGjd5zowzkBnEmz9Q-8THWnUDmsbtljEbclcg_Vwp0MC36DZuPhhVaXqGGR_T7DLgiOrX9fdc=@ciel.dev>
 <Y3MAglSvf3L6O56g@mdounin.ru>
 <51506F21-C497-44DF-9DE8-D785F48A2D67@nginx.com>
 <e7f6HUbKMV0GtSI6OAhBYA9eyS2Uu7VOc6q25C5-2gMmKgezhGDMQOnQE-zjMf2oKxvK-bTcX3ZCOtn2gB3v4It_yGkuooFpO0wSwJPGBYE=@ciel.dev>
Message-ID: <Y3uFYRldepAnSl9E@mdounin.ru>

Hello!

On Mon, Nov 21, 2022 at 11:33:42AM +0000, Ciel via nginx-devel wrote:

> On Monday, November 21st, 2022 at 17:28, Sergey Kandaurov <pluknet at nginx.com> wrote:
> 
> > The patch doesn't cover regex positional captures in the "if" command.
> > They are also used to be stored in the main request in ctx->captures.
> > 
> > Upgrading main context makes them unavailable, so the fields need to be
> > copied under mctx condition as well if we want to care about that.
> 
> Thanks for pointing out that. I've been scanning this code finding `mctx`s, but
> the one in `ngx_http_ssi_get_variable` used `ctx` instead.
> 
> Now I've changed that variable to `mctx` to align with other references, and
> also made sure that every context of `r->main` gets correct name. If that's not
> appropriate, feel free to trim that part.

That's an unrelated style change, and it should be done in a 
separate patch, if at all (and I would rather refrain from 
changing it).

> I'm not familiar with nginx's roll-out strategy, could anyone kindly tell me when
> shall I expect to see this usable in a stable/mainline release?

Once committed, it will be available in the next mainline release.  
Likely to happen in a couple of months.

> Updated patch below.
> 
> 
> # HG changeset patch
> # User Ciel Zhao <i at ciel.dev>
> # Date 1669029675 -28800
> #      Mon Nov 21 19:21:15 2022 +0800
> # Node ID a42cc9c1b1fd6c07798c7ad6c6a2a3da24e9cc21
> # Parent  17d6a537fb1bb587e4de22961bf5be5f0c648fa8
> SSI: handling of subrequests from other modules (ticket #1263).
> 
> As the SSI parser always uses the context from the main request for storing
> variables and blocks, that context should always exist for subrequests using
> SSI, even though the main request does not necessarily have SSI enabled.
> 
> However, `ngx_http_get_module_ctx(r->main, ...)` is getting NULL in such cases,
> resulting in the worker crashing SIGSEGV when accessing its attributes.
> 
> This patch links the first initialized context to the main request, and
> upgrades it only when main context is initialized.
> 
> diff -r 17d6a537fb1b -r a42cc9c1b1fd src/http/modules/ngx_http_ssi_filter_module.c
> --- a/src/http/modules/ngx_http_ssi_filter_module.c     Wed Nov 02 13:46:16 2022 +0400
> +++ b/src/http/modules/ngx_http_ssi_filter_module.c     Mon Nov 21 19:21:15 2022 +0800
> @@ -329,7 +329,7 @@
>  static ngx_int_t
>  ngx_http_ssi_header_filter(ngx_http_request_t *r)
>  {
> -    ngx_http_ssi_ctx_t       *ctx;
> +    ngx_http_ssi_ctx_t       *ctx, *mctx;
>      ngx_http_ssi_loc_conf_t  *slcf;
>  
>      slcf = ngx_http_get_module_loc_conf(r, ngx_http_ssi_filter_module);
> @@ -341,6 +341,8 @@
>          return ngx_http_next_header_filter(r);
>      }
>  
> +    mctx = ngx_http_get_module_ctx(r->main, ngx_http_ssi_filter_module);
> +
>      ctx = ngx_pcalloc(r->pool, sizeof(ngx_http_ssi_ctx_t));
>      if (ctx == NULL) {
>          return NGX_ERROR;
> @@ -367,6 +369,26 @@
>      r->filter_need_in_memory = 1;
>  
>      if (r == r->main) {
> +
> +        if (mctx) {
> +
> +            /*
> +             * if there was a shared context previously used as main,
> +             * copy variables and blocks
> +             */
> +
> +            ctx->variables = mctx->variables;
> +            ctx->blocks = mctx->blocks;
> +
> +#if (NGX_PCRE)
> +            ctx->ncaptures = mctx->ncaptures;
> +            ctx->captures = mctx->captures;
> +            ctx->captures_data = mctx->captures_data;
> +#endif
> +
> +            mctx->shared = 0;
> +        }
> +
>          ngx_http_clear_content_length(r);
>          ngx_http_clear_accept_ranges(r);
>  

Looks good to me.

Pushed to http://mdounin.ru/hg/nginx without the variable renaming 
part, with just

@@ -380,6 +380,12 @@ ngx_http_ssi_header_filter(ngx_http_requ
             ctx->variables = mctx->variables;
             ctx->blocks = mctx->blocks;
 
+#if (NGX_PCRE)
+            ctx->ncaptures = mctx->ncaptures;
+            ctx->captures = mctx->captures;
+            ctx->captures_data = mctx->captures_data;
+#endif
+
             mctx->shared = 0;
         }
 

compared to the previous version of the patch.

Thanks.

-- 
Maxim Dounin
http://mdounin.ru/


From pluknet at nginx.com  Mon Nov 21 15:29:33 2022
From: pluknet at nginx.com (Sergey Kandaurov)
Date: Mon, 21 Nov 2022 15:29:33 +0000
Subject: [nginx] Added logging to PROXY protocol write buffer check.
Message-ID: <hg.251daa98cc87.1669044573.6026610855610030274@dev.nginx>

details:   https://hg.nginx.org/nginx/rev/251daa98cc87
branches:  
changeset: 8100:251daa98cc87
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Tue Nov 08 12:48:19 2022 +0300
description:
Added logging to PROXY protocol write buffer check.

The check is not expected to fail unless there is a bug in the calling
code.  But given the check is here, it should log an alert if it fails
instead of silently closing the connection.

diffstat:

 src/core/ngx_proxy_protocol.c |  2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diffs (12 lines):

diff -r 17d6a537fb1b -r 251daa98cc87 src/core/ngx_proxy_protocol.c
--- a/src/core/ngx_proxy_protocol.c	Wed Nov 02 13:46:16 2022 +0400
+++ b/src/core/ngx_proxy_protocol.c	Tue Nov 08 12:48:19 2022 +0300
@@ -282,6 +282,8 @@ ngx_proxy_protocol_write(ngx_connection_
     ngx_uint_t  port, lport;
 
     if (last - buf < NGX_PROXY_PROTOCOL_V1_MAX_HEADER) {
+        ngx_log_error(NGX_LOG_ALERT, c->log, 0,
+                      "too small buffer for PROXY protocol");
         return NULL;
     }
 


From pluknet at nginx.com  Mon Nov 21 15:29:36 2022
From: pluknet at nginx.com (Sergey Kandaurov)
Date: Mon, 21 Nov 2022 15:29:36 +0000
Subject: [nginx] Fixed PROXY protocol to use ngx_memcpy()/ngx_memcmp().
Message-ID: <hg.42bc158a47ec.1669044576.6026610855610030274@dev.nginx>

details:   https://hg.nginx.org/nginx/rev/42bc158a47ec
branches:  
changeset: 8101:42bc158a47ec
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Tue Nov 08 12:48:21 2022 +0300
description:
Fixed PROXY protocol to use ngx_memcpy()/ngx_memcmp().

diffstat:

 src/core/ngx_proxy_protocol.c |  10 +++++-----
 1 files changed, 5 insertions(+), 5 deletions(-)

diffs (40 lines):

diff -r 251daa98cc87 -r 42bc158a47ec src/core/ngx_proxy_protocol.c
--- a/src/core/ngx_proxy_protocol.c	Tue Nov 08 12:48:19 2022 +0300
+++ b/src/core/ngx_proxy_protocol.c	Tue Nov 08 12:48:21 2022 +0300
@@ -109,7 +109,7 @@ ngx_proxy_protocol_read(ngx_connection_t
     len = last - buf;
 
     if (len >= sizeof(ngx_proxy_protocol_header_t)
-        && memcmp(p, signature, sizeof(signature) - 1) == 0)
+        && ngx_memcmp(p, signature, sizeof(signature) - 1) == 0)
     {
         return ngx_proxy_protocol_v2_read(c, buf, last);
     }
@@ -396,11 +396,11 @@ ngx_proxy_protocol_v2_read(ngx_connectio
 
         src_sockaddr.sockaddr_in.sin_family = AF_INET;
         src_sockaddr.sockaddr_in.sin_port = 0;
-        memcpy(&src_sockaddr.sockaddr_in.sin_addr, in->src_addr, 4);
+        ngx_memcpy(&src_sockaddr.sockaddr_in.sin_addr, in->src_addr, 4);
 
         dst_sockaddr.sockaddr_in.sin_family = AF_INET;
         dst_sockaddr.sockaddr_in.sin_port = 0;
-        memcpy(&dst_sockaddr.sockaddr_in.sin_addr, in->dst_addr, 4);
+        ngx_memcpy(&dst_sockaddr.sockaddr_in.sin_addr, in->dst_addr, 4);
 
         pp->src_port = ngx_proxy_protocol_parse_uint16(in->src_port);
         pp->dst_port = ngx_proxy_protocol_parse_uint16(in->dst_port);
@@ -423,11 +423,11 @@ ngx_proxy_protocol_v2_read(ngx_connectio
 
         src_sockaddr.sockaddr_in6.sin6_family = AF_INET6;
         src_sockaddr.sockaddr_in6.sin6_port = 0;
-        memcpy(&src_sockaddr.sockaddr_in6.sin6_addr, in6->src_addr, 16);
+        ngx_memcpy(&src_sockaddr.sockaddr_in6.sin6_addr, in6->src_addr, 16);
 
         dst_sockaddr.sockaddr_in6.sin6_family = AF_INET6;
         dst_sockaddr.sockaddr_in6.sin6_port = 0;
-        memcpy(&dst_sockaddr.sockaddr_in6.sin6_addr, in6->dst_addr, 16);
+        ngx_memcpy(&dst_sockaddr.sockaddr_in6.sin6_addr, in6->dst_addr, 16);
 
         pp->src_port = ngx_proxy_protocol_parse_uint16(in6->src_port);
         pp->dst_port = ngx_proxy_protocol_parse_uint16(in6->dst_port);


From pluknet at nginx.com  Mon Nov 21 15:29:39 2022
From: pluknet at nginx.com (Sergey Kandaurov)
Date: Mon, 21 Nov 2022 15:29:39 +0000
Subject: [nginx] SSI: handling of subrequests from other modules (ticket
 #1263).
Message-ID: <hg.49e7db44b57c.1669044579.6026610855610030274@dev.nginx>

details:   https://hg.nginx.org/nginx/rev/49e7db44b57c
branches:  
changeset: 8102:49e7db44b57c
user:      Ciel Zhao <i at ciel.dev>
date:      Mon Nov 21 17:01:34 2022 +0300
description:
SSI: handling of subrequests from other modules (ticket #1263).

As the SSI parser always uses the context from the main request for storing
variables and blocks, that context should always exist for subrequests using
SSI, even though the main request does not necessarily have SSI enabled.

However, `ngx_http_get_module_ctx(r->main, ...)` is getting NULL in such cases,
resulting in the worker crashing SIGSEGV when accessing its attributes.

This patch links the first initialized context to the main request, and
upgrades it only when main context is initialized.

diffstat:

 src/http/modules/ngx_http_ssi_filter_module.c |  29 ++++++++++++++++++++++++++-
 src/http/modules/ngx_http_ssi_filter_module.h |   1 +
 2 files changed, 29 insertions(+), 1 deletions(-)

diffs (78 lines):

diff -r 42bc158a47ec -r 49e7db44b57c src/http/modules/ngx_http_ssi_filter_module.c
--- a/src/http/modules/ngx_http_ssi_filter_module.c	Tue Nov 08 12:48:21 2022 +0300
+++ b/src/http/modules/ngx_http_ssi_filter_module.c	Mon Nov 21 17:01:34 2022 +0300
@@ -329,7 +329,7 @@ static ngx_http_variable_t  ngx_http_ssi
 static ngx_int_t
 ngx_http_ssi_header_filter(ngx_http_request_t *r)
 {
-    ngx_http_ssi_ctx_t       *ctx;
+    ngx_http_ssi_ctx_t       *ctx, *mctx;
     ngx_http_ssi_loc_conf_t  *slcf;
 
     slcf = ngx_http_get_module_loc_conf(r, ngx_http_ssi_filter_module);
@@ -341,6 +341,8 @@ ngx_http_ssi_header_filter(ngx_http_requ
         return ngx_http_next_header_filter(r);
     }
 
+    mctx = ngx_http_get_module_ctx(r->main, ngx_http_ssi_filter_module);
+
     ctx = ngx_pcalloc(r->pool, sizeof(ngx_http_ssi_ctx_t));
     if (ctx == NULL) {
         return NGX_ERROR;
@@ -367,6 +369,26 @@ ngx_http_ssi_header_filter(ngx_http_requ
     r->filter_need_in_memory = 1;
 
     if (r == r->main) {
+
+        if (mctx) {
+
+            /*
+             * if there was a shared context previously used as main,
+             * copy variables and blocks
+             */
+
+            ctx->variables = mctx->variables;
+            ctx->blocks = mctx->blocks;
+
+#if (NGX_PCRE)
+            ctx->ncaptures = mctx->ncaptures;
+            ctx->captures = mctx->captures;
+            ctx->captures_data = mctx->captures_data;
+#endif
+
+            mctx->shared = 0;
+        }
+
         ngx_http_clear_content_length(r);
         ngx_http_clear_accept_ranges(r);
 
@@ -379,6 +401,10 @@ ngx_http_ssi_header_filter(ngx_http_requ
         } else {
             ngx_http_weak_etag(r);
         }
+
+    } else if (mctx == NULL) {
+        ngx_http_set_ctx(r->main, ctx, ngx_http_ssi_filter_module);
+        ctx->shared = 1;
     }
 
     return ngx_http_next_header_filter(r);
@@ -405,6 +431,7 @@ ngx_http_ssi_body_filter(ngx_http_reques
     ctx = ngx_http_get_module_ctx(r, ngx_http_ssi_filter_module);
 
     if (ctx == NULL
+        || (ctx->shared && r == r->main)
         || (in == NULL
             && ctx->buf == NULL
             && ctx->in == NULL
diff -r 42bc158a47ec -r 49e7db44b57c src/http/modules/ngx_http_ssi_filter_module.h
--- a/src/http/modules/ngx_http_ssi_filter_module.h	Tue Nov 08 12:48:21 2022 +0300
+++ b/src/http/modules/ngx_http_ssi_filter_module.h	Mon Nov 21 17:01:34 2022 +0300
@@ -71,6 +71,7 @@ typedef struct {
     u_char                   *captures_data;
 #endif
 
+    unsigned                  shared:1;
     unsigned                  conditional:2;
     unsigned                  encoding:2;
     unsigned                  block:1;


From pluknet at nginx.com  Mon Nov 21 15:29:58 2022
From: pluknet at nginx.com (Sergey Kandaurov)
Date: Mon, 21 Nov 2022 19:29:58 +0400
Subject: [PATCH] Added logging to PROXY protocol write buffer check
In-Reply-To: <Y3eOdIOQAWm1RCHC@mdounin.ru>
References: <22c65e5f1c372f251e2c.1667900802@vm-bsd.mdounin.ru>
 <20221108104510.stbjwbq4zqo2si6v@N00W24XTQX> <Y2t+i8h0Ic32fWRT@mdounin.ru>
 <Y3eOdIOQAWm1RCHC@mdounin.ru>
Message-ID: <14BD1505-6186-4975-8F32-BB809F4830DD@nginx.com>


> On 18 Nov 2022, at 17:53, Maxim Dounin <mdounin at mdounin.ru> wrote:
> 
> Hello!
> 
> On Wed, Nov 09, 2022 at 01:18:51PM +0300, Maxim Dounin wrote:
> 
>> On Tue, Nov 08, 2022 at 02:45:10PM +0400, Roman Arutyunyan wrote:
>> 
>>> On Tue, Nov 08, 2022 at 12:46:42PM +0300, Maxim Dounin wrote:
>>>> # HG changeset patch
>>>> # User Maxim Dounin <mdounin at mdounin.ru>
>>>> # Date 1667891773 -10800
>>>> #      Tue Nov 08 10:16:13 2022 +0300
>>>> # Node ID 22c65e5f1c372f251e2cefdd7aae743794ecfa9e
>>>> # Parent  17d6a537fb1bb587e4de22961bf5be5f0c648fa8
>>>> Added logging to PROXY protocol write buffer check.
>>>> 
>>>> The check is not expected to fail unless there is a bug in the calling
>>>> code.  But given the check is here, it should log an alert if it fails
>>>> instead of silently closing the connection.
>>>> 
>>>> diff --git a/src/core/ngx_proxy_protocol.c b/src/core/ngx_proxy_protocol.c
>>>> --- a/src/core/ngx_proxy_protocol.c
>>>> +++ b/src/core/ngx_proxy_protocol.c
>>>> @@ -282,6 +282,8 @@ ngx_proxy_protocol_write(ngx_connection_
>>>>     ngx_uint_t  port, lport;
>>>> 
>>>>     if (last - buf < NGX_PROXY_PROTOCOL_V1_MAX_HEADER) {
>>>> +        ngx_log_error(NGX_LOG_ALERT, c->log, 0,
>>>> +                      "too small buffer for PROXY protocol");
>>>>         return NULL;
>>>>     }
>>>> 
>>> 
>>> Looks fine
>> 
>> Pushed to http://mdounin.ru/hg/nginx along with the second patch, 
>> thnx.
> 
> Ping.  That's still not pulled into http://hg.nginx.org/nginx.

Pulled now, sorry for the delay.

-- 
Sergey Kandaurov


From pluknet at nginx.com  Tue Nov 22 10:29:47 2022
From: pluknet at nginx.com (Sergey Kandaurov)
Date: Tue, 22 Nov 2022 14:29:47 +0400
Subject: QUIC: fixed computation of nonce
In-Reply-To: <6e54119a.44a7.1848ba488b5.Coremail.yolkking@126.com>
References: <6e54119a.44a7.1848ba488b5.Coremail.yolkking@126.com>
Message-ID: <D561E0F8-1395-4864-AEF4-C5FEA8E41307@nginx.com>


> On 18 Nov 2022, at 20:48, Yu Zhu <yolkking at 126.com> wrote:
> 
> # HG changeset patch
> # User Yu Zhu <lishu.zy at alibaba-inc.com>
> # Date 1668789115 -28800
> #      Sat Nov 19 00:31:55 2022 +0800
> # Branch quic
> # Node ID 1a320805265db14904ca9deaae8330f4979619ce
> # Parent  6cf8ed15fd00668b7efa0226c06f47d7238f26e8
> QUIC: fixed computation of nonce
> 
> RFC 9001, 5.3. AEAD Usage
> The nonce, N, is formed by combining the packet protection IV with the packet number. The 62 bits of the reconstructed QUIC packet number in network byte order are left-padded with zeros to the size of the IV. The exclusive OR of the padded packet number and the IV forms the AEAD nonce.

While citing RFCs is certainly good, I would limit this
to a specific description what this patch intends to do.

> 
> diff -r 6cf8ed15fd00 -r 1a320805265d src/event/quic/ngx_event_quic_protection.c
> --- a/src/event/quic/ngx_event_quic_protection.c        Tue Nov 01 17:00:35 2022 +0400
> +++ b/src/event/quic/ngx_event_quic_protection.c        Sat Nov 19 00:31:55 2022 +0800
> @@ -969,10 +969,11 @@
>  static void
>  ngx_quic_compute_nonce(u_char *nonce, size_t len, uint64_t pn)
>  {
> -    nonce[len - 4] ^= (pn & 0xff000000) >> 24;
> -    nonce[len - 3] ^= (pn & 0x00ff0000) >> 16;
> -    nonce[len - 2] ^= (pn & 0x0000ff00) >> 8;
> -    nonce[len - 1] ^= (pn & 0x000000ff);
> +    size_t  i;
> +
> +    for (i = 0; i < 8; i++) {
> +        nonce[len - 8 + i] ^= (pn >> (8 - i - 1) * 8) & 0xff;
> +    }
>  }

Thanks for the patch.
I tend to prefer computation with unrolled loops, though.
Microbenchmarking shows that it performs better.
Also, this allows to peel two high bits without additional branching.

100m loop, user time in seconds

                -O0     -O1     -O2
gcc12/arm64
old             0.609   0.233   0.024
new             1.030   0.454   0.683
clang14/arm64
old             0.447   0.256   0.096*
new             1.292   0.453   0.096*
gcc8/i386
old             1.519   0.541   0.551
new             2.915   1.318   1.476

# HG changeset patch
# User Sergey Kandaurov <pluknet at nginx.com>
# Date 1669112795 -14400
#      Tue Nov 22 14:26:35 2022 +0400
# Branch quic
# Node ID 29d8d12b6d35eb172465db2dff47bd8e98b36fc7
# Parent  0f5fc7a320db621da8731a8832f486c7167b236b
QUIC: fixed computation of nonce with packet numbers beyond 2^32.

Prodded by Yu Zhu.

diff --git a/src/event/quic/ngx_event_quic_protection.c b/src/event/quic/ngx_event_quic_protection.c
--- a/src/event/quic/ngx_event_quic_protection.c
+++ b/src/event/quic/ngx_event_quic_protection.c
@@ -969,10 +969,14 @@ ngx_quic_parse_pn(u_char **pos, ngx_int_
 static void
 ngx_quic_compute_nonce(u_char *nonce, size_t len, uint64_t pn)
 {
-    nonce[len - 4] ^= (pn & 0xff000000) >> 24;
-    nonce[len - 3] ^= (pn & 0x00ff0000) >> 16;
-    nonce[len - 2] ^= (pn & 0x0000ff00) >> 8;
-    nonce[len - 1] ^= (pn & 0x000000ff);
+    nonce[len - 8] ^= (pn >> 56) & 0x3f;
+    nonce[len - 7] ^= (pn >> 48) & 0xff;
+    nonce[len - 6] ^= (pn >> 40) & 0xff;
+    nonce[len - 5] ^= (pn >> 32) & 0xff;
+    nonce[len - 4] ^= (pn >> 24) & 0xff;
+    nonce[len - 3] ^= (pn >> 16) & 0xff;
+    nonce[len - 2] ^= (pn >> 8) & 0xff;
+    nonce[len - 1] ^= pn & 0xff;
 }
 
 

-- 
Sergey Kandaurov


From pluknet at nginx.com  Tue Nov 22 23:03:07 2022
From: pluknet at nginx.com (Sergey Kandaurov)
Date: Wed, 23 Nov 2022 03:03:07 +0400
Subject: [PATCH] Process events posted by ngx_close_idle_connections()
 immediately
In-Reply-To: <c185d439cbaecfe8ec5d.1668785582@arut-laptop>
References: <c185d439cbaecfe8ec5d.1668785582@arut-laptop>
Message-ID: <20221122230307.c2s74wcdfpiw3s7x@Y9MQ9X2QVV>

On Fri, Nov 18, 2022 at 07:33:02PM +0400, Roman Arutyunyan wrote:
> # HG changeset patch
> # User Roman Arutyunyan <arut at nginx.com>
> # Date 1668785498 -14400
> #      Fri Nov 18 19:31:38 2022 +0400
> # Node ID c185d439cbaecfe8ec5deb0fadd1a6e9a769990b
> # Parent  17d6a537fb1bb587e4de22961bf5be5f0c648fa8
> Process events posted by ngx_close_idle_connections() immediately.
> 
> Previously, if an event was posted by a read event handler, called by
> ngx_close_idle_connections(), that event was not processed until the next
> event loop iteration, which could happen after a timeout.
> 
> diff --git a/src/os/unix/ngx_process_cycle.c b/src/os/unix/ngx_process_cycle.c
> --- a/src/os/unix/ngx_process_cycle.c
> +++ b/src/os/unix/ngx_process_cycle.c
> @@ -736,6 +736,7 @@ ngx_worker_process_cycle(ngx_cycle_t *cy
>                  ngx_set_shutdown_timer(cycle);
>                  ngx_close_listening_sockets(cycle);
>                  ngx_close_idle_connections(cycle);
> +                ngx_event_process_posted(cycle, &ngx_posted_events);
>              }
>          }
>  
> diff --git a/src/os/win32/ngx_process_cycle.c b/src/os/win32/ngx_process_cycle.c
> --- a/src/os/win32/ngx_process_cycle.c
> +++ b/src/os/win32/ngx_process_cycle.c
> @@ -804,6 +804,7 @@ ngx_worker_thread(void *data)
>                  ngx_set_shutdown_timer(cycle);
>                  ngx_close_listening_sockets(cycle);
>                  ngx_close_idle_connections(cycle);
> +                ngx_event_process_posted(cycle, &ngx_posted_events);
>              }
>          }
>  

Looks good.

Just for the record:
although this change doesn't seem to have in-tree consumers
(in particular, GOAWAY is sent synchronously in HTTP/2 connections),
it allows to correctly close (in certain edge cases) QUIC connections,
and should be good in general.  A different approach to instead close
idle connections using posted next events should also work but assumes
a read on connection that may not be read ready on graceful shutdown.


From mdounin at mdounin.ru  Wed Nov 23 00:19:28 2022
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Wed, 23 Nov 2022 03:19:28 +0300
Subject: [PATCH] Process events posted by ngx_close_idle_connections()
 immediately
In-Reply-To: <c185d439cbaecfe8ec5d.1668785582@arut-laptop>
References: <c185d439cbaecfe8ec5d.1668785582@arut-laptop>
Message-ID: <Y31nEDmIEbg76VN8@mdounin.ru>

Hello!

On Fri, Nov 18, 2022 at 07:33:02PM +0400, Roman Arutyunyan wrote:

> # HG changeset patch
> # User Roman Arutyunyan <arut at nginx.com>
> # Date 1668785498 -14400
> #      Fri Nov 18 19:31:38 2022 +0400
> # Node ID c185d439cbaecfe8ec5deb0fadd1a6e9a769990b
> # Parent  17d6a537fb1bb587e4de22961bf5be5f0c648fa8
> Process events posted by ngx_close_idle_connections() immediately.
> 
> Previously, if an event was posted by a read event handler, called by
> ngx_close_idle_connections(), that event was not processed until the next
> event loop iteration, which could happen after a timeout.
> 
> diff --git a/src/os/unix/ngx_process_cycle.c b/src/os/unix/ngx_process_cycle.c
> --- a/src/os/unix/ngx_process_cycle.c
> +++ b/src/os/unix/ngx_process_cycle.c
> @@ -736,6 +736,7 @@ ngx_worker_process_cycle(ngx_cycle_t *cy
>                  ngx_set_shutdown_timer(cycle);
>                  ngx_close_listening_sockets(cycle);
>                  ngx_close_idle_connections(cycle);
> +                ngx_event_process_posted(cycle, &ngx_posted_events);
>              }
>          }
>  
> diff --git a/src/os/win32/ngx_process_cycle.c b/src/os/win32/ngx_process_cycle.c
> --- a/src/os/win32/ngx_process_cycle.c
> +++ b/src/os/win32/ngx_process_cycle.c
> @@ -804,6 +804,7 @@ ngx_worker_thread(void *data)
>                  ngx_set_shutdown_timer(cycle);
>                  ngx_close_listening_sockets(cycle);
>                  ngx_close_idle_connections(cycle);
> +                ngx_event_process_posted(cycle, &ngx_posted_events);
>              }
>          }
>  
> 

Looks good.

-- 
Maxim Dounin
http://mdounin.ru/


From arut at nginx.com  Wed Nov 23 08:48:32 2022
From: arut at nginx.com (Roman Arutyunyan)
Date: Wed, 23 Nov 2022 08:48:32 +0000
Subject: [nginx] Process events posted by ngx_close_idle_connections()
 immediately.
Message-ID: <hg.b809f53d3f5b.1669193312.6026610855610030274@dev.nginx>

details:   https://hg.nginx.org/nginx/rev/b809f53d3f5b
branches:  
changeset: 8103:b809f53d3f5b
user:      Roman Arutyunyan <arut at nginx.com>
date:      Fri Nov 18 19:31:38 2022 +0400
description:
Process events posted by ngx_close_idle_connections() immediately.

Previously, if an event was posted by a read event handler, called by
ngx_close_idle_connections(), that event was not processed until the next
event loop iteration, which could happen after a timeout.

diffstat:

 src/os/unix/ngx_process_cycle.c  |  1 +
 src/os/win32/ngx_process_cycle.c |  1 +
 2 files changed, 2 insertions(+), 0 deletions(-)

diffs (22 lines):

diff -r 49e7db44b57c -r b809f53d3f5b src/os/unix/ngx_process_cycle.c
--- a/src/os/unix/ngx_process_cycle.c	Mon Nov 21 17:01:34 2022 +0300
+++ b/src/os/unix/ngx_process_cycle.c	Fri Nov 18 19:31:38 2022 +0400
@@ -736,6 +736,7 @@ ngx_worker_process_cycle(ngx_cycle_t *cy
                 ngx_set_shutdown_timer(cycle);
                 ngx_close_listening_sockets(cycle);
                 ngx_close_idle_connections(cycle);
+                ngx_event_process_posted(cycle, &ngx_posted_events);
             }
         }
 
diff -r 49e7db44b57c -r b809f53d3f5b src/os/win32/ngx_process_cycle.c
--- a/src/os/win32/ngx_process_cycle.c	Mon Nov 21 17:01:34 2022 +0300
+++ b/src/os/win32/ngx_process_cycle.c	Fri Nov 18 19:31:38 2022 +0400
@@ -804,6 +804,7 @@ ngx_worker_thread(void *data)
                 ngx_set_shutdown_timer(cycle);
                 ngx_close_listening_sockets(cycle);
                 ngx_close_idle_connections(cycle);
+                ngx_event_process_posted(cycle, &ngx_posted_events);
             }
         }
 


From pluknet at nginx.com  Wed Nov 23 12:56:10 2022
From: pluknet at nginx.com (Sergey Kandaurov)
Date: Wed, 23 Nov 2022 16:56:10 +0400
Subject: [PATCH] Filtering duplicate addresses in listen (ticket #2400)
In-Reply-To: <55bcf8dc4ee35ccf40f5.1667097660@vm-bsd.mdounin.ru>
References: <55bcf8dc4ee35ccf40f5.1667097660@vm-bsd.mdounin.ru>
Message-ID: <20221123125610.yh4iupkprsc2twpt@Y9MQ9X2QVV>

On Sun, Oct 30, 2022 at 05:41:00AM +0300, Maxim Dounin wrote:
> # HG changeset patch
> # User Maxim Dounin <mdounin at mdounin.ru>
> # Date 1667097653 -10800
> #      Sun Oct 30 05:40:53 2022 +0300
> # Node ID 55bcf8dc4ee35ccf40f5b8a7cffde63e7edb9494
> # Parent  1ae25660c0c76edef14121ca64362f28b9d57a70
> Filtering duplicate addresses in listen (ticket #2400).
> 
> Due to the glibc bug[1], getaddrinfo("localhost") with AI_ADDRCONFIG
> on a typical host with glibc and without IPv6 returns two 127.0.0.1
> addresses, and therefore "listen localhost:80;" used to result the

result in ?

> "duplicate ... address and port pair" after 4f9b72a229c1.
> 
> Fix is to explicitly filter out duplicate addresses returned during
> resolution of a name.
> 
> [1] https://sourceware.org/bugzilla/show_bug.cgi?id=14969
> 
> diff --git a/src/http/ngx_http_core_module.c b/src/http/ngx_http_core_module.c
> --- a/src/http/ngx_http_core_module.c
> +++ b/src/http/ngx_http_core_module.c
> @@ -3963,7 +3963,7 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx
>  
>      ngx_str_t              *value, size;
>      ngx_url_t               u;
> -    ngx_uint_t              n;
> +    ngx_uint_t              n, i;
>      ngx_http_listen_opt_t   lsopt;
>  
>      cscf->listen = 1;
> @@ -4289,6 +4289,16 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx
>      }
>  
>      for (n = 0; n < u.naddrs; n++) {
> +
> +        for (i = 0; i < n; i++) {
> +            if (ngx_cmp_sockaddr(u.addrs[n].sockaddr, u.addrs[n].socklen,
> +                                 u.addrs[i].sockaddr, u.addrs[i].socklen, 0)
> +                == NGX_OK)
> +            {
> +                goto next;
> +            }
> +        }
> +
>          lsopt.sockaddr = u.addrs[n].sockaddr;
>          lsopt.socklen = u.addrs[n].socklen;
>          lsopt.addr_text = u.addrs[n].name;
> @@ -4297,6 +4307,9 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx
>          if (ngx_http_add_listen(cf, cscf, &lsopt) != NGX_OK) {
>              return NGX_CONF_ERROR;
>          }
> +
> +    next:
> +        continue;
>      }
>  
>      return NGX_CONF_OK;
> diff --git a/src/mail/ngx_mail_core_module.c b/src/mail/ngx_mail_core_module.c
> --- a/src/mail/ngx_mail_core_module.c
> +++ b/src/mail/ngx_mail_core_module.c
> @@ -308,7 +308,7 @@ ngx_mail_core_listen(ngx_conf_t *cf, ngx
>      ngx_str_t                  *value, size;
>      ngx_url_t                   u;
>      ngx_uint_t                  i, n, m;
> -    ngx_mail_listen_t          *ls, *als;
> +    ngx_mail_listen_t          *ls, *als, *nls;
>      ngx_mail_module_t          *module;
>      ngx_mail_core_main_conf_t  *cmcf;
>  
> @@ -333,7 +333,7 @@ ngx_mail_core_listen(ngx_conf_t *cf, ngx
>  
>      cmcf = ngx_mail_conf_get_module_main_conf(cf, ngx_mail_core_module);
>  
> -    ls = ngx_array_push_n(&cmcf->listen, u.naddrs);
> +    ls = ngx_array_push(&cmcf->listen);
>      if (ls == NULL) {
>          return NGX_CONF_ERROR;
>      }
> @@ -571,17 +571,37 @@ ngx_mail_core_listen(ngx_conf_t *cf, ngx
>      als = cmcf->listen.elts;

With push of additional cmcf->listen elements moved inside loop,
als can become a stale pointer due to reallocation in ngx_array_push().
As such, als assignment should be kept updated (here and in stream).

Otherwise, looks good.

>  
>      for (n = 0; n < u.naddrs; n++) {
> -        ls[n] = ls[0];
> +
> +        for (i = 0; i < n; i++) {
> +            if (ngx_cmp_sockaddr(u.addrs[n].sockaddr, u.addrs[n].socklen,
> +                                 u.addrs[i].sockaddr, u.addrs[i].socklen, 0)
> +                == NGX_OK)
> +            {
> +                goto next;
> +            }
> +        }
>  
> -        ls[n].sockaddr = u.addrs[n].sockaddr;
> -        ls[n].socklen = u.addrs[n].socklen;
> -        ls[n].addr_text = u.addrs[n].name;
> -        ls[n].wildcard = ngx_inet_wildcard(ls[n].sockaddr);
> +        if (n != 0) {
> +            nls = ngx_array_push(&cmcf->listen);
> +            if (nls == NULL) {
> +                return NGX_CONF_ERROR;
> +            }
> +
> +            *nls = *ls;
>  
> -        for (i = 0; i < cmcf->listen.nelts - u.naddrs + n; i++) {
> +        } else {
> +            nls = ls;
> +        }
> +
> +        nls->sockaddr = u.addrs[n].sockaddr;
> +        nls->socklen = u.addrs[n].socklen;
> +        nls->addr_text = u.addrs[n].name;
> +        nls->wildcard = ngx_inet_wildcard(nls->sockaddr);
> +
> +        for (i = 0; i < cmcf->listen.nelts - 1; i++) {
>  
>              if (ngx_cmp_sockaddr(als[i].sockaddr, als[i].socklen,
> -                                 ls[n].sockaddr, ls[n].socklen, 1)
> +                                 nls->sockaddr, nls->socklen, 1)
>                  != NGX_OK)
>              {
>                  continue;
> @@ -589,9 +609,12 @@ ngx_mail_core_listen(ngx_conf_t *cf, ngx
>  
>              ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
>                                 "duplicate \"%V\" address and port pair",
> -                               &ls[n].addr_text);
> +                               &nls->addr_text);
>              return NGX_CONF_ERROR;
>          }
> +
> +    next:
> +        continue;
>      }
>  
>      return NGX_CONF_OK;
> diff --git a/src/stream/ngx_stream_core_module.c b/src/stream/ngx_stream_core_module.c
> --- a/src/stream/ngx_stream_core_module.c
> +++ b/src/stream/ngx_stream_core_module.c
> @@ -578,7 +578,7 @@ ngx_stream_core_listen(ngx_conf_t *cf, n
>      ngx_str_t                    *value, size;
>      ngx_url_t                     u;
>      ngx_uint_t                    i, n, backlog;
> -    ngx_stream_listen_t          *ls, *als;
> +    ngx_stream_listen_t          *ls, *als, *nls;
>      ngx_stream_core_main_conf_t  *cmcf;
>  
>      cscf->listen = 1;
> @@ -602,7 +602,7 @@ ngx_stream_core_listen(ngx_conf_t *cf, n
>  
>      cmcf = ngx_stream_conf_get_module_main_conf(cf, ngx_stream_core_module);
>  
> -    ls = ngx_array_push_n(&cmcf->listen, u.naddrs);
> +    ls = ngx_array_push(&cmcf->listen);
>      if (ls == NULL) {
>          return NGX_CONF_ERROR;
>      }
> @@ -889,20 +889,40 @@ ngx_stream_core_listen(ngx_conf_t *cf, n
>      als = cmcf->listen.elts;
>  
>      for (n = 0; n < u.naddrs; n++) {
> -        ls[n] = ls[0];
> +
> +        for (i = 0; i < n; i++) {
> +            if (ngx_cmp_sockaddr(u.addrs[n].sockaddr, u.addrs[n].socklen,
> +                                 u.addrs[i].sockaddr, u.addrs[i].socklen, 0)
> +                == NGX_OK)
> +            {
> +                goto next;
> +            }
> +        }
>  
> -        ls[n].sockaddr = u.addrs[n].sockaddr;
> -        ls[n].socklen = u.addrs[n].socklen;
> -        ls[n].addr_text = u.addrs[n].name;
> -        ls[n].wildcard = ngx_inet_wildcard(ls[n].sockaddr);
> +        if (n != 0) {
> +            nls = ngx_array_push(&cmcf->listen);
> +            if (nls == NULL) {
> +                return NGX_CONF_ERROR;
> +            }
> +
> +            *nls = *ls;
>  
> -        for (i = 0; i < cmcf->listen.nelts - u.naddrs + n; i++) {
> -            if (ls[n].type != als[i].type) {
> +        } else {
> +            nls = ls;
> +        }
> +
> +        nls->sockaddr = u.addrs[n].sockaddr;
> +        nls->socklen = u.addrs[n].socklen;
> +        nls->addr_text = u.addrs[n].name;
> +        nls->wildcard = ngx_inet_wildcard(nls->sockaddr);
> +
> +        for (i = 0; i < cmcf->listen.nelts - 1; i++) {
> +            if (nls->type != als[i].type) {
>                  continue;
>              }
>  
>              if (ngx_cmp_sockaddr(als[i].sockaddr, als[i].socklen,
> -                                 ls[n].sockaddr, ls[n].socklen, 1)
> +                                 nls->sockaddr, nls->socklen, 1)
>                  != NGX_OK)
>              {
>                  continue;
> @@ -910,9 +930,12 @@ ngx_stream_core_listen(ngx_conf_t *cf, n
>  
>              ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
>                                 "duplicate \"%V\" address and port pair",
> -                               &ls[n].addr_text);
> +                               &nls->addr_text);
>              return NGX_CONF_ERROR;
>          }
> +
> +    next:
> +        continue;
>      }
>  
>      return NGX_CONF_OK;
> 


From pluknet at nginx.com  Wed Nov 23 13:56:01 2022
From: pluknet at nginx.com (Sergey Kandaurov)
Date: Wed, 23 Nov 2022 17:56:01 +0400
Subject: [PATCH] Disabled cloning of sockets without master process
 (ticket #2403)
In-Reply-To: <b73d95226c84b93e51f2.1667097698@vm-bsd.mdounin.ru>
References: <b73d95226c84b93e51f2.1667097698@vm-bsd.mdounin.ru>
Message-ID: <089F17A0-DD71-411D-A1F9-82134D16DE3B@nginx.com>


> On 30 Oct 2022, at 06:41, Maxim Dounin <mdounin at mdounin.ru> wrote:
> 
> # HG changeset patch
> # User Maxim Dounin <mdounin at mdounin.ru>
> # Date 1667097682 -10800
> #      Sun Oct 30 05:41:22 2022 +0300
> # Node ID b73d95226c84b93e51f23f7b35782d98d3b516b9
> # Parent  55bcf8dc4ee35ccf40f5b8a7cffde63e7edb9494
> Disabled cloning of sockets without master process (ticket #2403).
> 
> Cloning of listening sockets for each worker process does not make sense
> when working without master process, and causes some of the connections
> not to be accepted if worker_processes is set to more than one and there
> are listening sockets configured with the reuseport flag.  Fix is to
> disable cloning when master process is disabled.
> 
> diff --git a/src/event/ngx_event.c b/src/event/ngx_event.c
> --- a/src/event/ngx_event.c
> +++ b/src/event/ngx_event.c
> @@ -416,6 +416,7 @@ ngx_event_init_conf(ngx_cycle_t *cycle, 
> {
> #if (NGX_HAVE_REUSEPORT)
>     ngx_uint_t        i;
> +    ngx_core_conf_t  *ccf;
>     ngx_listening_t  *ls;
> #endif
> 
> @@ -442,7 +443,9 @@ ngx_event_init_conf(ngx_cycle_t *cycle, 
> 
> #if (NGX_HAVE_REUSEPORT)
> 
> -    if (!ngx_test_config) {
> +    ccf = (ngx_core_conf_t *) ngx_get_conf(cycle->conf_ctx, ngx_core_module);
> +
> +    if (!ngx_test_config && ccf->master) {
> 
>         ls = cycle->listening.elts;
>         for (i = 0; i < cycle->listening.nelts; i++) {

Looks good.

-- 
Sergey Kandaurov


From arut at nginx.com  Wed Nov 23 13:56:25 2022
From: arut at nginx.com (Roman Arutyunyan)
Date: Wed, 23 Nov 2022 17:56:25 +0400
Subject: [PATCH] Fixed segfault when switching off master process during
 upgrade
In-Reply-To: <ef9c94be7fe4685f0eee.1667097753@vm-bsd.mdounin.ru>
References: <ef9c94be7fe4685f0eee.1667097753@vm-bsd.mdounin.ru>
Message-ID: <20221123135625.lxe5gkuzm6gyzykj@N00W24XTQX>

Hi,

On Sun, Oct 30, 2022 at 05:42:33AM +0300, Maxim Dounin wrote:
> # HG changeset patch
> # User Maxim Dounin <mdounin at mdounin.ru>
> # Date 1667097733 -10800
> #      Sun Oct 30 05:42:13 2022 +0300
> # Node ID ef9c94be7fe4685f0eeee41f76b964ea252f519f
> # Parent  b73d95226c84b93e51f23f7b35782d98d3b516b9
> Fixed segfault when switching off master process during upgrade.
> 
> Binary upgrades are not supported without master process, but it is,
> however, possible, that nginx running with master process is asked
> to upgrade binary, and the configuration file as available on disk
> at this time includes "master_process off;".
> 
> If this happens, listening sockets inherited from the previous binary
> will have ls[i].previous set.  But the old cycle on initial process
> startup, including startup after binary upgrade, is destroyed by
> ngx_init_cycle() once configuration parsing is complete.  As a result,
> an attempt to dereference ls[i].previous in ngx_event_process_init()
> accesses already freed memory.
> 
> Fix is to avoid looking into ls[i].previous if the old cycle is already
> freed.
> 
> diff --git a/src/event/ngx_event.c b/src/event/ngx_event.c
> --- a/src/event/ngx_event.c
> +++ b/src/event/ngx_event.c
> @@ -813,7 +813,9 @@ ngx_event_process_init(ngx_cycle_t *cycl
>          rev->deferred_accept = ls[i].deferred_accept;
>  #endif
>  
> -        if (!(ngx_event_flags & NGX_USE_IOCP_EVENT)) {
> +        if (!(ngx_event_flags & NGX_USE_IOCP_EVENT)
> +            && cycle->old_cycle)
> +        {
>              if (ls[i].previous) {
>  
>                  /*

Doesn't this also mean that we can throw away zeroing ls->previous?

diff --git a/src/os/unix/ngx_process_cycle.c b/src/os/unix/ngx_process_cycle.c
--- a/src/os/unix/ngx_process_cycle.c
+++ b/src/os/unix/ngx_process_cycle.c
@@ -888,15 +888,6 @@ ngx_worker_process_init(ngx_cycle_t *cyc
     tp = ngx_timeofday();
     srandom(((unsigned) ngx_pid << 16) ^ tp->sec ^ tp->msec);
 
-    /*
-     * disable deleting previous events for the listening sockets because
-     * in the worker processes there are no events at all at this point
-     */
-    ls = cycle->listening.elts;
-    for (i = 0; i < cycle->listening.nelts; i++) {
-        ls[i].previous = NULL;
-    }
-
     for (i = 0; cycle->modules[i]; i++) {
         if (cycle->modules[i]->init_process) {
             if (cycle->modules[i]->init_process(cycle) == NGX_ERROR) {

--
Roman Arutyunyan


From mdounin at mdounin.ru  Wed Nov 23 14:46:23 2022
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Wed, 23 Nov 2022 17:46:23 +0300
Subject: [PATCH] Filtering duplicate addresses in listen (ticket #2400)
In-Reply-To: <20221123125610.yh4iupkprsc2twpt@Y9MQ9X2QVV>
References: <55bcf8dc4ee35ccf40f5.1667097660@vm-bsd.mdounin.ru>
 <20221123125610.yh4iupkprsc2twpt@Y9MQ9X2QVV>
Message-ID: <Y34yP1+SB6VAmiEg@mdounin.ru>

Hello!

On Wed, Nov 23, 2022 at 04:56:10PM +0400, Sergey Kandaurov wrote:

> On Sun, Oct 30, 2022 at 05:41:00AM +0300, Maxim Dounin wrote:
> > # HG changeset patch
> > # User Maxim Dounin <mdounin at mdounin.ru>
> > # Date 1667097653 -10800
> > #      Sun Oct 30 05:40:53 2022 +0300
> > # Node ID 55bcf8dc4ee35ccf40f5b8a7cffde63e7edb9494
> > # Parent  1ae25660c0c76edef14121ca64362f28b9d57a70
> > Filtering duplicate addresses in listen (ticket #2400).
> > 
> > Due to the glibc bug[1], getaddrinfo("localhost") with AI_ADDRCONFIG
> > on a typical host with glibc and without IPv6 returns two 127.0.0.1
> > addresses, and therefore "listen localhost:80;" used to result the
> 
> result in ?

Fixed, thnx.

> > "duplicate ... address and port pair" after 4f9b72a229c1.
> > 
> > Fix is to explicitly filter out duplicate addresses returned during
> > resolution of a name.
> > 
> > [1] https://sourceware.org/bugzilla/show_bug.cgi?id=14969
> > 
> > diff --git a/src/http/ngx_http_core_module.c b/src/http/ngx_http_core_module.c
> > --- a/src/http/ngx_http_core_module.c
> > +++ b/src/http/ngx_http_core_module.c
> > @@ -3963,7 +3963,7 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx
> >  
> >      ngx_str_t              *value, size;
> >      ngx_url_t               u;
> > -    ngx_uint_t              n;
> > +    ngx_uint_t              n, i;
> >      ngx_http_listen_opt_t   lsopt;
> >  
> >      cscf->listen = 1;
> > @@ -4289,6 +4289,16 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx
> >      }
> >  
> >      for (n = 0; n < u.naddrs; n++) {
> > +
> > +        for (i = 0; i < n; i++) {
> > +            if (ngx_cmp_sockaddr(u.addrs[n].sockaddr, u.addrs[n].socklen,
> > +                                 u.addrs[i].sockaddr, u.addrs[i].socklen, 0)
> > +                == NGX_OK)
> > +            {
> > +                goto next;
> > +            }
> > +        }
> > +
> >          lsopt.sockaddr = u.addrs[n].sockaddr;
> >          lsopt.socklen = u.addrs[n].socklen;
> >          lsopt.addr_text = u.addrs[n].name;
> > @@ -4297,6 +4307,9 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx
> >          if (ngx_http_add_listen(cf, cscf, &lsopt) != NGX_OK) {
> >              return NGX_CONF_ERROR;
> >          }
> > +
> > +    next:
> > +        continue;
> >      }
> >  
> >      return NGX_CONF_OK;
> > diff --git a/src/mail/ngx_mail_core_module.c b/src/mail/ngx_mail_core_module.c
> > --- a/src/mail/ngx_mail_core_module.c
> > +++ b/src/mail/ngx_mail_core_module.c
> > @@ -308,7 +308,7 @@ ngx_mail_core_listen(ngx_conf_t *cf, ngx
> >      ngx_str_t                  *value, size;
> >      ngx_url_t                   u;
> >      ngx_uint_t                  i, n, m;
> > -    ngx_mail_listen_t          *ls, *als;
> > +    ngx_mail_listen_t          *ls, *als, *nls;
> >      ngx_mail_module_t          *module;
> >      ngx_mail_core_main_conf_t  *cmcf;
> >  
> > @@ -333,7 +333,7 @@ ngx_mail_core_listen(ngx_conf_t *cf, ngx
> >  
> >      cmcf = ngx_mail_conf_get_module_main_conf(cf, ngx_mail_core_module);
> >  
> > -    ls = ngx_array_push_n(&cmcf->listen, u.naddrs);
> > +    ls = ngx_array_push(&cmcf->listen);
> >      if (ls == NULL) {
> >          return NGX_CONF_ERROR;
> >      }
> > @@ -571,17 +571,37 @@ ngx_mail_core_listen(ngx_conf_t *cf, ngx
> >      als = cmcf->listen.elts;
> 
> With push of additional cmcf->listen elements moved inside loop,
> als can become a stale pointer due to reallocation in ngx_array_push().
> As such, als assignment should be kept updated (here and in stream).
> 
> Otherwise, looks good.

Sure, thanks for catching this.  Updated with the following 
additional change:

diff --git a/src/mail/ngx_mail_core_module.c b/src/mail/ngx_mail_core_module.c
--- a/src/mail/ngx_mail_core_module.c
+++ b/src/mail/ngx_mail_core_module.c
@@ -568,8 +568,6 @@ ngx_mail_core_listen(ngx_conf_t *cf, ngx
         return NGX_CONF_ERROR;
     }
 
-    als = cmcf->listen.elts;
-
     for (n = 0; n < u.naddrs; n++) {
 
         for (i = 0; i < n; i++) {
@@ -598,6 +596,8 @@ ngx_mail_core_listen(ngx_conf_t *cf, ngx
         nls->addr_text = u.addrs[n].name;
         nls->wildcard = ngx_inet_wildcard(nls->sockaddr);
 
+        als = cmcf->listen.elts;
+
         for (i = 0; i < cmcf->listen.nelts - 1; i++) {
 
             if (ngx_cmp_sockaddr(als[i].sockaddr, als[i].socklen,
diff --git a/src/stream/ngx_stream_core_module.c b/src/stream/ngx_stream_core_module.c
--- a/src/stream/ngx_stream_core_module.c
+++ b/src/stream/ngx_stream_core_module.c
@@ -886,8 +886,6 @@ ngx_stream_core_listen(ngx_conf_t *cf, n
 #endif
     }
 
-    als = cmcf->listen.elts;
-
     for (n = 0; n < u.naddrs; n++) {
 
         for (i = 0; i < n; i++) {
@@ -916,6 +914,8 @@ ngx_stream_core_listen(ngx_conf_t *cf, n
         nls->addr_text = u.addrs[n].name;
         nls->wildcard = ngx_inet_wildcard(nls->sockaddr);
 
+        als = cmcf->listen.elts;
+
         for (i = 0; i < cmcf->listen.nelts - 1; i++) {
             if (nls->type != als[i].type) {
                 continue;


Full patch:

# HG changeset patch
# User Maxim Dounin <mdounin at mdounin.ru>
# Date 1669213808 -10800
#      Wed Nov 23 17:30:08 2022 +0300
# Node ID 4cc2bfeff46c7b7ee2b098f39ebfc1e44754df1e
# Parent  b809f53d3f5bd04df36ac338845289d8e60a888b
Filtering duplicate addresses in listen (ticket #2400).

Due to the glibc bug[1], getaddrinfo("localhost") with AI_ADDRCONFIG
on a typical host with glibc and without IPv6 returns two 127.0.0.1
addresses, and therefore "listen localhost:80;" used to result in
"duplicate ... address and port pair" after 4f9b72a229c1.

Fix is to explicitly filter out duplicate addresses returned during
resolution of a name.

[1] https://sourceware.org/bugzilla/show_bug.cgi?id=14969

diff --git a/src/http/ngx_http_core_module.c b/src/http/ngx_http_core_module.c
--- a/src/http/ngx_http_core_module.c
+++ b/src/http/ngx_http_core_module.c
@@ -3963,7 +3963,7 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx
 
     ngx_str_t              *value, size;
     ngx_url_t               u;
-    ngx_uint_t              n;
+    ngx_uint_t              n, i;
     ngx_http_listen_opt_t   lsopt;
 
     cscf->listen = 1;
@@ -4289,6 +4289,16 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx
     }
 
     for (n = 0; n < u.naddrs; n++) {
+
+        for (i = 0; i < n; i++) {
+            if (ngx_cmp_sockaddr(u.addrs[n].sockaddr, u.addrs[n].socklen,
+                                 u.addrs[i].sockaddr, u.addrs[i].socklen, 0)
+                == NGX_OK)
+            {
+                goto next;
+            }
+        }
+
         lsopt.sockaddr = u.addrs[n].sockaddr;
         lsopt.socklen = u.addrs[n].socklen;
         lsopt.addr_text = u.addrs[n].name;
@@ -4297,6 +4307,9 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx
         if (ngx_http_add_listen(cf, cscf, &lsopt) != NGX_OK) {
             return NGX_CONF_ERROR;
         }
+
+    next:
+        continue;
     }
 
     return NGX_CONF_OK;
diff --git a/src/mail/ngx_mail_core_module.c b/src/mail/ngx_mail_core_module.c
--- a/src/mail/ngx_mail_core_module.c
+++ b/src/mail/ngx_mail_core_module.c
@@ -308,7 +308,7 @@ ngx_mail_core_listen(ngx_conf_t *cf, ngx
     ngx_str_t                  *value, size;
     ngx_url_t                   u;
     ngx_uint_t                  i, n, m;
-    ngx_mail_listen_t          *ls, *als;
+    ngx_mail_listen_t          *ls, *als, *nls;
     ngx_mail_module_t          *module;
     ngx_mail_core_main_conf_t  *cmcf;
 
@@ -333,7 +333,7 @@ ngx_mail_core_listen(ngx_conf_t *cf, ngx
 
     cmcf = ngx_mail_conf_get_module_main_conf(cf, ngx_mail_core_module);
 
-    ls = ngx_array_push_n(&cmcf->listen, u.naddrs);
+    ls = ngx_array_push(&cmcf->listen);
     if (ls == NULL) {
         return NGX_CONF_ERROR;
     }
@@ -568,20 +568,40 @@ ngx_mail_core_listen(ngx_conf_t *cf, ngx
         return NGX_CONF_ERROR;
     }
 
-    als = cmcf->listen.elts;
+    for (n = 0; n < u.naddrs; n++) {
 
-    for (n = 0; n < u.naddrs; n++) {
-        ls[n] = ls[0];
+        for (i = 0; i < n; i++) {
+            if (ngx_cmp_sockaddr(u.addrs[n].sockaddr, u.addrs[n].socklen,
+                                 u.addrs[i].sockaddr, u.addrs[i].socklen, 0)
+                == NGX_OK)
+            {
+                goto next;
+            }
+        }
 
-        ls[n].sockaddr = u.addrs[n].sockaddr;
-        ls[n].socklen = u.addrs[n].socklen;
-        ls[n].addr_text = u.addrs[n].name;
-        ls[n].wildcard = ngx_inet_wildcard(ls[n].sockaddr);
+        if (n != 0) {
+            nls = ngx_array_push(&cmcf->listen);
+            if (nls == NULL) {
+                return NGX_CONF_ERROR;
+            }
+
+            *nls = *ls;
 
-        for (i = 0; i < cmcf->listen.nelts - u.naddrs + n; i++) {
+        } else {
+            nls = ls;
+        }
+
+        nls->sockaddr = u.addrs[n].sockaddr;
+        nls->socklen = u.addrs[n].socklen;
+        nls->addr_text = u.addrs[n].name;
+        nls->wildcard = ngx_inet_wildcard(nls->sockaddr);
+
+        als = cmcf->listen.elts;
+
+        for (i = 0; i < cmcf->listen.nelts - 1; i++) {
 
             if (ngx_cmp_sockaddr(als[i].sockaddr, als[i].socklen,
-                                 ls[n].sockaddr, ls[n].socklen, 1)
+                                 nls->sockaddr, nls->socklen, 1)
                 != NGX_OK)
             {
                 continue;
@@ -589,9 +609,12 @@ ngx_mail_core_listen(ngx_conf_t *cf, ngx
 
             ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                                "duplicate \"%V\" address and port pair",
-                               &ls[n].addr_text);
+                               &nls->addr_text);
             return NGX_CONF_ERROR;
         }
+
+    next:
+        continue;
     }
 
     return NGX_CONF_OK;
diff --git a/src/stream/ngx_stream_core_module.c b/src/stream/ngx_stream_core_module.c
--- a/src/stream/ngx_stream_core_module.c
+++ b/src/stream/ngx_stream_core_module.c
@@ -578,7 +578,7 @@ ngx_stream_core_listen(ngx_conf_t *cf, n
     ngx_str_t                    *value, size;
     ngx_url_t                     u;
     ngx_uint_t                    i, n, backlog;
-    ngx_stream_listen_t          *ls, *als;
+    ngx_stream_listen_t          *ls, *als, *nls;
     ngx_stream_core_main_conf_t  *cmcf;
 
     cscf->listen = 1;
@@ -602,7 +602,7 @@ ngx_stream_core_listen(ngx_conf_t *cf, n
 
     cmcf = ngx_stream_conf_get_module_main_conf(cf, ngx_stream_core_module);
 
-    ls = ngx_array_push_n(&cmcf->listen, u.naddrs);
+    ls = ngx_array_push(&cmcf->listen);
     if (ls == NULL) {
         return NGX_CONF_ERROR;
     }
@@ -886,23 +886,43 @@ ngx_stream_core_listen(ngx_conf_t *cf, n
 #endif
     }
 
-    als = cmcf->listen.elts;
+    for (n = 0; n < u.naddrs; n++) {
 
-    for (n = 0; n < u.naddrs; n++) {
-        ls[n] = ls[0];
+        for (i = 0; i < n; i++) {
+            if (ngx_cmp_sockaddr(u.addrs[n].sockaddr, u.addrs[n].socklen,
+                                 u.addrs[i].sockaddr, u.addrs[i].socklen, 0)
+                == NGX_OK)
+            {
+                goto next;
+            }
+        }
 
-        ls[n].sockaddr = u.addrs[n].sockaddr;
-        ls[n].socklen = u.addrs[n].socklen;
-        ls[n].addr_text = u.addrs[n].name;
-        ls[n].wildcard = ngx_inet_wildcard(ls[n].sockaddr);
+        if (n != 0) {
+            nls = ngx_array_push(&cmcf->listen);
+            if (nls == NULL) {
+                return NGX_CONF_ERROR;
+            }
+
+            *nls = *ls;
 
-        for (i = 0; i < cmcf->listen.nelts - u.naddrs + n; i++) {
-            if (ls[n].type != als[i].type) {
+        } else {
+            nls = ls;
+        }
+
+        nls->sockaddr = u.addrs[n].sockaddr;
+        nls->socklen = u.addrs[n].socklen;
+        nls->addr_text = u.addrs[n].name;
+        nls->wildcard = ngx_inet_wildcard(nls->sockaddr);
+
+        als = cmcf->listen.elts;
+
+        for (i = 0; i < cmcf->listen.nelts - 1; i++) {
+            if (nls->type != als[i].type) {
                 continue;
             }
 
             if (ngx_cmp_sockaddr(als[i].sockaddr, als[i].socklen,
-                                 ls[n].sockaddr, ls[n].socklen, 1)
+                                 nls->sockaddr, nls->socklen, 1)
                 != NGX_OK)
             {
                 continue;
@@ -910,9 +930,12 @@ ngx_stream_core_listen(ngx_conf_t *cf, n
 
             ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                                "duplicate \"%V\" address and port pair",
-                               &ls[n].addr_text);
+                               &nls->addr_text);
             return NGX_CONF_ERROR;
         }
+
+    next:
+        continue;
     }
 
     return NGX_CONF_OK;

-- 
Maxim Dounin
http://mdounin.ru/


From pluknet at nginx.com  Wed Nov 23 15:03:32 2022
From: pluknet at nginx.com (Sergey Kandaurov)
Date: Wed, 23 Nov 2022 19:03:32 +0400
Subject: [PATCH] Filtering duplicate addresses in listen (ticket #2400)
In-Reply-To: <Y34yP1+SB6VAmiEg@mdounin.ru>
References: <55bcf8dc4ee35ccf40f5.1667097660@vm-bsd.mdounin.ru>
 <20221123125610.yh4iupkprsc2twpt@Y9MQ9X2QVV> <Y34yP1+SB6VAmiEg@mdounin.ru>
Message-ID: <1E74A87E-A48E-432C-AB33-FA6BFC51157B@nginx.com>


> On 23 Nov 2022, at 18:46, Maxim Dounin <mdounin at mdounin.ru> wrote:
> 
> Hello!
> 
> On Wed, Nov 23, 2022 at 04:56:10PM +0400, Sergey Kandaurov wrote:
> 
>> On Sun, Oct 30, 2022 at 05:41:00AM +0300, Maxim Dounin wrote:
>>> # HG changeset patch
>>> # User Maxim Dounin <mdounin at mdounin.ru>
>>> # Date 1667097653 -10800
>>> #      Sun Oct 30 05:40:53 2022 +0300
>>> # Node ID 55bcf8dc4ee35ccf40f5b8a7cffde63e7edb9494
>>> # Parent  1ae25660c0c76edef14121ca64362f28b9d57a70
>>> Filtering duplicate addresses in listen (ticket #2400).
>>> 
>>> Due to the glibc bug[1], getaddrinfo("localhost") with AI_ADDRCONFIG
>>> on a typical host with glibc and without IPv6 returns two 127.0.0.1
>>> addresses, and therefore "listen localhost:80;" used to result the
>> 
>> result in ?
> 
> Fixed, thnx.
> 
>>> "duplicate ... address and port pair" after 4f9b72a229c1.
>>> 
>>> Fix is to explicitly filter out duplicate addresses returned during
>>> resolution of a name.
>>> 
>>> [1] https://sourceware.org/bugzilla/show_bug.cgi?id=14969
>>> 
>>> diff --git a/src/http/ngx_http_core_module.c b/src/http/ngx_http_core_module.c
>>> --- a/src/http/ngx_http_core_module.c
>>> +++ b/src/http/ngx_http_core_module.c
>>> @@ -3963,7 +3963,7 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx
>>> 
>>>     ngx_str_t              *value, size;
>>>     ngx_url_t               u;
>>> -    ngx_uint_t              n;
>>> +    ngx_uint_t              n, i;
>>>     ngx_http_listen_opt_t   lsopt;
>>> 
>>>     cscf->listen = 1;
>>> @@ -4289,6 +4289,16 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx
>>>     }
>>> 
>>>     for (n = 0; n < u.naddrs; n++) {
>>> +
>>> +        for (i = 0; i < n; i++) {
>>> +            if (ngx_cmp_sockaddr(u.addrs[n].sockaddr, u.addrs[n].socklen,
>>> +                                 u.addrs[i].sockaddr, u.addrs[i].socklen, 0)
>>> +                == NGX_OK)
>>> +            {
>>> +                goto next;
>>> +            }
>>> +        }
>>> +
>>>         lsopt.sockaddr = u.addrs[n].sockaddr;
>>>         lsopt.socklen = u.addrs[n].socklen;
>>>         lsopt.addr_text = u.addrs[n].name;
>>> @@ -4297,6 +4307,9 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx
>>>         if (ngx_http_add_listen(cf, cscf, &lsopt) != NGX_OK) {
>>>             return NGX_CONF_ERROR;
>>>         }
>>> +
>>> +    next:
>>> +        continue;
>>>     }
>>> 
>>>     return NGX_CONF_OK;
>>> diff --git a/src/mail/ngx_mail_core_module.c b/src/mail/ngx_mail_core_module.c
>>> --- a/src/mail/ngx_mail_core_module.c
>>> +++ b/src/mail/ngx_mail_core_module.c
>>> @@ -308,7 +308,7 @@ ngx_mail_core_listen(ngx_conf_t *cf, ngx
>>>     ngx_str_t                  *value, size;
>>>     ngx_url_t                   u;
>>>     ngx_uint_t                  i, n, m;
>>> -    ngx_mail_listen_t          *ls, *als;
>>> +    ngx_mail_listen_t          *ls, *als, *nls;
>>>     ngx_mail_module_t          *module;
>>>     ngx_mail_core_main_conf_t  *cmcf;
>>> 
>>> @@ -333,7 +333,7 @@ ngx_mail_core_listen(ngx_conf_t *cf, ngx
>>> 
>>>     cmcf = ngx_mail_conf_get_module_main_conf(cf, ngx_mail_core_module);
>>> 
>>> -    ls = ngx_array_push_n(&cmcf->listen, u.naddrs);
>>> +    ls = ngx_array_push(&cmcf->listen);
>>>     if (ls == NULL) {
>>>         return NGX_CONF_ERROR;
>>>     }
>>> @@ -571,17 +571,37 @@ ngx_mail_core_listen(ngx_conf_t *cf, ngx
>>>     als = cmcf->listen.elts;
>> 
>> With push of additional cmcf->listen elements moved inside loop,
>> als can become a stale pointer due to reallocation in ngx_array_push().
>> As such, als assignment should be kept updated (here and in stream).
>> 
>> Otherwise, looks good.
> 
> Sure, thanks for catching this.  Updated with the following 
> additional change:
> 
> diff --git a/src/mail/ngx_mail_core_module.c b/src/mail/ngx_mail_core_module.c
> --- a/src/mail/ngx_mail_core_module.c
> +++ b/src/mail/ngx_mail_core_module.c
> @@ -568,8 +568,6 @@ ngx_mail_core_listen(ngx_conf_t *cf, ngx
>         return NGX_CONF_ERROR;
>     }
> 
> -    als = cmcf->listen.elts;
> -
>     for (n = 0; n < u.naddrs; n++) {
> 
>         for (i = 0; i < n; i++) {
> @@ -598,6 +596,8 @@ ngx_mail_core_listen(ngx_conf_t *cf, ngx
>         nls->addr_text = u.addrs[n].name;
>         nls->wildcard = ngx_inet_wildcard(nls->sockaddr);
> 
> +        als = cmcf->listen.elts;
> +
>         for (i = 0; i < cmcf->listen.nelts - 1; i++) {
> 
>             if (ngx_cmp_sockaddr(als[i].sockaddr, als[i].socklen,
> diff --git a/src/stream/ngx_stream_core_module.c b/src/stream/ngx_stream_core_module.c
> --- a/src/stream/ngx_stream_core_module.c
> +++ b/src/stream/ngx_stream_core_module.c
> @@ -886,8 +886,6 @@ ngx_stream_core_listen(ngx_conf_t *cf, n
> #endif
>     }
> 
> -    als = cmcf->listen.elts;
> -
>     for (n = 0; n < u.naddrs; n++) {
> 
>         for (i = 0; i < n; i++) {
> @@ -916,6 +914,8 @@ ngx_stream_core_listen(ngx_conf_t *cf, n
>         nls->addr_text = u.addrs[n].name;
>         nls->wildcard = ngx_inet_wildcard(nls->sockaddr);
> 
> +        als = cmcf->listen.elts;
> +
>         for (i = 0; i < cmcf->listen.nelts - 1; i++) {
>             if (nls->type != als[i].type) {
>                 continue;
> 
> 
> Full patch:
> 
> # HG changeset patch
> # User Maxim Dounin <mdounin at mdounin.ru>
> # Date 1669213808 -10800
> #      Wed Nov 23 17:30:08 2022 +0300
> # Node ID 4cc2bfeff46c7b7ee2b098f39ebfc1e44754df1e
> # Parent  b809f53d3f5bd04df36ac338845289d8e60a888b
> Filtering duplicate addresses in listen (ticket #2400).
> 
> Due to the glibc bug[1], getaddrinfo("localhost") with AI_ADDRCONFIG
> on a typical host with glibc and without IPv6 returns two 127.0.0.1
> addresses, and therefore "listen localhost:80;" used to result in
> "duplicate ... address and port pair" after 4f9b72a229c1.
> 
> Fix is to explicitly filter out duplicate addresses returned during
> resolution of a name.
> 
> [1] https://sourceware.org/bugzilla/show_bug.cgi?id=14969
> 

[..]

Looks good.

-- 
Sergey Kandaurov


From vl at inspert.ru  Wed Nov 23 17:50:54 2022
From: vl at inspert.ru (Vladimir Homutov)
Date: Wed, 23 Nov 2022 20:50:54 +0300
Subject: [patch] ngx_cpp_test module build issue cleanup
Message-ID: <Y35dfmxg4WVcyuUo@vlws>

Hello,

the simplest ./configure --with-cpp_test_module leads to build error
after successful configuration:

src/misc/ngx_cpp_test_module.cpp:13:12: fatal error: ngx_mail.h: No such file or directory
   13 |   #include <ngx_mail.h>
      |            ^~~~~~~~~~~~
compilation terminated.


# HG changeset patch
# User Vladimir Khomutov <vl at inspert.ru>
# Date 1669225034 -10800
#      Wed Nov 23 20:37:14 2022 +0300
# Node ID 6237563c81707c8c2453cb0a7509ddaf64c02f4e
# Parent  49e7db44b57c9f4d54b87d19a696178b913aec5c
The ngx_cpp_test_module build requires mail and stream.

# HG changeset patch
# User Vladimir Khomutov <vl at inspert.ru>
# Date 1669225742 -10800
#      Wed Nov 23 20:49:02 2022 +0300
# Node ID 12c04127e3fe4d6aa689ef3bcf3ae0834e7e9ed5
# Parent  b809f53d3f5bd04df36ac338845289d8e60a888b
The ngx_cpp_test_module build requires mail and stream.

diff --git a/auto/modules b/auto/modules
--- a/auto/modules
+++ b/auto/modules
@@ -1358,6 +1358,17 @@ if [ $NGX_GOOGLE_PERFTOOLS = YES ]; then
 fi

 if [ $NGX_CPP_TEST = YES ]; then
+
+    if [ $MAIL = NO ]; then
+        echo "$0: error: ngx_cpp_test_module assumes \"--with-mail\""
+        exit 1
+    fi
+
+    if [ $STREAM = NO ]; then
+        echo "$0: error: ngx_cpp_test_module assumes \"--with-stream\""
+        exit 1
+    fi
+
     ngx_module_name=
     ngx_module_incs=
     ngx_module_deps=


From mdounin at mdounin.ru  Wed Nov 23 20:52:00 2022
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Wed, 23 Nov 2022 23:52:00 +0300
Subject: [PATCH] Fixed segfault when switching off master process during
 upgrade
In-Reply-To: <20221123135625.lxe5gkuzm6gyzykj@N00W24XTQX>
References: <ef9c94be7fe4685f0eee.1667097753@vm-bsd.mdounin.ru>
 <20221123135625.lxe5gkuzm6gyzykj@N00W24XTQX>
Message-ID: <Y36H8Dn/ZUOttPPk@mdounin.ru>

Hello!

On Wed, Nov 23, 2022 at 05:56:25PM +0400, Roman Arutyunyan wrote:

> Hi,
> 
> On Sun, Oct 30, 2022 at 05:42:33AM +0300, Maxim Dounin wrote:
> > # HG changeset patch
> > # User Maxim Dounin <mdounin at mdounin.ru>
> > # Date 1667097733 -10800
> > #      Sun Oct 30 05:42:13 2022 +0300
> > # Node ID ef9c94be7fe4685f0eeee41f76b964ea252f519f
> > # Parent  b73d95226c84b93e51f23f7b35782d98d3b516b9
> > Fixed segfault when switching off master process during upgrade.
> > 
> > Binary upgrades are not supported without master process, but it is,
> > however, possible, that nginx running with master process is asked
> > to upgrade binary, and the configuration file as available on disk
> > at this time includes "master_process off;".
> > 
> > If this happens, listening sockets inherited from the previous binary
> > will have ls[i].previous set.  But the old cycle on initial process
> > startup, including startup after binary upgrade, is destroyed by
> > ngx_init_cycle() once configuration parsing is complete.  As a result,
> > an attempt to dereference ls[i].previous in ngx_event_process_init()
> > accesses already freed memory.
> > 
> > Fix is to avoid looking into ls[i].previous if the old cycle is already
> > freed.
> > 
> > diff --git a/src/event/ngx_event.c b/src/event/ngx_event.c
> > --- a/src/event/ngx_event.c
> > +++ b/src/event/ngx_event.c
> > @@ -813,7 +813,9 @@ ngx_event_process_init(ngx_cycle_t *cycl
> >          rev->deferred_accept = ls[i].deferred_accept;
> >  #endif
> >  
> > -        if (!(ngx_event_flags & NGX_USE_IOCP_EVENT)) {
> > +        if (!(ngx_event_flags & NGX_USE_IOCP_EVENT)
> > +            && cycle->old_cycle)
> > +        {
> >              if (ls[i].previous) {
> >  
> >                  /*
> 
> Doesn't this also mean that we can throw away zeroing ls->previous?
> 
> diff --git a/src/os/unix/ngx_process_cycle.c b/src/os/unix/ngx_process_cycle.c
> --- a/src/os/unix/ngx_process_cycle.c
> +++ b/src/os/unix/ngx_process_cycle.c
> @@ -888,15 +888,6 @@ ngx_worker_process_init(ngx_cycle_t *cyc
>      tp = ngx_timeofday();
>      srandom(((unsigned) ngx_pid << 16) ^ tp->sec ^ tp->msec);
>  
> -    /*
> -     * disable deleting previous events for the listening sockets because
> -     * in the worker processes there are no events at all at this point
> -     */
> -    ls = cycle->listening.elts;
> -    for (i = 0; i < cycle->listening.nelts; i++) {
> -        ls[i].previous = NULL;
> -    }
> -
>      for (i = 0; cycle->modules[i]; i++) {
>          if (cycle->modules[i]->init_process) {
>              if (cycle->modules[i]->init_process(cycle) == NGX_ERROR) {
> 

Yes, this code is now redundant.  Added to the patch, with the 
following paragraph in the commit log:

: With this change it is also no longer needed to clear ls[i].previous in
: worker processes, so the relevant code was removed.

Full patch:

# HG changeset patch
# User Maxim Dounin <mdounin at mdounin.ru>
# Date 1669236533 -10800
#      Wed Nov 23 23:48:53 2022 +0300
# Node ID 8852f39311de9913435fb2f2fc601a3655f42a99
# Parent  09463dd9c50463e35856a3bd6b57bc6ca382e6f0
Fixed segfault when switching off master process during upgrade.

Binary upgrades are not supported without master process, but it is,
however, possible, that nginx running with master process is asked
to upgrade binary, and the configuration file as available on disk
at this time includes "master_process off;".

If this happens, listening sockets inherited from the previous binary
will have ls[i].previous set.  But the old cycle on initial process
startup, including startup after binary upgrade, is destroyed by
ngx_init_cycle() once configuration parsing is complete.  As a result,
an attempt to dereference ls[i].previous in ngx_event_process_init()
accesses already freed memory.

Fix is to avoid looking into ls[i].previous if the old cycle is already
freed.

With this change it is also no longer needed to clear ls[i].previous in
worker processes, so the relevant code was removed.

diff --git a/src/event/ngx_event.c b/src/event/ngx_event.c
--- a/src/event/ngx_event.c
+++ b/src/event/ngx_event.c
@@ -813,7 +813,9 @@ ngx_event_process_init(ngx_cycle_t *cycl
         rev->deferred_accept = ls[i].deferred_accept;
 #endif
 
-        if (!(ngx_event_flags & NGX_USE_IOCP_EVENT)) {
+        if (!(ngx_event_flags & NGX_USE_IOCP_EVENT)
+            && cycle->old_cycle)
+        {
             if (ls[i].previous) {
 
                 /*
diff --git a/src/os/unix/ngx_process_cycle.c b/src/os/unix/ngx_process_cycle.c
--- a/src/os/unix/ngx_process_cycle.c
+++ b/src/os/unix/ngx_process_cycle.c
@@ -759,7 +759,6 @@ ngx_worker_process_init(ngx_cycle_t *cyc
     ngx_cpuset_t     *cpu_affinity;
     struct rlimit     rlmt;
     ngx_core_conf_t  *ccf;
-    ngx_listening_t  *ls;
 
     if (ngx_set_environment(cycle, NULL) == NULL) {
         /* fatal */
@@ -889,15 +888,6 @@ ngx_worker_process_init(ngx_cycle_t *cyc
     tp = ngx_timeofday();
     srandom(((unsigned) ngx_pid << 16) ^ tp->sec ^ tp->msec);
 
-    /*
-     * disable deleting previous events for the listening sockets because
-     * in the worker processes there are no events at all at this point
-     */
-    ls = cycle->listening.elts;
-    for (i = 0; i < cycle->listening.nelts; i++) {
-        ls[i].previous = NULL;
-    }
-
     for (i = 0; cycle->modules[i]; i++) {
         if (cycle->modules[i]->init_process) {
             if (cycle->modules[i]->init_process(cycle) == NGX_ERROR) {

-- 
Maxim Dounin
http://mdounin.ru/


From arut at nginx.com  Thu Nov 24 09:09:41 2022
From: arut at nginx.com (Roman Arutyunyan)
Date: Thu, 24 Nov 2022 13:09:41 +0400
Subject: [PATCH] Fixed segfault when switching off master process during
 upgrade
In-Reply-To: <Y36H8Dn/ZUOttPPk@mdounin.ru>
References: <ef9c94be7fe4685f0eee.1667097753@vm-bsd.mdounin.ru>
 <20221123135625.lxe5gkuzm6gyzykj@N00W24XTQX>
 <Y36H8Dn/ZUOttPPk@mdounin.ru>
Message-ID: <20221124090941.cpt6fxve2ce7qcye@N00W24XTQX>

Hi,

On Wed, Nov 23, 2022 at 11:52:00PM +0300, Maxim Dounin wrote:
> Hello!
> 
> On Wed, Nov 23, 2022 at 05:56:25PM +0400, Roman Arutyunyan wrote:
> 
> > Hi,
> > 
> > On Sun, Oct 30, 2022 at 05:42:33AM +0300, Maxim Dounin wrote:
> > > # HG changeset patch
> > > # User Maxim Dounin <mdounin at mdounin.ru>
> > > # Date 1667097733 -10800
> > > #      Sun Oct 30 05:42:13 2022 +0300
> > > # Node ID ef9c94be7fe4685f0eeee41f76b964ea252f519f
> > > # Parent  b73d95226c84b93e51f23f7b35782d98d3b516b9
> > > Fixed segfault when switching off master process during upgrade.
> > > 
> > > Binary upgrades are not supported without master process, but it is,
> > > however, possible, that nginx running with master process is asked
> > > to upgrade binary, and the configuration file as available on disk
> > > at this time includes "master_process off;".
> > > 
> > > If this happens, listening sockets inherited from the previous binary
> > > will have ls[i].previous set.  But the old cycle on initial process
> > > startup, including startup after binary upgrade, is destroyed by
> > > ngx_init_cycle() once configuration parsing is complete.  As a result,
> > > an attempt to dereference ls[i].previous in ngx_event_process_init()
> > > accesses already freed memory.
> > > 
> > > Fix is to avoid looking into ls[i].previous if the old cycle is already
> > > freed.
> > > 
> > > diff --git a/src/event/ngx_event.c b/src/event/ngx_event.c
> > > --- a/src/event/ngx_event.c
> > > +++ b/src/event/ngx_event.c
> > > @@ -813,7 +813,9 @@ ngx_event_process_init(ngx_cycle_t *cycl
> > >          rev->deferred_accept = ls[i].deferred_accept;
> > >  #endif
> > >  
> > > -        if (!(ngx_event_flags & NGX_USE_IOCP_EVENT)) {
> > > +        if (!(ngx_event_flags & NGX_USE_IOCP_EVENT)
> > > +            && cycle->old_cycle)
> > > +        {
> > >              if (ls[i].previous) {
> > >  
> > >                  /*
> > 
> > Doesn't this also mean that we can throw away zeroing ls->previous?
> > 
> > diff --git a/src/os/unix/ngx_process_cycle.c b/src/os/unix/ngx_process_cycle.c
> > --- a/src/os/unix/ngx_process_cycle.c
> > +++ b/src/os/unix/ngx_process_cycle.c
> > @@ -888,15 +888,6 @@ ngx_worker_process_init(ngx_cycle_t *cyc
> >      tp = ngx_timeofday();
> >      srandom(((unsigned) ngx_pid << 16) ^ tp->sec ^ tp->msec);
> >  
> > -    /*
> > -     * disable deleting previous events for the listening sockets because
> > -     * in the worker processes there are no events at all at this point
> > -     */
> > -    ls = cycle->listening.elts;
> > -    for (i = 0; i < cycle->listening.nelts; i++) {
> > -        ls[i].previous = NULL;
> > -    }
> > -
> >      for (i = 0; cycle->modules[i]; i++) {
> >          if (cycle->modules[i]->init_process) {
> >              if (cycle->modules[i]->init_process(cycle) == NGX_ERROR) {
> > 
> 
> Yes, this code is now redundant.  Added to the patch, with the 
> following paragraph in the commit log:
> 
> : With this change it is also no longer needed to clear ls[i].previous in
> : worker processes, so the relevant code was removed.
> 
> Full patch:
> 
> # HG changeset patch
> # User Maxim Dounin <mdounin at mdounin.ru>
> # Date 1669236533 -10800
> #      Wed Nov 23 23:48:53 2022 +0300
> # Node ID 8852f39311de9913435fb2f2fc601a3655f42a99
> # Parent  09463dd9c50463e35856a3bd6b57bc6ca382e6f0
> Fixed segfault when switching off master process during upgrade.
> 
> Binary upgrades are not supported without master process, but it is,
> however, possible, that nginx running with master process is asked
> to upgrade binary, and the configuration file as available on disk
> at this time includes "master_process off;".
> 
> If this happens, listening sockets inherited from the previous binary
> will have ls[i].previous set.  But the old cycle on initial process
> startup, including startup after binary upgrade, is destroyed by
> ngx_init_cycle() once configuration parsing is complete.  As a result,
> an attempt to dereference ls[i].previous in ngx_event_process_init()
> accesses already freed memory.
> 
> Fix is to avoid looking into ls[i].previous if the old cycle is already
> freed.
> 
> With this change it is also no longer needed to clear ls[i].previous in
> worker processes, so the relevant code was removed.
> 
> diff --git a/src/event/ngx_event.c b/src/event/ngx_event.c
> --- a/src/event/ngx_event.c
> +++ b/src/event/ngx_event.c
> @@ -813,7 +813,9 @@ ngx_event_process_init(ngx_cycle_t *cycl
>          rev->deferred_accept = ls[i].deferred_accept;
>  #endif
>  
> -        if (!(ngx_event_flags & NGX_USE_IOCP_EVENT)) {
> +        if (!(ngx_event_flags & NGX_USE_IOCP_EVENT)
> +            && cycle->old_cycle)
> +        {
>              if (ls[i].previous) {
>  
>                  /*
> diff --git a/src/os/unix/ngx_process_cycle.c b/src/os/unix/ngx_process_cycle.c
> --- a/src/os/unix/ngx_process_cycle.c
> +++ b/src/os/unix/ngx_process_cycle.c
> @@ -759,7 +759,6 @@ ngx_worker_process_init(ngx_cycle_t *cyc
>      ngx_cpuset_t     *cpu_affinity;
>      struct rlimit     rlmt;
>      ngx_core_conf_t  *ccf;
> -    ngx_listening_t  *ls;
>  
>      if (ngx_set_environment(cycle, NULL) == NULL) {
>          /* fatal */
> @@ -889,15 +888,6 @@ ngx_worker_process_init(ngx_cycle_t *cyc
>      tp = ngx_timeofday();
>      srandom(((unsigned) ngx_pid << 16) ^ tp->sec ^ tp->msec);
>  
> -    /*
> -     * disable deleting previous events for the listening sockets because
> -     * in the worker processes there are no events at all at this point
> -     */
> -    ls = cycle->listening.elts;
> -    for (i = 0; i < cycle->listening.nelts; i++) {
> -        ls[i].previous = NULL;
> -    }
> -
>      for (i = 0; cycle->modules[i]; i++) {
>          if (cycle->modules[i]->init_process) {
>              if (cycle->modules[i]->init_process(cycle) == NGX_ERROR) {

Looks good.

--
Roman Arutyunyan


From pluknet at nginx.com  Thu Nov 24 09:21:51 2022
From: pluknet at nginx.com (Sergey Kandaurov)
Date: Thu, 24 Nov 2022 13:21:51 +0400
Subject: [PATCH 2 of 2] SSL: SSL_sendfile() support with kernel TLS
In-Reply-To: <ff514bf17f7f2257dcf0.1632748688@vm-bsd.mdounin.ru>
References: <patchbomb.1632748686@vm-bsd.mdounin.ru>
 <ff514bf17f7f2257dcf0.1632748688@vm-bsd.mdounin.ru>
Message-ID: <DF6E1511-F69C-4AD8-88B7-B011E6A163E1@nginx.com>


> On 27 Sep 2021, at 17:18, Maxim Dounin <mdounin at mdounin.ru> wrote:
> 
> # HG changeset patch
> # User Maxim Dounin <mdounin at mdounin.ru>
> # Date 1632717779 -10800
> #      Mon Sep 27 07:42:59 2021 +0300
> # Node ID ff514bf17f7f2257dcf036c5c973b74672cefa9a
> # Parent  8f0fd60c33c106fba5f1ce3cafe990f15fcccc0c
> SSL: SSL_sendfile() support with kernel TLS.
> 
> Requires OpenSSL 3.0 compiled with "enable-ktls" option.  Further, KTLS
> needs to be enabled in kernel, and in OpenSSL, either via OpenSSL
> configuration file or with "ssl_conf_command Options KTLS;" in nginx
> configuration.
> 
> On FreeBSD, kernel TLS is available starting with FreeBSD 13.0, and
> can be enabled with "sysctl kern.ipc.tls.enable=1" and "kldload ktls_ocf".
> 
> On Linux, kernel TLS is available starting with kernel 4.13 (at least 5.2
> is recommended), and needs kernel compiled with CONFIG_TLS=y (with
> CONFIG_TLS=m, which is used at least on Ubuntu 21.04 by default,
> the tls module needs to be loaded with "modprobe tls").
> 
> diff --git a/src/event/ngx_event_openssl.c b/src/event/ngx_event_openssl.c

[..]

> @@ -2882,6 +2937,150 @@ ngx_ssl_write_early(ngx_connection_t *c,
> #endif
> 
> 
> +static ssize_t
> +ngx_ssl_sendfile(ngx_connection_t *c, ngx_buf_t *file, size_t size)
> +{
> +#ifdef BIO_get_ktls_send
> +
> +    int        sslerr;
> +    ssize_t    n;
> +    ngx_err_t  err;
> +
> +    ngx_ssl_clear_error(c->log);
> +
> +    ngx_log_debug2(NGX_LOG_DEBUG_EVENT, c->log, 0,
> +                   "SSL to sendfile: @%O %uz",
> +                   file->file_pos, size);
> +
> +    ngx_set_errno(0);
> +
> +    n = SSL_sendfile(c->ssl->connection, file->file->fd, file->file_pos,
> +                     size, 0);
> +
> +    ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0, "SSL_sendfile: %d", n);
> +

Hello,

looks like a wrong format specifier slipped through review.
Patch to address this:

# HG changeset patch
# User Sergey Kandaurov <pluknet at nginx.com>
# Date 1669241969 -14400
#      Thu Nov 24 02:19:29 2022 +0400
# Node ID 66ad8bd4b9e4347bda8fe64423632ded19093db2
# Parent  75bfb3a97ca3d6292fb7877b89bb46720c15da80
SSL: fixed debug logging of SSL_sendfile() return value.

diff --git a/src/event/ngx_event_openssl.c b/src/event/ngx_event_openssl.c
--- a/src/event/ngx_event_openssl.c
+++ b/src/event/ngx_event_openssl.c
@@ -3020,7 +3020,7 @@ ngx_ssl_sendfile(ngx_connection_t *c, ng
     n = SSL_sendfile(c->ssl->connection, file->file->fd, file->file_pos,
                      size, flags);
 
-    ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0, "SSL_sendfile: %d", n);
+    ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0, "SSL_sendfile: %z", n);
 
     if (n > 0) {
 

> [..]

-- 
Sergey Kandaurov


From pluknet at nginx.com  Thu Nov 24 09:25:30 2022
From: pluknet at nginx.com (Sergey Kandaurov)
Date: Thu, 24 Nov 2022 13:25:30 +0400
Subject: [patch] ngx_cpp_test module build issue cleanup
In-Reply-To: <Y35dfmxg4WVcyuUo@vlws>
References: <Y35dfmxg4WVcyuUo@vlws>
Message-ID: <6FBB2B8F-A082-4EC6-8C00-98F9A5DBA663@nginx.com>


> On 23 Nov 2022, at 21:50, Vladimir Homutov via nginx-devel <nginx-devel at nginx.org> wrote:
> 
> Hello,
> 
> the simplest ./configure --with-cpp_test_module leads to build error
> after successful configuration:
> 
> src/misc/ngx_cpp_test_module.cpp:13:12: fatal error: ngx_mail.h: No such file or directory
>   13 |   #include <ngx_mail.h>
>      |            ^~~~~~~~~~~~
> compilation terminated.
> 
> 
> # HG changeset patch
> # User Vladimir Khomutov <vl at inspert.ru>
> # Date 1669225034 -10800
> #      Wed Nov 23 20:37:14 2022 +0300
> # Node ID 6237563c81707c8c2453cb0a7509ddaf64c02f4e
> # Parent  49e7db44b57c9f4d54b87d19a696178b913aec5c
> The ngx_cpp_test_module build requires mail and stream.
> 
> # HG changeset patch
> # User Vladimir Khomutov <vl at inspert.ru>
> # Date 1669225742 -10800
> #      Wed Nov 23 20:49:02 2022 +0300
> # Node ID 12c04127e3fe4d6aa689ef3bcf3ae0834e7e9ed5
> # Parent  b809f53d3f5bd04df36ac338845289d8e60a888b
> The ngx_cpp_test_module build requires mail and stream.
> 
> diff --git a/auto/modules b/auto/modules
> --- a/auto/modules
> +++ b/auto/modules
> @@ -1358,6 +1358,17 @@ if [ $NGX_GOOGLE_PERFTOOLS = YES ]; then
> fi
> 
> if [ $NGX_CPP_TEST = YES ]; then
> +
> +    if [ $MAIL = NO ]; then
> +        echo "$0: error: ngx_cpp_test_module assumes \"--with-mail\""
> +        exit 1
> +    fi
> +
> +    if [ $STREAM = NO ]; then
> +        echo "$0: error: ngx_cpp_test_module assumes \"--with-stream\""
> +        exit 1
> +    fi
> +
>     ngx_module_name=
>     ngx_module_incs=
>     ngx_module_deps=
> 

Hello,

if at all try to fix it,
--without-http would also need to be addressed.

-- 
Sergey Kandaurov


From vl at inspert.ru  Thu Nov 24 11:31:33 2022
From: vl at inspert.ru (Vladimir Homutov)
Date: Thu, 24 Nov 2022 14:31:33 +0300
Subject: [patch] ngx_cpp_test module build issue cleanup
In-Reply-To: <6FBB2B8F-A082-4EC6-8C00-98F9A5DBA663@nginx.com>
References: <Y35dfmxg4WVcyuUo@vlws>
 <6FBB2B8F-A082-4EC6-8C00-98F9A5DBA663@nginx.com>
Message-ID: <Y39WFdPv94BDopht@vlws>

On Thu, Nov 24, 2022 at 01:25:30PM +0400, Sergey Kandaurov wrote:
>
> > On 23 Nov 2022, at 21:50, Vladimir Homutov via nginx-devel <nginx-devel at nginx.org> wrote:
> >
> > Hello,
> >
> > the simplest ./configure --with-cpp_test_module leads to build error
> > after successful configuration:
> >
> > src/misc/ngx_cpp_test_module.cpp:13:12: fatal error: ngx_mail.h: No such file or directory
> >   13 |   #include <ngx_mail.h>
> >      |            ^~~~~~~~~~~~
> > compilation terminated.
> >
> >
> > # HG changeset patch
> > # User Vladimir Khomutov <vl at inspert.ru>
> > # Date 1669225034 -10800
> > #      Wed Nov 23 20:37:14 2022 +0300
> > # Node ID 6237563c81707c8c2453cb0a7509ddaf64c02f4e
> > # Parent  49e7db44b57c9f4d54b87d19a696178b913aec5c
> > The ngx_cpp_test_module build requires mail and stream.
> >
> > # HG changeset patch
> > # User Vladimir Khomutov <vl at inspert.ru>
> > # Date 1669225742 -10800
> > #      Wed Nov 23 20:49:02 2022 +0300
> > # Node ID 12c04127e3fe4d6aa689ef3bcf3ae0834e7e9ed5
> > # Parent  b809f53d3f5bd04df36ac338845289d8e60a888b
> > The ngx_cpp_test_module build requires mail and stream.
> >
> > diff --git a/auto/modules b/auto/modules
> > --- a/auto/modules
> > +++ b/auto/modules
> > @@ -1358,6 +1358,17 @@ if [ $NGX_GOOGLE_PERFTOOLS = YES ]; then
> > fi
> >
> > if [ $NGX_CPP_TEST = YES ]; then
> > +
> > +    if [ $MAIL = NO ]; then
> > +        echo "$0: error: ngx_cpp_test_module assumes \"--with-mail\""
> > +        exit 1
> > +    fi
> > +
> > +    if [ $STREAM = NO ]; then
> > +        echo "$0: error: ngx_cpp_test_module assumes \"--with-stream\""
> > +        exit 1
> > +    fi
> > +
> >     ngx_module_name=
> >     ngx_module_incs=
> >     ngx_module_deps=
> >
>
> Hello,
>
> if at all try to fix it,
> --without-http would also need to be addressed.

yes, you are right. missed that since it is enabled by default.

A bit shorter patch:

# HG changeset patch
# User Vladimir Khomutov <vl at inspert.ru>
# Date 1669289342 -10800
#      Thu Nov 24 14:29:02 2022 +0300
# Node ID fd671044ba73ab8a32e558ba9d4dbe718f2b7a54
# Parent  b809f53d3f5bd04df36ac338845289d8e60a888b
The ngx_cpp_test_module build requires http, mail and stream.

diff --git a/auto/modules b/auto/modules
--- a/auto/modules
+++ b/auto/modules
@@ -1358,6 +1358,12 @@ if [ $NGX_GOOGLE_PERFTOOLS = YES ]; then
 fi

 if [ $NGX_CPP_TEST = YES ]; then
+
+    if [ $HTTP = NO -o $MAIL = NO -o $STREAM = NO ]; then
+        echo "$0: error: ngx_cpp_test_module requires http, mail and stream"
+        exit 1
+    fi
+
     ngx_module_name=
     ngx_module_incs=
     ngx_module_deps=


From arut at nginx.com  Thu Nov 24 15:15:31 2022
From: arut at nginx.com (Roman Arutyunyan)
Date: Thu, 24 Nov 2022 19:15:31 +0400
Subject: [PATCH 08 of 10] QUIC: idle mode for main connection
In-Reply-To: <20221020142503.d327lsrunx6hzw63@N00W24XTQX>
References: <patchbomb.1662627987@arut-laptop>
 <e0634a484d9a2d82d43f.1662627995@arut-laptop>
 <20221020115015.azqjb5wd7fisgm2f@Y9MQ9X2QVV>
 <20221020142503.d327lsrunx6hzw63@N00W24XTQX>
Message-ID: <20221124151531.s4oldl3jeo4wtobv@N00W24XTQX>

Hi,

On Thu, Oct 20, 2022 at 06:25:03PM +0400, Roman Arutyunyan wrote:
> Hi,
> 
> On Thu, Oct 20, 2022 at 03:50:15PM +0400, Sergey Kandaurov wrote:
> > On Thu, Sep 08, 2022 at 01:06:35PM +0400, Roman Arutyunyan wrote:
> > > # HG changeset patch
> > > # User Roman Arutyunyan <arut at nginx.com>
> > > # Date 1662627133 -14400
> > > #      Thu Sep 08 12:52:13 2022 +0400
> > > # Branch quic
> > > # Node ID e0634a484d9a2d82d43f565d64a0a22e989ac1cb
> > > # Parent  1dd6fabfdcb5b52af495f9d8fc00f64ae36a537c
> > > QUIC: idle mode for main connection.
> > > 
> > > Now main QUIC connection for HTTP/3 always has c->idle flag set.  This allows
> > > the connection to receive worker shutdown notification.  It is passed to
> > > application level via a new conf->shutdown() callback.
> > > 
> > > The HTTP/3 shutdown callback sends GOAWAY to client and gracefully shuts down
> > > the QUIC connection.
> > > 
> > > diff --git a/src/event/quic/ngx_event_quic.c b/src/event/quic/ngx_event_quic.c
> > > --- a/src/event/quic/ngx_event_quic.c
> > > +++ b/src/event/quic/ngx_event_quic.c
> > > @@ -341,6 +341,7 @@ ngx_quic_new_connection(ngx_connection_t
> > >          return NULL;
> > >      }
> > >  
> > > +    c->idle = 1;
> > >      ngx_reusable_connection(c, 1);
> > >  
> > >      ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0,
> > > @@ -420,9 +421,9 @@ ngx_quic_input_handler(ngx_event_t *rev)
> > >      }
> > >  
> > >      if (c->close) {
> > > -        qc->error = NGX_QUIC_ERR_NO_ERROR;
> > > -        qc->error_reason = "graceful shutdown";
> > > -        ngx_quic_close_connection(c, NGX_ERROR);
> > > +        if (qc->conf->shutdown) {
> > 
> > As previously discussed in private, this will need an additional check
> > that we are not yet in qc->closing.
> > 
> > > +            qc->conf->shutdown(c);
> > > +        }
> > >          return;
> > >      }
> 
> 
> Yes, added the check.  Also, c->close is reset here similarly to HTTP/2
> since we want to be able to handle future packets normally.
> 
> Also, current code which closes the connection instantly should remain for
> connection reuse.  To tell reuse from shutdown we can check ngx_exiting.
> Assuming reuse does not make sense in shutdown mode, this will work good.
> 
> > > diff --git a/src/event/quic/ngx_event_quic.h b/src/event/quic/ngx_event_quic.h
> > > --- a/src/event/quic/ngx_event_quic.h
> > > +++ b/src/event/quic/ngx_event_quic.h
> > > @@ -28,6 +28,9 @@
> > >  #define NGX_QUIC_STREAM_UNIDIRECTIONAL       0x02
> > >  
> > >  
> > > +typedef void (*ngx_quic_shutdown_pt)(ngx_connection_t *c);
> > > +
> > > +
> > >  typedef enum {
> > >      NGX_QUIC_STREAM_SEND_READY = 0,
> > >      NGX_QUIC_STREAM_SEND_SEND,
> > > @@ -74,6 +77,8 @@ typedef struct {
> > >      ngx_int_t                      stream_reject_code_uni;
> > >      ngx_int_t                      stream_reject_code_bidi;
> > >  
> > > +    ngx_quic_shutdown_pt           shutdown;
> > > +
> > >      u_char                         av_token_key[NGX_QUIC_AV_KEY_LEN];
> > >      u_char                         sr_token_key[NGX_QUIC_SR_KEY_LEN];
> > >  } ngx_quic_conf_t;
> > > diff --git a/src/http/v3/ngx_http_v3.h b/src/http/v3/ngx_http_v3.h
> > > --- a/src/http/v3/ngx_http_v3.h
> > > +++ b/src/http/v3/ngx_http_v3.h
> > > @@ -141,6 +141,7 @@ struct ngx_http_v3_session_s {
> > >      uint64_t                      next_push_id;
> > >      uint64_t                      max_push_id;
> > >      uint64_t                      goaway_push_id;
> > > +    uint64_t                      next_request_id;
> > >  
> > >      off_t                         total_bytes;
> > >      off_t                         payload_bytes;
> > > @@ -158,6 +159,7 @@ void ngx_http_v3_init(ngx_connection_t *
> > >  void ngx_http_v3_reset_connection(ngx_connection_t *c);
> > >  ngx_int_t ngx_http_v3_init_session(ngx_connection_t *c);
> > >  ngx_int_t ngx_http_v3_check_flood(ngx_connection_t *c);
> > > +void ngx_http_v3_shutdown(ngx_connection_t *c);
> > >  
> > >  ngx_int_t ngx_http_v3_read_request_body(ngx_http_request_t *r);
> > >  ngx_int_t ngx_http_v3_read_unbuffered_request_body(ngx_http_request_t *r);
> > > diff --git a/src/http/v3/ngx_http_v3_module.c b/src/http/v3/ngx_http_v3_module.c
> > > --- a/src/http/v3/ngx_http_v3_module.c
> > > +++ b/src/http/v3/ngx_http_v3_module.c
> > > @@ -249,6 +249,8 @@ ngx_http_v3_create_srv_conf(ngx_conf_t *
> > >      h3scf->quic.stream_reject_code_bidi = NGX_HTTP_V3_ERR_REQUEST_REJECTED;
> > >      h3scf->quic.active_connection_id_limit = NGX_CONF_UNSET_UINT;
> > >  
> > > +    h3scf->quic.shutdown = ngx_http_v3_shutdown;
> > > +
> > >      return h3scf;
> > >  }
> > >  
> > > diff --git a/src/http/v3/ngx_http_v3_request.c b/src/http/v3/ngx_http_v3_request.c
> > > --- a/src/http/v3/ngx_http_v3_request.c
> > > +++ b/src/http/v3/ngx_http_v3_request.c
> > > @@ -97,6 +97,37 @@ ngx_http_v3_init(ngx_connection_t *c)
> > >  }
> > >  
> > >  
> > > +void
> > > +ngx_http_v3_shutdown(ngx_connection_t *c)
> > > +{
> > > +    ngx_http_v3_session_t   *h3c;
> > 
> > extra indent
> > 
> > > +
> > > +    ngx_log_debug0(NGX_LOG_DEBUG_HTTP, c->log, 0, "http3 shutdown");
> > > +
> > > +    h3c = ngx_http_v3_get_session(c);
> > > +
> > > +    if (h3c == NULL) {
> > > +        ngx_quic_finalize_connection(c, NGX_HTTP_V3_ERR_NO_ERROR,
> > > +                                     "connection shutdown");
> > > +        return;
> > > +    }
> > > +
> > > +    if (!h3c->goaway) {
> > > +        h3c->goaway = 1;
> > > +
> > > +#if (NGX_HTTP_V3_HQ)
> > > +        if (!h3c->hq)
> > > +#endif
> > > +        {
> > > +            (void) ngx_http_v3_send_goaway(c, h3c->next_request_id);
> > > +        }
> > > +
> > > +        ngx_http_v3_shutdown_connection(c, NGX_HTTP_V3_ERR_NO_ERROR,
> > > +                                        "connection shutdown");
> > > +    }
> > 
> > Note that this callback is used to be called from a read event as part of
> > graceful shutdown.
> > With ngx_quic_finalize_connection() remade in patch #4 (reusable mode)
> > to defer closing QUIC connection to a posted event, this call now results
> > in a posted event, which no one can fulfill, hence no further action until
> > quic idle timeout fires.
> > It could be fixed by executing known posted events after shutdown callback
> > or more globally - as part of graceful shutdown itself.
> 
> Yes, events posted from ngx_close_idle_connections() are not handled right away.
> Instead, they are handled at the end of the next cycle, which normally
> happens after a timeout.  There seems to be no pretty way to fix this, unless
> we handle posted events in ngx_worker_process_cycle() right after
> ngx_close_idle_connections().  We are trying to avoid global changes like this.
> 
> I suggest posting current connection read event as a next posted event.  This
> will effectively set next cycle timeout to be zero and eliminate the problem.
> 
> > > +}
> > > +
> > > +
> > >  static void
> > >  ngx_http_v3_init_request_stream(ngx_connection_t *c)
> > >  {
> > > @@ -137,6 +168,8 @@ ngx_http_v3_init_request_stream(ngx_conn
> > >  
> > >      pc = c->quic->parent;
> > >  
> > > +    h3c->next_request_id = c->quic->id + 0x04;
> > > +
> > >      if (n + 1 == clcf->keepalive_requests
> > >          || ngx_current_msec - pc->start_time > clcf->keepalive_time)
> > >      {
> > > @@ -146,7 +179,7 @@ ngx_http_v3_init_request_stream(ngx_conn
> > >          if (!h3c->hq)
> > >  #endif
> > >          {
> > > -            if (ngx_http_v3_send_goaway(c, (n + 1) << 2) != NGX_OK) {
> > > +            if (ngx_http_v3_send_goaway(c, h3c->next_request_id) != NGX_OK) {
> > >                  ngx_http_close_connection(c);
> > >                  return;
> > >              }
> > > 
> > > _______________________________________________
> > > nginx-devel mailing list -- nginx-devel at nginx.org
> > > To unsubscribe send an email to nginx-devel-leave at nginx.org
> > _______________________________________________
> > nginx-devel mailing list -- nginx-devel at nginx.org
> > To unsubscribe send an email to nginx-devel-leave at nginx.org
> 
> Attached is a diff to the current patch.
> 
> --
> Roman

> # HG changeset patch
> # User Roman Arutyunyan <arut at nginx.com>
> # Date 1666273166 -14400
> #      Thu Oct 20 17:39:26 2022 +0400
> # Branch quic
> # Node ID d6c725081a0b024886822e1cc722fdace9c32621
> # Parent  a4ba2ac5fa55ef94bb75a66e66e0b19d792fed10
> [mq]: quic-idle-fix1
> 
> diff --git a/src/event/quic/ngx_event_quic.c b/src/event/quic/ngx_event_quic.c
> --- a/src/event/quic/ngx_event_quic.c
> +++ b/src/event/quic/ngx_event_quic.c
> @@ -421,9 +421,22 @@ ngx_quic_input_handler(ngx_event_t *rev)
>      }
>  
>      if (c->close) {
> -        if (qc->conf->shutdown) {
> +        c->close = 0;
> +
> +        if (!ngx_exiting) {
> +            qc->error = NGX_QUIC_ERR_NO_ERROR;
> +            qc->error_reason = "graceful shutdown";
> +            ngx_quic_close_connection(c, NGX_ERROR);
> +            return;
> +        }
> +
> +        if (!qc->closing && qc->conf->shutdown) {
> +            /* do not delay events posted by shutdown() */
> +            ngx_post_event(rev, &ngx_posted_next_events);

Following the change http://hg.nginx.org/nginx/rev/b809f53d3f5b, this part
is no longer needed.  The new diff attached.

> +
>              qc->conf->shutdown(c);
>          }
> +
>          return;
>      }


--
Roman Arutyunyan
-------------- next part --------------
# HG changeset patch
# User Roman Arutyunyan <arut at nginx.com>
# Date 1669302768 -14400
#      Thu Nov 24 19:12:48 2022 +0400
# Branch quic
# Node ID ec21165abb4333f77fb85e956cccf300c8ae1acf
# Parent  6fba6061b65f0ee41c231bde6b738cfae00bb179
[mq]: quic-idle-fix1

diff --git a/src/event/quic/ngx_event_quic.c b/src/event/quic/ngx_event_quic.c
--- a/src/event/quic/ngx_event_quic.c
+++ b/src/event/quic/ngx_event_quic.c
@@ -421,9 +421,19 @@ ngx_quic_input_handler(ngx_event_t *rev)
     }
 
     if (c->close) {
-        if (qc->conf->shutdown) {
+        c->close = 0;
+
+        if (!ngx_exiting) {
+            qc->error = NGX_QUIC_ERR_NO_ERROR;
+            qc->error_reason = "graceful shutdown";
+            ngx_quic_close_connection(c, NGX_ERROR);
+            return;
+        }
+
+        if (!qc->closing && qc->conf->shutdown) {
             qc->conf->shutdown(c);
         }
+
         return;
     }
 
diff --git a/src/http/v3/ngx_http_v3_request.c b/src/http/v3/ngx_http_v3_request.c
--- a/src/http/v3/ngx_http_v3_request.c
+++ b/src/http/v3/ngx_http_v3_request.c
@@ -101,7 +101,7 @@ ngx_http_v3_init(ngx_connection_t *c)
 void
 ngx_http_v3_shutdown(ngx_connection_t *c)
 {
-    ngx_http_v3_session_t   *h3c;
+    ngx_http_v3_session_t  *h3c;
 
     ngx_log_debug0(NGX_LOG_DEBUG_HTTP, c->log, 0, "http3 shutdown");
 

From arut at nginx.com  Thu Nov 24 15:33:51 2022
From: arut at nginx.com (Roman Arutyunyan)
Date: Thu, 24 Nov 2022 19:33:51 +0400
Subject: QUIC: fixed computation of nonce
In-Reply-To: <D561E0F8-1395-4864-AEF4-C5FEA8E41307@nginx.com>
References: <6e54119a.44a7.1848ba488b5.Coremail.yolkking@126.com>
 <D561E0F8-1395-4864-AEF4-C5FEA8E41307@nginx.com>
Message-ID: <20221124153351.ablijt3rinybgrl5@N00W24XTQX>

Hi,

On Tue, Nov 22, 2022 at 02:29:47PM +0400, Sergey Kandaurov wrote:
> 
> > On 18 Nov 2022, at 20:48, Yu Zhu <yolkking at 126.com> wrote:
> > 
> > # HG changeset patch
> > # User Yu Zhu <lishu.zy at alibaba-inc.com>
> > # Date 1668789115 -28800
> > #      Sat Nov 19 00:31:55 2022 +0800
> > # Branch quic
> > # Node ID 1a320805265db14904ca9deaae8330f4979619ce
> > # Parent  6cf8ed15fd00668b7efa0226c06f47d7238f26e8
> > QUIC: fixed computation of nonce
> > 
> > RFC 9001, 5.3. AEAD Usage
> > The nonce, N, is formed by combining the packet protection IV with the packet number. The 62 bits of the reconstructed QUIC packet number in network byte order are left-padded with zeros to the size of the IV. The exclusive OR of the padded packet number and the IV forms the AEAD nonce.
> 
> While citing RFCs is certainly good, I would limit this
> to a specific description what this patch intends to do.
> 
> > 
> > diff -r 6cf8ed15fd00 -r 1a320805265d src/event/quic/ngx_event_quic_protection.c
> > --- a/src/event/quic/ngx_event_quic_protection.c        Tue Nov 01 17:00:35 2022 +0400
> > +++ b/src/event/quic/ngx_event_quic_protection.c        Sat Nov 19 00:31:55 2022 +0800
> > @@ -969,10 +969,11 @@
> >  static void
> >  ngx_quic_compute_nonce(u_char *nonce, size_t len, uint64_t pn)
> >  {
> > -    nonce[len - 4] ^= (pn & 0xff000000) >> 24;
> > -    nonce[len - 3] ^= (pn & 0x00ff0000) >> 16;
> > -    nonce[len - 2] ^= (pn & 0x0000ff00) >> 8;
> > -    nonce[len - 1] ^= (pn & 0x000000ff);
> > +    size_t  i;
> > +
> > +    for (i = 0; i < 8; i++) {
> > +        nonce[len - 8 + i] ^= (pn >> (8 - i - 1) * 8) & 0xff;
> > +    }
> >  }
> 
> Thanks for the patch.
> I tend to prefer computation with unrolled loops, though.
> Microbenchmarking shows that it performs better.
> Also, this allows to peel two high bits without additional branching.
> 
> 100m loop, user time in seconds
> 
>                 -O0     -O1     -O2
> gcc12/arm64
> old             0.609   0.233   0.024
> new             1.030   0.454   0.683
> clang14/arm64
> old             0.447   0.256   0.096*
> new             1.292   0.453   0.096*
> gcc8/i386
> old             1.519   0.541   0.551
> new             2.915   1.318   1.476
> 
> # HG changeset patch
> # User Sergey Kandaurov <pluknet at nginx.com>
> # Date 1669112795 -14400
> #      Tue Nov 22 14:26:35 2022 +0400
> # Branch quic
> # Node ID 29d8d12b6d35eb172465db2dff47bd8e98b36fc7
> # Parent  0f5fc7a320db621da8731a8832f486c7167b236b
> QUIC: fixed computation of nonce with packet numbers beyond 2^32.
> 
> Prodded by Yu Zhu.
> 
> diff --git a/src/event/quic/ngx_event_quic_protection.c b/src/event/quic/ngx_event_quic_protection.c
> --- a/src/event/quic/ngx_event_quic_protection.c
> +++ b/src/event/quic/ngx_event_quic_protection.c
> @@ -969,10 +969,14 @@ ngx_quic_parse_pn(u_char **pos, ngx_int_
>  static void
>  ngx_quic_compute_nonce(u_char *nonce, size_t len, uint64_t pn)
>  {
> -    nonce[len - 4] ^= (pn & 0xff000000) >> 24;
> -    nonce[len - 3] ^= (pn & 0x00ff0000) >> 16;
> -    nonce[len - 2] ^= (pn & 0x0000ff00) >> 8;
> -    nonce[len - 1] ^= (pn & 0x000000ff);
> +    nonce[len - 8] ^= (pn >> 56) & 0x3f;
> +    nonce[len - 7] ^= (pn >> 48) & 0xff;
> +    nonce[len - 6] ^= (pn >> 40) & 0xff;
> +    nonce[len - 5] ^= (pn >> 32) & 0xff;
> +    nonce[len - 4] ^= (pn >> 24) & 0xff;
> +    nonce[len - 3] ^= (pn >> 16) & 0xff;
> +    nonce[len - 2] ^= (pn >> 8) & 0xff;
> +    nonce[len - 1] ^= pn & 0xff;
>  }

Looks ok

--
Roman Arutyunyan


From mdounin at mdounin.ru  Thu Nov 24 15:46:15 2022
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Thu, 24 Nov 2022 18:46:15 +0300
Subject: [patch] ngx_cpp_test module build issue cleanup
In-Reply-To: <Y39WFdPv94BDopht@vlws>
References: <Y35dfmxg4WVcyuUo@vlws>
 <6FBB2B8F-A082-4EC6-8C00-98F9A5DBA663@nginx.com>
 <Y39WFdPv94BDopht@vlws>
Message-ID: <Y3+Rx6IoDpen3qLH@mdounin.ru>

Hello!

On Thu, Nov 24, 2022 at 02:31:33PM +0300, Vladimir Homutov via nginx-devel wrote:

> On Thu, Nov 24, 2022 at 01:25:30PM +0400, Sergey Kandaurov wrote:
> >
> > > On 23 Nov 2022, at 21:50, Vladimir Homutov via nginx-devel <nginx-devel at nginx.org> wrote:
> > >
> > > Hello,
> > >
> > > the simplest ./configure --with-cpp_test_module leads to build error
> > > after successful configuration:
> > >
> > > src/misc/ngx_cpp_test_module.cpp:13:12: fatal error: ngx_mail.h: No such file or directory
> > >   13 |   #include <ngx_mail.h>
> > >      |            ^~~~~~~~~~~~
> > > compilation terminated.
> > >
> > >
> > > # HG changeset patch
> > > # User Vladimir Khomutov <vl at inspert.ru>
> > > # Date 1669225034 -10800
> > > #      Wed Nov 23 20:37:14 2022 +0300
> > > # Node ID 6237563c81707c8c2453cb0a7509ddaf64c02f4e
> > > # Parent  49e7db44b57c9f4d54b87d19a696178b913aec5c
> > > The ngx_cpp_test_module build requires mail and stream.
> > >
> > > # HG changeset patch
> > > # User Vladimir Khomutov <vl at inspert.ru>
> > > # Date 1669225742 -10800
> > > #      Wed Nov 23 20:49:02 2022 +0300
> > > # Node ID 12c04127e3fe4d6aa689ef3bcf3ae0834e7e9ed5
> > > # Parent  b809f53d3f5bd04df36ac338845289d8e60a888b
> > > The ngx_cpp_test_module build requires mail and stream.
> > >
> > > diff --git a/auto/modules b/auto/modules
> > > --- a/auto/modules
> > > +++ b/auto/modules
> > > @@ -1358,6 +1358,17 @@ if [ $NGX_GOOGLE_PERFTOOLS = YES ]; then
> > > fi
> > >
> > > if [ $NGX_CPP_TEST = YES ]; then
> > > +
> > > +    if [ $MAIL = NO ]; then
> > > +        echo "$0: error: ngx_cpp_test_module assumes \"--with-mail\""
> > > +        exit 1
> > > +    fi
> > > +
> > > +    if [ $STREAM = NO ]; then
> > > +        echo "$0: error: ngx_cpp_test_module assumes \"--with-stream\""
> > > +        exit 1
> > > +    fi
> > > +
> > >     ngx_module_name=
> > >     ngx_module_incs=
> > >     ngx_module_deps=
> > >
> >
> > Hello,
> >
> > if at all try to fix it,
> > --without-http would also need to be addressed.
> 
> yes, you are right. missed that since it is enabled by default.
> 
> A bit shorter patch:
> 
> # HG changeset patch
> # User Vladimir Khomutov <vl at inspert.ru>
> # Date 1669289342 -10800
> #      Thu Nov 24 14:29:02 2022 +0300
> # Node ID fd671044ba73ab8a32e558ba9d4dbe718f2b7a54
> # Parent  b809f53d3f5bd04df36ac338845289d8e60a888b
> The ngx_cpp_test_module build requires http, mail and stream.
> 
> diff --git a/auto/modules b/auto/modules
> --- a/auto/modules
> +++ b/auto/modules
> @@ -1358,6 +1358,12 @@ if [ $NGX_GOOGLE_PERFTOOLS = YES ]; then
>  fi
> 
>  if [ $NGX_CPP_TEST = YES ]; then
> +
> +    if [ $HTTP = NO -o $MAIL = NO -o $STREAM = NO ]; then
> +        echo "$0: error: ngx_cpp_test_module requires http, mail and stream"
> +        exit 1
> +    fi
> +
>      ngx_module_name=
>      ngx_module_incs=
>      ngx_module_deps=

Following other configure error messages (see auto/lib/zlib/conf, 
auto/lib/pcre/conf, or auto/lib/google-perftools/conf), this 
should be "the C++ test module ...", if at all.

Also, it probably should be in auto/options, since it is a 
verification of configure options, while the auto/modules is 
expected to construct various internal lists based on the modules 
being enabled.

Overall, I'm not convinced it is actually needed, since the module 
is basically a development tool, and not expected to be used by 
users who are not aware of how it is expected to be used, why the 
compilation could fail if mail or stream aren't compiled in, and 
how to fix it even without reconfiguring nginx with mail and 
stream.

-- 
Maxim Dounin
http://mdounin.ru/


From vl at inspert.ru  Thu Nov 24 16:51:36 2022
From: vl at inspert.ru (Vladimir Homutov)
Date: Thu, 24 Nov 2022 19:51:36 +0300
Subject: [patch] ngx_cpp_test module build issue cleanup
In-Reply-To: <Y3+Rx6IoDpen3qLH@mdounin.ru>
References: <Y35dfmxg4WVcyuUo@vlws>
 <6FBB2B8F-A082-4EC6-8C00-98F9A5DBA663@nginx.com>
 <Y39WFdPv94BDopht@vlws>
 <Y3+Rx6IoDpen3qLH@mdounin.ru>
Message-ID: <Y3+hGJC54NjaFZGX@vlws>

On Thu, Nov 24, 2022 at 06:46:15PM +0300, Maxim Dounin wrote:
> Hello!
>
> On Thu, Nov 24, 2022 at 02:31:33PM +0300, Vladimir Homutov via nginx-devel wrote:
>
> > On Thu, Nov 24, 2022 at 01:25:30PM +0400, Sergey Kandaurov wrote:
> > >
> > > > On 23 Nov 2022, at 21:50, Vladimir Homutov via nginx-devel <nginx-devel at nginx.org> wrote:
> > > >
> > > > Hello,
> > > >
> > > > the simplest ./configure --with-cpp_test_module leads to build error
> > > > after successful configuration:
> > > >
> > > > src/misc/ngx_cpp_test_module.cpp:13:12: fatal error: ngx_mail.h: No such file or directory
> > > >   13 |   #include <ngx_mail.h>
> > > >      |            ^~~~~~~~~~~~
> > > > compilation terminated.
> > > >
> > > >
> > > > # HG changeset patch
> > > > # User Vladimir Khomutov <vl at inspert.ru>
> > > > # Date 1669225034 -10800
> > > > #      Wed Nov 23 20:37:14 2022 +0300
> > > > # Node ID 6237563c81707c8c2453cb0a7509ddaf64c02f4e
> > > > # Parent  49e7db44b57c9f4d54b87d19a696178b913aec5c
> > > > The ngx_cpp_test_module build requires mail and stream.
> > > >
> > > > # HG changeset patch
> > > > # User Vladimir Khomutov <vl at inspert.ru>
> > > > # Date 1669225742 -10800
> > > > #      Wed Nov 23 20:49:02 2022 +0300
> > > > # Node ID 12c04127e3fe4d6aa689ef3bcf3ae0834e7e9ed5
> > > > # Parent  b809f53d3f5bd04df36ac338845289d8e60a888b
> > > > The ngx_cpp_test_module build requires mail and stream.
> > > >
> > > > diff --git a/auto/modules b/auto/modules
> > > > --- a/auto/modules
> > > > +++ b/auto/modules
> > > > @@ -1358,6 +1358,17 @@ if [ $NGX_GOOGLE_PERFTOOLS = YES ]; then
> > > > fi
> > > >
> > > > if [ $NGX_CPP_TEST = YES ]; then
> > > > +
> > > > +    if [ $MAIL = NO ]; then
> > > > +        echo "$0: error: ngx_cpp_test_module assumes \"--with-mail\""
> > > > +        exit 1
> > > > +    fi
> > > > +
> > > > +    if [ $STREAM = NO ]; then
> > > > +        echo "$0: error: ngx_cpp_test_module assumes \"--with-stream\""
> > > > +        exit 1
> > > > +    fi
> > > > +
> > > >     ngx_module_name=
> > > >     ngx_module_incs=
> > > >     ngx_module_deps=
> > > >
> > >
> > > Hello,
> > >
> > > if at all try to fix it,
> > > --without-http would also need to be addressed.
> >
> > yes, you are right. missed that since it is enabled by default.
> >
> > A bit shorter patch:
> >
> > # HG changeset patch
> > # User Vladimir Khomutov <vl at inspert.ru>
> > # Date 1669289342 -10800
> > #      Thu Nov 24 14:29:02 2022 +0300
> > # Node ID fd671044ba73ab8a32e558ba9d4dbe718f2b7a54
> > # Parent  b809f53d3f5bd04df36ac338845289d8e60a888b
> > The ngx_cpp_test_module build requires http, mail and stream.
> >
> > diff --git a/auto/modules b/auto/modules
> > --- a/auto/modules
> > +++ b/auto/modules
> > @@ -1358,6 +1358,12 @@ if [ $NGX_GOOGLE_PERFTOOLS = YES ]; then
> >  fi
> >
> >  if [ $NGX_CPP_TEST = YES ]; then
> > +
> > +    if [ $HTTP = NO -o $MAIL = NO -o $STREAM = NO ]; then
> > +        echo "$0: error: ngx_cpp_test_module requires http, mail and stream"
> > +        exit 1
> > +    fi
> > +
> >      ngx_module_name=
> >      ngx_module_incs=
> >      ngx_module_deps=
>
> Following other configure error messages (see auto/lib/zlib/conf,
> auto/lib/pcre/conf, or auto/lib/google-perftools/conf), this
> should be "the C++ test module ...", if at all.
>
> Also, it probably should be in auto/options, since it is a
> verification of configure options, while the auto/modules is
> expected to construct various internal lists based on the modules
> being enabled.
>
> Overall, I'm not convinced it is actually needed, since the module
> is basically a development tool, and not expected to be used by
> users who are not aware of how it is expected to be used, why the
> compilation could fail if mail or stream aren't compiled in, and
> how to fix it even without reconfiguring nginx with mail and
> stream.
>

I tend to agree that it's not worth effort.
Probably we need to have macro defined depending on presence of
HTTP/MAIL/STREAM subsystems, and use them in source code.
But I can hardly imaging code that needs it.


From mdounin at mdounin.ru  Thu Nov 24 17:54:59 2022
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Thu, 24 Nov 2022 20:54:59 +0300
Subject: [patch] ngx_cpp_test module build issue cleanup
In-Reply-To: <Y3+hGJC54NjaFZGX@vlws>
References: <Y35dfmxg4WVcyuUo@vlws>
 <6FBB2B8F-A082-4EC6-8C00-98F9A5DBA663@nginx.com>
 <Y39WFdPv94BDopht@vlws>
 <Y3+Rx6IoDpen3qLH@mdounin.ru>
 <Y3+hGJC54NjaFZGX@vlws>
Message-ID: <Y3+v8z5mj9/0MdXu@mdounin.ru>

Hello!

On Thu, Nov 24, 2022 at 07:51:36PM +0300, Vladimir Homutov via nginx-devel wrote:

> On Thu, Nov 24, 2022 at 06:46:15PM +0300, Maxim Dounin wrote:
> > Hello!
> >
> > On Thu, Nov 24, 2022 at 02:31:33PM +0300, Vladimir Homutov via nginx-devel wrote:
> >
> > > On Thu, Nov 24, 2022 at 01:25:30PM +0400, Sergey Kandaurov wrote:
> > > >
> > > > > On 23 Nov 2022, at 21:50, Vladimir Homutov via nginx-devel <nginx-devel at nginx.org> wrote:
> > > > >
> > > > > Hello,
> > > > >
> > > > > the simplest ./configure --with-cpp_test_module leads to build error
> > > > > after successful configuration:
> > > > >
> > > > > src/misc/ngx_cpp_test_module.cpp:13:12: fatal error: ngx_mail.h: No such file or directory
> > > > >   13 |   #include <ngx_mail.h>
> > > > >      |            ^~~~~~~~~~~~
> > > > > compilation terminated.
> > > > >
> > > > >
> > > > > # HG changeset patch
> > > > > # User Vladimir Khomutov <vl at inspert.ru>
> > > > > # Date 1669225034 -10800
> > > > > #      Wed Nov 23 20:37:14 2022 +0300
> > > > > # Node ID 6237563c81707c8c2453cb0a7509ddaf64c02f4e
> > > > > # Parent  49e7db44b57c9f4d54b87d19a696178b913aec5c
> > > > > The ngx_cpp_test_module build requires mail and stream.
> > > > >
> > > > > # HG changeset patch
> > > > > # User Vladimir Khomutov <vl at inspert.ru>
> > > > > # Date 1669225742 -10800
> > > > > #      Wed Nov 23 20:49:02 2022 +0300
> > > > > # Node ID 12c04127e3fe4d6aa689ef3bcf3ae0834e7e9ed5
> > > > > # Parent  b809f53d3f5bd04df36ac338845289d8e60a888b
> > > > > The ngx_cpp_test_module build requires mail and stream.
> > > > >
> > > > > diff --git a/auto/modules b/auto/modules
> > > > > --- a/auto/modules
> > > > > +++ b/auto/modules
> > > > > @@ -1358,6 +1358,17 @@ if [ $NGX_GOOGLE_PERFTOOLS = YES ]; then
> > > > > fi
> > > > >
> > > > > if [ $NGX_CPP_TEST = YES ]; then
> > > > > +
> > > > > +    if [ $MAIL = NO ]; then
> > > > > +        echo "$0: error: ngx_cpp_test_module assumes \"--with-mail\""
> > > > > +        exit 1
> > > > > +    fi
> > > > > +
> > > > > +    if [ $STREAM = NO ]; then
> > > > > +        echo "$0: error: ngx_cpp_test_module assumes \"--with-stream\""
> > > > > +        exit 1
> > > > > +    fi
> > > > > +
> > > > >     ngx_module_name=
> > > > >     ngx_module_incs=
> > > > >     ngx_module_deps=
> > > > >
> > > >
> > > > Hello,
> > > >
> > > > if at all try to fix it,
> > > > --without-http would also need to be addressed.
> > >
> > > yes, you are right. missed that since it is enabled by default.
> > >
> > > A bit shorter patch:
> > >
> > > # HG changeset patch
> > > # User Vladimir Khomutov <vl at inspert.ru>
> > > # Date 1669289342 -10800
> > > #      Thu Nov 24 14:29:02 2022 +0300
> > > # Node ID fd671044ba73ab8a32e558ba9d4dbe718f2b7a54
> > > # Parent  b809f53d3f5bd04df36ac338845289d8e60a888b
> > > The ngx_cpp_test_module build requires http, mail and stream.
> > >
> > > diff --git a/auto/modules b/auto/modules
> > > --- a/auto/modules
> > > +++ b/auto/modules
> > > @@ -1358,6 +1358,12 @@ if [ $NGX_GOOGLE_PERFTOOLS = YES ]; then
> > >  fi
> > >
> > >  if [ $NGX_CPP_TEST = YES ]; then
> > > +
> > > +    if [ $HTTP = NO -o $MAIL = NO -o $STREAM = NO ]; then
> > > +        echo "$0: error: ngx_cpp_test_module requires http, mail and stream"
> > > +        exit 1
> > > +    fi
> > > +
> > >      ngx_module_name=
> > >      ngx_module_incs=
> > >      ngx_module_deps=
> >
> > Following other configure error messages (see auto/lib/zlib/conf,
> > auto/lib/pcre/conf, or auto/lib/google-perftools/conf), this
> > should be "the C++ test module ...", if at all.
> >
> > Also, it probably should be in auto/options, since it is a
> > verification of configure options, while the auto/modules is
> > expected to construct various internal lists based on the modules
> > being enabled.
> >
> > Overall, I'm not convinced it is actually needed, since the module
> > is basically a development tool, and not expected to be used by
> > users who are not aware of how it is expected to be used, why the
> > compilation could fail if mail or stream aren't compiled in, and
> > how to fix it even without reconfiguring nginx with mail and
> > stream.
> >
> 
> I tend to agree that it's not worth effort.
> Probably we need to have macro defined depending on presence of
> HTTP/MAIL/STREAM subsystems, and use them in source code.
> But I can hardly imaging code that needs it.

This was previously discussed, and does not seem to be correct 
solution at least for mail and stream, since these can be 
dynamically loaded.

-- 
Maxim Dounin
http://mdounin.ru/


From mdounin at mdounin.ru  Thu Nov 24 18:25:11 2022
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Thu, 24 Nov 2022 21:25:11 +0300
Subject: [PATCH] Filtering duplicate addresses in listen (ticket #2400)
In-Reply-To: <1E74A87E-A48E-432C-AB33-FA6BFC51157B@nginx.com>
References: <55bcf8dc4ee35ccf40f5.1667097660@vm-bsd.mdounin.ru>
 <20221123125610.yh4iupkprsc2twpt@Y9MQ9X2QVV>
 <Y34yP1+SB6VAmiEg@mdounin.ru>
 <1E74A87E-A48E-432C-AB33-FA6BFC51157B@nginx.com>
Message-ID: <Y3+3B/NVV3ad1bXc@mdounin.ru>

Hello!

On Wed, Nov 23, 2022 at 07:03:32PM +0400, Sergey Kandaurov wrote:

> 
> > On 23 Nov 2022, at 18:46, Maxim Dounin <mdounin at mdounin.ru> wrote:
> > 
> > Hello!
> > 
> > On Wed, Nov 23, 2022 at 04:56:10PM +0400, Sergey Kandaurov wrote:
> > 
> >> On Sun, Oct 30, 2022 at 05:41:00AM +0300, Maxim Dounin wrote:
> >>> # HG changeset patch
> >>> # User Maxim Dounin <mdounin at mdounin.ru>
> >>> # Date 1667097653 -10800
> >>> #      Sun Oct 30 05:40:53 2022 +0300
> >>> # Node ID 55bcf8dc4ee35ccf40f5b8a7cffde63e7edb9494
> >>> # Parent  1ae25660c0c76edef14121ca64362f28b9d57a70
> >>> Filtering duplicate addresses in listen (ticket #2400).
> >>> 
> >>> Due to the glibc bug[1], getaddrinfo("localhost") with AI_ADDRCONFIG
> >>> on a typical host with glibc and without IPv6 returns two 127.0.0.1
> >>> addresses, and therefore "listen localhost:80;" used to result the
> >> 
> >> result in ?
> > 
> > Fixed, thnx.
> > 
> >>> "duplicate ... address and port pair" after 4f9b72a229c1.
> >>> 
> >>> Fix is to explicitly filter out duplicate addresses returned during
> >>> resolution of a name.
> >>> 
> >>> [1] https://sourceware.org/bugzilla/show_bug.cgi?id=14969
> >>> 
> >>> diff --git a/src/http/ngx_http_core_module.c b/src/http/ngx_http_core_module.c
> >>> --- a/src/http/ngx_http_core_module.c
> >>> +++ b/src/http/ngx_http_core_module.c
> >>> @@ -3963,7 +3963,7 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx
> >>> 
> >>>     ngx_str_t              *value, size;
> >>>     ngx_url_t               u;
> >>> -    ngx_uint_t              n;
> >>> +    ngx_uint_t              n, i;
> >>>     ngx_http_listen_opt_t   lsopt;
> >>> 
> >>>     cscf->listen = 1;
> >>> @@ -4289,6 +4289,16 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx
> >>>     }
> >>> 
> >>>     for (n = 0; n < u.naddrs; n++) {
> >>> +
> >>> +        for (i = 0; i < n; i++) {
> >>> +            if (ngx_cmp_sockaddr(u.addrs[n].sockaddr, u.addrs[n].socklen,
> >>> +                                 u.addrs[i].sockaddr, u.addrs[i].socklen, 0)
> >>> +                == NGX_OK)
> >>> +            {
> >>> +                goto next;
> >>> +            }
> >>> +        }
> >>> +
> >>>         lsopt.sockaddr = u.addrs[n].sockaddr;
> >>>         lsopt.socklen = u.addrs[n].socklen;
> >>>         lsopt.addr_text = u.addrs[n].name;
> >>> @@ -4297,6 +4307,9 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx
> >>>         if (ngx_http_add_listen(cf, cscf, &lsopt) != NGX_OK) {
> >>>             return NGX_CONF_ERROR;
> >>>         }
> >>> +
> >>> +    next:
> >>> +        continue;
> >>>     }
> >>> 
> >>>     return NGX_CONF_OK;
> >>> diff --git a/src/mail/ngx_mail_core_module.c b/src/mail/ngx_mail_core_module.c
> >>> --- a/src/mail/ngx_mail_core_module.c
> >>> +++ b/src/mail/ngx_mail_core_module.c
> >>> @@ -308,7 +308,7 @@ ngx_mail_core_listen(ngx_conf_t *cf, ngx
> >>>     ngx_str_t                  *value, size;
> >>>     ngx_url_t                   u;
> >>>     ngx_uint_t                  i, n, m;
> >>> -    ngx_mail_listen_t          *ls, *als;
> >>> +    ngx_mail_listen_t          *ls, *als, *nls;
> >>>     ngx_mail_module_t          *module;
> >>>     ngx_mail_core_main_conf_t  *cmcf;
> >>> 
> >>> @@ -333,7 +333,7 @@ ngx_mail_core_listen(ngx_conf_t *cf, ngx
> >>> 
> >>>     cmcf = ngx_mail_conf_get_module_main_conf(cf, ngx_mail_core_module);
> >>> 
> >>> -    ls = ngx_array_push_n(&cmcf->listen, u.naddrs);
> >>> +    ls = ngx_array_push(&cmcf->listen);
> >>>     if (ls == NULL) {
> >>>         return NGX_CONF_ERROR;
> >>>     }
> >>> @@ -571,17 +571,37 @@ ngx_mail_core_listen(ngx_conf_t *cf, ngx
> >>>     als = cmcf->listen.elts;
> >> 
> >> With push of additional cmcf->listen elements moved inside loop,
> >> als can become a stale pointer due to reallocation in ngx_array_push().
> >> As such, als assignment should be kept updated (here and in stream).
> >> 
> >> Otherwise, looks good.
> > 
> > Sure, thanks for catching this.  Updated with the following 
> > additional change:
> > 
> > diff --git a/src/mail/ngx_mail_core_module.c b/src/mail/ngx_mail_core_module.c
> > --- a/src/mail/ngx_mail_core_module.c
> > +++ b/src/mail/ngx_mail_core_module.c
> > @@ -568,8 +568,6 @@ ngx_mail_core_listen(ngx_conf_t *cf, ngx
> >         return NGX_CONF_ERROR;
> >     }
> > 
> > -    als = cmcf->listen.elts;
> > -
> >     for (n = 0; n < u.naddrs; n++) {
> > 
> >         for (i = 0; i < n; i++) {
> > @@ -598,6 +596,8 @@ ngx_mail_core_listen(ngx_conf_t *cf, ngx
> >         nls->addr_text = u.addrs[n].name;
> >         nls->wildcard = ngx_inet_wildcard(nls->sockaddr);
> > 
> > +        als = cmcf->listen.elts;
> > +
> >         for (i = 0; i < cmcf->listen.nelts - 1; i++) {
> > 
> >             if (ngx_cmp_sockaddr(als[i].sockaddr, als[i].socklen,
> > diff --git a/src/stream/ngx_stream_core_module.c b/src/stream/ngx_stream_core_module.c
> > --- a/src/stream/ngx_stream_core_module.c
> > +++ b/src/stream/ngx_stream_core_module.c
> > @@ -886,8 +886,6 @@ ngx_stream_core_listen(ngx_conf_t *cf, n
> > #endif
> >     }
> > 
> > -    als = cmcf->listen.elts;
> > -
> >     for (n = 0; n < u.naddrs; n++) {
> > 
> >         for (i = 0; i < n; i++) {
> > @@ -916,6 +914,8 @@ ngx_stream_core_listen(ngx_conf_t *cf, n
> >         nls->addr_text = u.addrs[n].name;
> >         nls->wildcard = ngx_inet_wildcard(nls->sockaddr);
> > 
> > +        als = cmcf->listen.elts;
> > +
> >         for (i = 0; i < cmcf->listen.nelts - 1; i++) {
> >             if (nls->type != als[i].type) {
> >                 continue;
> > 
> > 
> > Full patch:
> > 
> > # HG changeset patch
> > # User Maxim Dounin <mdounin at mdounin.ru>
> > # Date 1669213808 -10800
> > #      Wed Nov 23 17:30:08 2022 +0300
> > # Node ID 4cc2bfeff46c7b7ee2b098f39ebfc1e44754df1e
> > # Parent  b809f53d3f5bd04df36ac338845289d8e60a888b
> > Filtering duplicate addresses in listen (ticket #2400).
> > 
> > Due to the glibc bug[1], getaddrinfo("localhost") with AI_ADDRCONFIG
> > on a typical host with glibc and without IPv6 returns two 127.0.0.1
> > addresses, and therefore "listen localhost:80;" used to result in
> > "duplicate ... address and port pair" after 4f9b72a229c1.
> > 
> > Fix is to explicitly filter out duplicate addresses returned during
> > resolution of a name.
> > 
> > [1] https://sourceware.org/bugzilla/show_bug.cgi?id=14969
> > 
> 
> [..]
> 
> Looks good.

Pushed to http://mdounin.ru/hg/nginx, thanks.

-- 
Maxim Dounin
http://mdounin.ru/


From mdounin at mdounin.ru  Thu Nov 24 18:25:37 2022
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Thu, 24 Nov 2022 21:25:37 +0300
Subject: [PATCH] Disabled cloning of sockets without master process
 (ticket #2403)
In-Reply-To: <089F17A0-DD71-411D-A1F9-82134D16DE3B@nginx.com>
References: <b73d95226c84b93e51f2.1667097698@vm-bsd.mdounin.ru>
 <089F17A0-DD71-411D-A1F9-82134D16DE3B@nginx.com>
Message-ID: <Y3+3IcVZdDtqWF1w@mdounin.ru>

Hello!

On Wed, Nov 23, 2022 at 05:56:01PM +0400, Sergey Kandaurov wrote:

> 
> > On 30 Oct 2022, at 06:41, Maxim Dounin <mdounin at mdounin.ru> wrote:
> > 
> > # HG changeset patch
> > # User Maxim Dounin <mdounin at mdounin.ru>
> > # Date 1667097682 -10800
> > #      Sun Oct 30 05:41:22 2022 +0300
> > # Node ID b73d95226c84b93e51f23f7b35782d98d3b516b9
> > # Parent  55bcf8dc4ee35ccf40f5b8a7cffde63e7edb9494
> > Disabled cloning of sockets without master process (ticket #2403).
> > 
> > Cloning of listening sockets for each worker process does not make sense
> > when working without master process, and causes some of the connections
> > not to be accepted if worker_processes is set to more than one and there
> > are listening sockets configured with the reuseport flag.  Fix is to
> > disable cloning when master process is disabled.
> > 
> > diff --git a/src/event/ngx_event.c b/src/event/ngx_event.c
> > --- a/src/event/ngx_event.c
> > +++ b/src/event/ngx_event.c
> > @@ -416,6 +416,7 @@ ngx_event_init_conf(ngx_cycle_t *cycle, 
> > {
> > #if (NGX_HAVE_REUSEPORT)
> >     ngx_uint_t        i;
> > +    ngx_core_conf_t  *ccf;
> >     ngx_listening_t  *ls;
> > #endif
> > 
> > @@ -442,7 +443,9 @@ ngx_event_init_conf(ngx_cycle_t *cycle, 
> > 
> > #if (NGX_HAVE_REUSEPORT)
> > 
> > -    if (!ngx_test_config) {
> > +    ccf = (ngx_core_conf_t *) ngx_get_conf(cycle->conf_ctx, ngx_core_module);
> > +
> > +    if (!ngx_test_config && ccf->master) {
> > 
> >         ls = cycle->listening.elts;
> >         for (i = 0; i < cycle->listening.nelts; i++) {
> 
> Looks good.

Pushed to http://mdounin.ru/hg/nginx, thanks.

-- 
Maxim Dounin
http://mdounin.ru/


From mdounin at mdounin.ru  Thu Nov 24 18:26:01 2022
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Thu, 24 Nov 2022 21:26:01 +0300
Subject: [PATCH] Fixed segfault when switching off master process during
 upgrade
In-Reply-To: <20221124090941.cpt6fxve2ce7qcye@N00W24XTQX>
References: <ef9c94be7fe4685f0eee.1667097753@vm-bsd.mdounin.ru>
 <20221123135625.lxe5gkuzm6gyzykj@N00W24XTQX>
 <Y36H8Dn/ZUOttPPk@mdounin.ru>
 <20221124090941.cpt6fxve2ce7qcye@N00W24XTQX>
Message-ID: <Y3+3OS1XWjIzDGmg@mdounin.ru>

Hello!

On Thu, Nov 24, 2022 at 01:09:41PM +0400, Roman Arutyunyan wrote:

> Hi,
> 
> On Wed, Nov 23, 2022 at 11:52:00PM +0300, Maxim Dounin wrote:
> > Hello!
> > 
> > On Wed, Nov 23, 2022 at 05:56:25PM +0400, Roman Arutyunyan wrote:
> > 
> > > Hi,
> > > 
> > > On Sun, Oct 30, 2022 at 05:42:33AM +0300, Maxim Dounin wrote:
> > > > # HG changeset patch
> > > > # User Maxim Dounin <mdounin at mdounin.ru>
> > > > # Date 1667097733 -10800
> > > > #      Sun Oct 30 05:42:13 2022 +0300
> > > > # Node ID ef9c94be7fe4685f0eeee41f76b964ea252f519f
> > > > # Parent  b73d95226c84b93e51f23f7b35782d98d3b516b9
> > > > Fixed segfault when switching off master process during upgrade.
> > > > 
> > > > Binary upgrades are not supported without master process, but it is,
> > > > however, possible, that nginx running with master process is asked
> > > > to upgrade binary, and the configuration file as available on disk
> > > > at this time includes "master_process off;".
> > > > 
> > > > If this happens, listening sockets inherited from the previous binary
> > > > will have ls[i].previous set.  But the old cycle on initial process
> > > > startup, including startup after binary upgrade, is destroyed by
> > > > ngx_init_cycle() once configuration parsing is complete.  As a result,
> > > > an attempt to dereference ls[i].previous in ngx_event_process_init()
> > > > accesses already freed memory.
> > > > 
> > > > Fix is to avoid looking into ls[i].previous if the old cycle is already
> > > > freed.
> > > > 
> > > > diff --git a/src/event/ngx_event.c b/src/event/ngx_event.c
> > > > --- a/src/event/ngx_event.c
> > > > +++ b/src/event/ngx_event.c
> > > > @@ -813,7 +813,9 @@ ngx_event_process_init(ngx_cycle_t *cycl
> > > >          rev->deferred_accept = ls[i].deferred_accept;
> > > >  #endif
> > > >  
> > > > -        if (!(ngx_event_flags & NGX_USE_IOCP_EVENT)) {
> > > > +        if (!(ngx_event_flags & NGX_USE_IOCP_EVENT)
> > > > +            && cycle->old_cycle)
> > > > +        {
> > > >              if (ls[i].previous) {
> > > >  
> > > >                  /*
> > > 
> > > Doesn't this also mean that we can throw away zeroing ls->previous?
> > > 
> > > diff --git a/src/os/unix/ngx_process_cycle.c b/src/os/unix/ngx_process_cycle.c
> > > --- a/src/os/unix/ngx_process_cycle.c
> > > +++ b/src/os/unix/ngx_process_cycle.c
> > > @@ -888,15 +888,6 @@ ngx_worker_process_init(ngx_cycle_t *cyc
> > >      tp = ngx_timeofday();
> > >      srandom(((unsigned) ngx_pid << 16) ^ tp->sec ^ tp->msec);
> > >  
> > > -    /*
> > > -     * disable deleting previous events for the listening sockets because
> > > -     * in the worker processes there are no events at all at this point
> > > -     */
> > > -    ls = cycle->listening.elts;
> > > -    for (i = 0; i < cycle->listening.nelts; i++) {
> > > -        ls[i].previous = NULL;
> > > -    }
> > > -
> > >      for (i = 0; cycle->modules[i]; i++) {
> > >          if (cycle->modules[i]->init_process) {
> > >              if (cycle->modules[i]->init_process(cycle) == NGX_ERROR) {
> > > 
> > 
> > Yes, this code is now redundant.  Added to the patch, with the 
> > following paragraph in the commit log:
> > 
> > : With this change it is also no longer needed to clear ls[i].previous in
> > : worker processes, so the relevant code was removed.
> > 
> > Full patch:
> > 
> > # HG changeset patch
> > # User Maxim Dounin <mdounin at mdounin.ru>
> > # Date 1669236533 -10800
> > #      Wed Nov 23 23:48:53 2022 +0300
> > # Node ID 8852f39311de9913435fb2f2fc601a3655f42a99
> > # Parent  09463dd9c50463e35856a3bd6b57bc6ca382e6f0
> > Fixed segfault when switching off master process during upgrade.
> > 
> > Binary upgrades are not supported without master process, but it is,
> > however, possible, that nginx running with master process is asked
> > to upgrade binary, and the configuration file as available on disk
> > at this time includes "master_process off;".
> > 
> > If this happens, listening sockets inherited from the previous binary
> > will have ls[i].previous set.  But the old cycle on initial process
> > startup, including startup after binary upgrade, is destroyed by
> > ngx_init_cycle() once configuration parsing is complete.  As a result,
> > an attempt to dereference ls[i].previous in ngx_event_process_init()
> > accesses already freed memory.
> > 
> > Fix is to avoid looking into ls[i].previous if the old cycle is already
> > freed.
> > 
> > With this change it is also no longer needed to clear ls[i].previous in
> > worker processes, so the relevant code was removed.
> > 
> > diff --git a/src/event/ngx_event.c b/src/event/ngx_event.c
> > --- a/src/event/ngx_event.c
> > +++ b/src/event/ngx_event.c
> > @@ -813,7 +813,9 @@ ngx_event_process_init(ngx_cycle_t *cycl
> >          rev->deferred_accept = ls[i].deferred_accept;
> >  #endif
> >  
> > -        if (!(ngx_event_flags & NGX_USE_IOCP_EVENT)) {
> > +        if (!(ngx_event_flags & NGX_USE_IOCP_EVENT)
> > +            && cycle->old_cycle)
> > +        {
> >              if (ls[i].previous) {
> >  
> >                  /*
> > diff --git a/src/os/unix/ngx_process_cycle.c b/src/os/unix/ngx_process_cycle.c
> > --- a/src/os/unix/ngx_process_cycle.c
> > +++ b/src/os/unix/ngx_process_cycle.c
> > @@ -759,7 +759,6 @@ ngx_worker_process_init(ngx_cycle_t *cyc
> >      ngx_cpuset_t     *cpu_affinity;
> >      struct rlimit     rlmt;
> >      ngx_core_conf_t  *ccf;
> > -    ngx_listening_t  *ls;
> >  
> >      if (ngx_set_environment(cycle, NULL) == NULL) {
> >          /* fatal */
> > @@ -889,15 +888,6 @@ ngx_worker_process_init(ngx_cycle_t *cyc
> >      tp = ngx_timeofday();
> >      srandom(((unsigned) ngx_pid << 16) ^ tp->sec ^ tp->msec);
> >  
> > -    /*
> > -     * disable deleting previous events for the listening sockets because
> > -     * in the worker processes there are no events at all at this point
> > -     */
> > -    ls = cycle->listening.elts;
> > -    for (i = 0; i < cycle->listening.nelts; i++) {
> > -        ls[i].previous = NULL;
> > -    }
> > -
> >      for (i = 0; cycle->modules[i]; i++) {
> >          if (cycle->modules[i]->init_process) {
> >              if (cycle->modules[i]->init_process(cycle) == NGX_ERROR) {
> 
> Looks good.

Pushed to http://mdounin.ru/hg/nginx, thanks.

-- 
Maxim Dounin
http://mdounin.ru/


From mdounin at mdounin.ru  Thu Nov 24 18:31:43 2022
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Thu, 24 Nov 2022 21:31:43 +0300
Subject: [PATCH 2 of 2] SSL: SSL_sendfile() support with kernel TLS
In-Reply-To: <DF6E1511-F69C-4AD8-88B7-B011E6A163E1@nginx.com>
References: <patchbomb.1632748686@vm-bsd.mdounin.ru>
 <ff514bf17f7f2257dcf0.1632748688@vm-bsd.mdounin.ru>
 <DF6E1511-F69C-4AD8-88B7-B011E6A163E1@nginx.com>
Message-ID: <Y3+4j8r2Wgaa5lhb@mdounin.ru>

Hello!

On Thu, Nov 24, 2022 at 01:21:51PM +0400, Sergey Kandaurov wrote:

> > On 27 Sep 2021, at 17:18, Maxim Dounin <mdounin at mdounin.ru> wrote:
> > 
> > # HG changeset patch
> > # User Maxim Dounin <mdounin at mdounin.ru>
> > # Date 1632717779 -10800
> > #      Mon Sep 27 07:42:59 2021 +0300
> > # Node ID ff514bf17f7f2257dcf036c5c973b74672cefa9a
> > # Parent  8f0fd60c33c106fba5f1ce3cafe990f15fcccc0c
> > SSL: SSL_sendfile() support with kernel TLS.
> > 
> > Requires OpenSSL 3.0 compiled with "enable-ktls" option.  Further, KTLS
> > needs to be enabled in kernel, and in OpenSSL, either via OpenSSL
> > configuration file or with "ssl_conf_command Options KTLS;" in nginx
> > configuration.
> > 
> > On FreeBSD, kernel TLS is available starting with FreeBSD 13.0, and
> > can be enabled with "sysctl kern.ipc.tls.enable=1" and "kldload ktls_ocf".
> > 
> > On Linux, kernel TLS is available starting with kernel 4.13 (at least 5.2
> > is recommended), and needs kernel compiled with CONFIG_TLS=y (with
> > CONFIG_TLS=m, which is used at least on Ubuntu 21.04 by default,
> > the tls module needs to be loaded with "modprobe tls").
> > 
> > diff --git a/src/event/ngx_event_openssl.c b/src/event/ngx_event_openssl.c
> 
> [..]
> 
> > @@ -2882,6 +2937,150 @@ ngx_ssl_write_early(ngx_connection_t *c,
> > #endif
> > 
> > 
> > +static ssize_t
> > +ngx_ssl_sendfile(ngx_connection_t *c, ngx_buf_t *file, size_t size)
> > +{
> > +#ifdef BIO_get_ktls_send
> > +
> > +    int        sslerr;
> > +    ssize_t    n;
> > +    ngx_err_t  err;
> > +
> > +    ngx_ssl_clear_error(c->log);
> > +
> > +    ngx_log_debug2(NGX_LOG_DEBUG_EVENT, c->log, 0,
> > +                   "SSL to sendfile: @%O %uz",
> > +                   file->file_pos, size);
> > +
> > +    ngx_set_errno(0);
> > +
> > +    n = SSL_sendfile(c->ssl->connection, file->file->fd, file->file_pos,
> > +                     size, 0);
> > +
> > +    ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0, "SSL_sendfile: %d", n);
> > +
> 
> Hello,
> 
> looks like a wrong format specifier slipped through review.
> Patch to address this:
> 
> # HG changeset patch
> # User Sergey Kandaurov <pluknet at nginx.com>
> # Date 1669241969 -14400
> #      Thu Nov 24 02:19:29 2022 +0400
> # Node ID 66ad8bd4b9e4347bda8fe64423632ded19093db2
> # Parent  75bfb3a97ca3d6292fb7877b89bb46720c15da80
> SSL: fixed debug logging of SSL_sendfile() return value.
> 
> diff --git a/src/event/ngx_event_openssl.c b/src/event/ngx_event_openssl.c
> --- a/src/event/ngx_event_openssl.c
> +++ b/src/event/ngx_event_openssl.c
> @@ -3020,7 +3020,7 @@ ngx_ssl_sendfile(ngx_connection_t *c, ng
>      n = SSL_sendfile(c->ssl->connection, file->file->fd, file->file_pos,
>                       size, flags);
>  
> -    ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0, "SSL_sendfile: %d", n);
> +    ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0, "SSL_sendfile: %z", n);
>  
>      if (n > 0) {
>  
> 

Looks good.

-- 
Maxim Dounin
http://mdounin.ru/


From jiuzhoucui at 163.com  Fri Nov 25 07:13:48 2022
From: jiuzhoucui at 163.com (Jiuzhou Cui)
Date: Fri, 25 Nov 2022 15:13:48 +0800 (CST)
Subject: HTTP/3: fixed build without NGX_PCRE (broken by 0f5fc7a320db).
Message-ID: <6d8b1e16.2efb.184ada2a53f.Coremail.jiuzhoucui@163.com>

Hello!

# HG changeset patch
# User Jiuzhou Cui <cuijiuzhou at alibaba-inc.com>
# Date 1669360043 -28800
#      Fri Nov 25 15:07:23 2022 +0800
# Branch quic
# Node ID da7d9eb93afad52bd5b231501590535be9660d8c
# Parent  c6580dce98a8951fcecce2daa4b6bdbda8307bf7
HTTP/3: fixed build without NGX_PCRE (broken by 0f5fc7a320db).
ssl_servername_regex defined under macro NGX_PCRE.

diff -r c6580dce98a8 -r da7d9eb93afa src/http/v3/ngx_http_v3_request.c
--- a/src/http/v3/ngx_http_v3_request.c Wed Nov 23 18:50:26 2022 +0400
+++ b/src/http/v3/ngx_http_v3_request.c Fri Nov 25 15:07:23 2022 +0800
@@ -81,7 +81,9 @@

     if (phc->ssl_servername) {
         hc->ssl_servername = phc->ssl_servername;
+#if (NGX_PCRE)
         hc->ssl_servername_regex = phc->ssl_servername_regex;
+#endif
         hc->conf_ctx = phc->conf_ctx;

         ngx_set_connection_log(c, clcf->error_log);

From thresh at nginx.com  Fri Nov 25 08:08:49 2022
From: thresh at nginx.com (=?iso-8859-1?q?Konstantin_Pavlov?=)
Date: Fri, 25 Nov 2022 12:08:49 +0400
Subject: [PATCH] Linux packages: actualized supported Alpine Linux versions
Message-ID: <a20b51e84c32af154412.1669363729@QGCD7XG9R9>

# HG changeset patch
# User Konstantin Pavlov <thresh at nginx.com>
# Date 1669360436 -14400
#      Fri Nov 25 11:13:56 2022 +0400
# Node ID a20b51e84c32af154412f0f11d0d890e7364d746
# Parent  7ebe15d6c68d6a7cad639a550fdf33d5bfdfbabb
Linux packages: actualized supported Alpine Linux versions.

diff -r 7ebe15d6c68d -r a20b51e84c32 xml/en/linux_packages.xml
--- a/xml/en/linux_packages.xml	Mon Nov 21 21:58:20 2022 +0000
+++ b/xml/en/linux_packages.xml	Fri Nov 25 11:13:56 2022 +0400
@@ -7,7 +7,7 @@
 <article name="nginx: Linux packages"
          link="/en/linux_packages.html"
          lang="en"
-         rev="81">
+         rev="82">
 
 <section name="Supported distributions and versions" id="distributions">
 
@@ -129,11 +129,6 @@ versions:
 </tr>
 
 <tr>
-<td width="30%">3.13</td>
-<td>x86_64, aarch64/arm64</td>
-</tr>
-
-<tr>
 <td width="30%">3.14</td>
 <td>x86_64, aarch64/arm64</td>
 </tr>
@@ -148,6 +143,11 @@ versions:
 <td>x86_64, aarch64/arm64</td>
 </tr>
 
+<tr>
+<td width="30%">3.17</td>
+<td>x86_64, aarch64/arm64</td>
+</tr>
+
 </table>
 </para>
 
diff -r 7ebe15d6c68d -r a20b51e84c32 xml/ru/linux_packages.xml
--- a/xml/ru/linux_packages.xml	Mon Nov 21 21:58:20 2022 +0000
+++ b/xml/ru/linux_packages.xml	Fri Nov 25 11:13:56 2022 +0400
@@ -7,7 +7,7 @@
 <article name="nginx: пакеты для Linux"
          link="/ru/linux_packages.html"
          lang="ru"
-         rev="81">
+         rev="82">
 
 <section name="Поддерживаемые дистрибутивы и версии" id="distributions">
 
@@ -129,11 +129,6 @@
 </tr>
 
 <tr>
-<td width="30%">3.13</td>
-<td>x86_64, aarch64/arm64</td>
-</tr>
-
-<tr>
 <td width="30%">3.14</td>
 <td>x86_64, aarch64/arm64</td>
 </tr>
@@ -148,6 +143,11 @@
 <td>x86_64, aarch64/arm64</td>
 </tr>
 
+<tr>
+<td width="30%">3.17</td>
+<td>x86_64, aarch64/arm64</td>
+</tr>
+
 </table>
 </para>
 

From agarwal.aakarshit13 at gmail.com  Fri Nov 25 08:11:37 2022
From: agarwal.aakarshit13 at gmail.com (Aakarshit Agarwal)
Date: Fri, 25 Nov 2022 13:41:37 +0530
Subject: How can I redirect a request via Another External proxy
Message-ID: <CALiXuobyUAjj_LaJhoAAYQ5WkKxufu3bHFHLu83qG-1g5PXkJA@mail.gmail.com>

Hi Team,

Inside a private network, I want to redirect a request via another External
proxy.

Brief ->

I want my URL ( https://example1.com ) when having the path specified as (
https://example1.com/tunnel/portnumber ) it will redirect my request to
another URL
( https://example2/api/portnumber/xyz/ ) *via an External proxy ( x.x.x.x )*

I tried many nginx.conf but was not able to redirect my request.

Please help me to know if it is possible, and if yes in what way.

Tech Enthusiast,
Aakarshit Agarwal
DevOps Engineer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20221125/dd649e54/attachment.htm>

From pluknet at nginx.com  Fri Nov 25 11:47:16 2022
From: pluknet at nginx.com (Sergey Kandaurov)
Date: Fri, 25 Nov 2022 11:47:16 +0000
Subject: [nginx] Filtering duplicate addresses in listen (ticket #2400).
Message-ID: <hg.4cc2bfeff46c.1669376836.6026610855610030274@dev.nginx>

details:   https://hg.nginx.org/nginx/rev/4cc2bfeff46c
branches:  
changeset: 8104:4cc2bfeff46c
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Wed Nov 23 17:30:08 2022 +0300
description:
Filtering duplicate addresses in listen (ticket #2400).

Due to the glibc bug[1], getaddrinfo("localhost") with AI_ADDRCONFIG
on a typical host with glibc and without IPv6 returns two 127.0.0.1
addresses, and therefore "listen localhost:80;" used to result in
"duplicate ... address and port pair" after 4f9b72a229c1.

Fix is to explicitly filter out duplicate addresses returned during
resolution of a name.

[1] https://sourceware.org/bugzilla/show_bug.cgi?id=14969

diffstat:

 src/http/ngx_http_core_module.c     |  15 ++++++++++-
 src/mail/ngx_mail_core_module.c     |  47 ++++++++++++++++++++++++++---------
 src/stream/ngx_stream_core_module.c |  49 +++++++++++++++++++++++++++---------
 3 files changed, 85 insertions(+), 26 deletions(-)

diffs (213 lines):

diff -r b809f53d3f5b -r 4cc2bfeff46c src/http/ngx_http_core_module.c
--- a/src/http/ngx_http_core_module.c	Fri Nov 18 19:31:38 2022 +0400
+++ b/src/http/ngx_http_core_module.c	Wed Nov 23 17:30:08 2022 +0300
@@ -3963,7 +3963,7 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx
 
     ngx_str_t              *value, size;
     ngx_url_t               u;
-    ngx_uint_t              n;
+    ngx_uint_t              n, i;
     ngx_http_listen_opt_t   lsopt;
 
     cscf->listen = 1;
@@ -4289,6 +4289,16 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx
     }
 
     for (n = 0; n < u.naddrs; n++) {
+
+        for (i = 0; i < n; i++) {
+            if (ngx_cmp_sockaddr(u.addrs[n].sockaddr, u.addrs[n].socklen,
+                                 u.addrs[i].sockaddr, u.addrs[i].socklen, 0)
+                == NGX_OK)
+            {
+                goto next;
+            }
+        }
+
         lsopt.sockaddr = u.addrs[n].sockaddr;
         lsopt.socklen = u.addrs[n].socklen;
         lsopt.addr_text = u.addrs[n].name;
@@ -4297,6 +4307,9 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx
         if (ngx_http_add_listen(cf, cscf, &lsopt) != NGX_OK) {
             return NGX_CONF_ERROR;
         }
+
+    next:
+        continue;
     }
 
     return NGX_CONF_OK;
diff -r b809f53d3f5b -r 4cc2bfeff46c src/mail/ngx_mail_core_module.c
--- a/src/mail/ngx_mail_core_module.c	Fri Nov 18 19:31:38 2022 +0400
+++ b/src/mail/ngx_mail_core_module.c	Wed Nov 23 17:30:08 2022 +0300
@@ -308,7 +308,7 @@ ngx_mail_core_listen(ngx_conf_t *cf, ngx
     ngx_str_t                  *value, size;
     ngx_url_t                   u;
     ngx_uint_t                  i, n, m;
-    ngx_mail_listen_t          *ls, *als;
+    ngx_mail_listen_t          *ls, *als, *nls;
     ngx_mail_module_t          *module;
     ngx_mail_core_main_conf_t  *cmcf;
 
@@ -333,7 +333,7 @@ ngx_mail_core_listen(ngx_conf_t *cf, ngx
 
     cmcf = ngx_mail_conf_get_module_main_conf(cf, ngx_mail_core_module);
 
-    ls = ngx_array_push_n(&cmcf->listen, u.naddrs);
+    ls = ngx_array_push(&cmcf->listen);
     if (ls == NULL) {
         return NGX_CONF_ERROR;
     }
@@ -568,20 +568,40 @@ ngx_mail_core_listen(ngx_conf_t *cf, ngx
         return NGX_CONF_ERROR;
     }
 
-    als = cmcf->listen.elts;
+    for (n = 0; n < u.naddrs; n++) {
 
-    for (n = 0; n < u.naddrs; n++) {
-        ls[n] = ls[0];
+        for (i = 0; i < n; i++) {
+            if (ngx_cmp_sockaddr(u.addrs[n].sockaddr, u.addrs[n].socklen,
+                                 u.addrs[i].sockaddr, u.addrs[i].socklen, 0)
+                == NGX_OK)
+            {
+                goto next;
+            }
+        }
 
-        ls[n].sockaddr = u.addrs[n].sockaddr;
-        ls[n].socklen = u.addrs[n].socklen;
-        ls[n].addr_text = u.addrs[n].name;
-        ls[n].wildcard = ngx_inet_wildcard(ls[n].sockaddr);
+        if (n != 0) {
+            nls = ngx_array_push(&cmcf->listen);
+            if (nls == NULL) {
+                return NGX_CONF_ERROR;
+            }
+
+            *nls = *ls;
 
-        for (i = 0; i < cmcf->listen.nelts - u.naddrs + n; i++) {
+        } else {
+            nls = ls;
+        }
+
+        nls->sockaddr = u.addrs[n].sockaddr;
+        nls->socklen = u.addrs[n].socklen;
+        nls->addr_text = u.addrs[n].name;
+        nls->wildcard = ngx_inet_wildcard(nls->sockaddr);
+
+        als = cmcf->listen.elts;
+
+        for (i = 0; i < cmcf->listen.nelts - 1; i++) {
 
             if (ngx_cmp_sockaddr(als[i].sockaddr, als[i].socklen,
-                                 ls[n].sockaddr, ls[n].socklen, 1)
+                                 nls->sockaddr, nls->socklen, 1)
                 != NGX_OK)
             {
                 continue;
@@ -589,9 +609,12 @@ ngx_mail_core_listen(ngx_conf_t *cf, ngx
 
             ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                                "duplicate \"%V\" address and port pair",
-                               &ls[n].addr_text);
+                               &nls->addr_text);
             return NGX_CONF_ERROR;
         }
+
+    next:
+        continue;
     }
 
     return NGX_CONF_OK;
diff -r b809f53d3f5b -r 4cc2bfeff46c src/stream/ngx_stream_core_module.c
--- a/src/stream/ngx_stream_core_module.c	Fri Nov 18 19:31:38 2022 +0400
+++ b/src/stream/ngx_stream_core_module.c	Wed Nov 23 17:30:08 2022 +0300
@@ -578,7 +578,7 @@ ngx_stream_core_listen(ngx_conf_t *cf, n
     ngx_str_t                    *value, size;
     ngx_url_t                     u;
     ngx_uint_t                    i, n, backlog;
-    ngx_stream_listen_t          *ls, *als;
+    ngx_stream_listen_t          *ls, *als, *nls;
     ngx_stream_core_main_conf_t  *cmcf;
 
     cscf->listen = 1;
@@ -602,7 +602,7 @@ ngx_stream_core_listen(ngx_conf_t *cf, n
 
     cmcf = ngx_stream_conf_get_module_main_conf(cf, ngx_stream_core_module);
 
-    ls = ngx_array_push_n(&cmcf->listen, u.naddrs);
+    ls = ngx_array_push(&cmcf->listen);
     if (ls == NULL) {
         return NGX_CONF_ERROR;
     }
@@ -886,23 +886,43 @@ ngx_stream_core_listen(ngx_conf_t *cf, n
 #endif
     }
 
-    als = cmcf->listen.elts;
+    for (n = 0; n < u.naddrs; n++) {
 
-    for (n = 0; n < u.naddrs; n++) {
-        ls[n] = ls[0];
+        for (i = 0; i < n; i++) {
+            if (ngx_cmp_sockaddr(u.addrs[n].sockaddr, u.addrs[n].socklen,
+                                 u.addrs[i].sockaddr, u.addrs[i].socklen, 0)
+                == NGX_OK)
+            {
+                goto next;
+            }
+        }
 
-        ls[n].sockaddr = u.addrs[n].sockaddr;
-        ls[n].socklen = u.addrs[n].socklen;
-        ls[n].addr_text = u.addrs[n].name;
-        ls[n].wildcard = ngx_inet_wildcard(ls[n].sockaddr);
+        if (n != 0) {
+            nls = ngx_array_push(&cmcf->listen);
+            if (nls == NULL) {
+                return NGX_CONF_ERROR;
+            }
+
+            *nls = *ls;
 
-        for (i = 0; i < cmcf->listen.nelts - u.naddrs + n; i++) {
-            if (ls[n].type != als[i].type) {
+        } else {
+            nls = ls;
+        }
+
+        nls->sockaddr = u.addrs[n].sockaddr;
+        nls->socklen = u.addrs[n].socklen;
+        nls->addr_text = u.addrs[n].name;
+        nls->wildcard = ngx_inet_wildcard(nls->sockaddr);
+
+        als = cmcf->listen.elts;
+
+        for (i = 0; i < cmcf->listen.nelts - 1; i++) {
+            if (nls->type != als[i].type) {
                 continue;
             }
 
             if (ngx_cmp_sockaddr(als[i].sockaddr, als[i].socklen,
-                                 ls[n].sockaddr, ls[n].socklen, 1)
+                                 nls->sockaddr, nls->socklen, 1)
                 != NGX_OK)
             {
                 continue;
@@ -910,9 +930,12 @@ ngx_stream_core_listen(ngx_conf_t *cf, n
 
             ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                                "duplicate \"%V\" address and port pair",
-                               &ls[n].addr_text);
+                               &nls->addr_text);
             return NGX_CONF_ERROR;
         }
+
+    next:
+        continue;
     }
 
     return NGX_CONF_OK;


From pluknet at nginx.com  Fri Nov 25 11:47:19 2022
From: pluknet at nginx.com (Sergey Kandaurov)
Date: Fri, 25 Nov 2022 11:47:19 +0000
Subject: [nginx] Disabled cloning of sockets without master process (ticket
 #2403).
Message-ID: <hg.09463dd9c504.1669376839.6026610855610030274@dev.nginx>

details:   https://hg.nginx.org/nginx/rev/09463dd9c504
branches:  
changeset: 8105:09463dd9c504
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Wed Nov 23 23:12:04 2022 +0300
description:
Disabled cloning of sockets without master process (ticket #2403).

Cloning of listening sockets for each worker process does not make sense
when working without master process, and causes some of the connections
not to be accepted if worker_processes is set to more than one and there
are listening sockets configured with the reuseport flag.  Fix is to
disable cloning when master process is disabled.

diffstat:

 src/event/ngx_event.c |  5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)

diffs (22 lines):

diff -r 4cc2bfeff46c -r 09463dd9c504 src/event/ngx_event.c
--- a/src/event/ngx_event.c	Wed Nov 23 17:30:08 2022 +0300
+++ b/src/event/ngx_event.c	Wed Nov 23 23:12:04 2022 +0300
@@ -416,6 +416,7 @@ ngx_event_init_conf(ngx_cycle_t *cycle, 
 {
 #if (NGX_HAVE_REUSEPORT)
     ngx_uint_t        i;
+    ngx_core_conf_t  *ccf;
     ngx_listening_t  *ls;
 #endif
 
@@ -442,7 +443,9 @@ ngx_event_init_conf(ngx_cycle_t *cycle, 
 
 #if (NGX_HAVE_REUSEPORT)
 
-    if (!ngx_test_config) {
+    ccf = (ngx_core_conf_t *) ngx_get_conf(cycle->conf_ctx, ngx_core_module);
+
+    if (!ngx_test_config && ccf->master) {
 
         ls = cycle->listening.elts;
         for (i = 0; i < cycle->listening.nelts; i++) {


From pluknet at nginx.com  Fri Nov 25 11:47:22 2022
From: pluknet at nginx.com (Sergey Kandaurov)
Date: Fri, 25 Nov 2022 11:47:22 +0000
Subject: [nginx] Fixed segfault when switching off master process during
 upgrade.
Message-ID: <hg.8852f39311de.1669376842.6026610855610030274@dev.nginx>

details:   https://hg.nginx.org/nginx/rev/8852f39311de
branches:  
changeset: 8106:8852f39311de
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Wed Nov 23 23:48:53 2022 +0300
description:
Fixed segfault when switching off master process during upgrade.

Binary upgrades are not supported without master process, but it is,
however, possible, that nginx running with master process is asked
to upgrade binary, and the configuration file as available on disk
at this time includes "master_process off;".

If this happens, listening sockets inherited from the previous binary
will have ls[i].previous set.  But the old cycle on initial process
startup, including startup after binary upgrade, is destroyed by
ngx_init_cycle() once configuration parsing is complete.  As a result,
an attempt to dereference ls[i].previous in ngx_event_process_init()
accesses already freed memory.

Fix is to avoid looking into ls[i].previous if the old cycle is already
freed.

With this change it is also no longer needed to clear ls[i].previous in
worker processes, so the relevant code was removed.

diffstat:

 src/event/ngx_event.c           |   4 +++-
 src/os/unix/ngx_process_cycle.c |  10 ----------
 2 files changed, 3 insertions(+), 11 deletions(-)

diffs (41 lines):

diff -r 09463dd9c504 -r 8852f39311de src/event/ngx_event.c
--- a/src/event/ngx_event.c	Wed Nov 23 23:12:04 2022 +0300
+++ b/src/event/ngx_event.c	Wed Nov 23 23:48:53 2022 +0300
@@ -813,7 +813,9 @@ ngx_event_process_init(ngx_cycle_t *cycl
         rev->deferred_accept = ls[i].deferred_accept;
 #endif
 
-        if (!(ngx_event_flags & NGX_USE_IOCP_EVENT)) {
+        if (!(ngx_event_flags & NGX_USE_IOCP_EVENT)
+            && cycle->old_cycle)
+        {
             if (ls[i].previous) {
 
                 /*
diff -r 09463dd9c504 -r 8852f39311de src/os/unix/ngx_process_cycle.c
--- a/src/os/unix/ngx_process_cycle.c	Wed Nov 23 23:12:04 2022 +0300
+++ b/src/os/unix/ngx_process_cycle.c	Wed Nov 23 23:48:53 2022 +0300
@@ -759,7 +759,6 @@ ngx_worker_process_init(ngx_cycle_t *cyc
     ngx_cpuset_t     *cpu_affinity;
     struct rlimit     rlmt;
     ngx_core_conf_t  *ccf;
-    ngx_listening_t  *ls;
 
     if (ngx_set_environment(cycle, NULL) == NULL) {
         /* fatal */
@@ -889,15 +888,6 @@ ngx_worker_process_init(ngx_cycle_t *cyc
     tp = ngx_timeofday();
     srandom(((unsigned) ngx_pid << 16) ^ tp->sec ^ tp->msec);
 
-    /*
-     * disable deleting previous events for the listening sockets because
-     * in the worker processes there are no events at all at this point
-     */
-    ls = cycle->listening.elts;
-    for (i = 0; i < cycle->listening.nelts; i++) {
-        ls[i].previous = NULL;
-    }
-
     for (i = 0; cycle->modules[i]; i++) {
         if (cycle->modules[i]->init_process) {
             if (cycle->modules[i]->init_process(cycle) == NGX_ERROR) {


From pluknet at nginx.com  Fri Nov 25 11:47:25 2022
From: pluknet at nginx.com (Sergey Kandaurov)
Date: Fri, 25 Nov 2022 11:47:25 +0000
Subject: [nginx] SSL: fixed debug logging of SSL_sendfile() return value.
Message-ID: <hg.0b360747c74e.1669376845.6026610855610030274@dev.nginx>

details:   https://hg.nginx.org/nginx/rev/0b360747c74e
branches:  
changeset: 8107:0b360747c74e
user:      Sergey Kandaurov <pluknet at nginx.com>
date:      Thu Nov 24 23:08:30 2022 +0400
description:
SSL: fixed debug logging of SSL_sendfile() return value.

diffstat:

 src/event/ngx_event_openssl.c |  2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diffs (12 lines):

diff -r 8852f39311de -r 0b360747c74e src/event/ngx_event_openssl.c
--- a/src/event/ngx_event_openssl.c	Wed Nov 23 23:48:53 2022 +0300
+++ b/src/event/ngx_event_openssl.c	Thu Nov 24 23:08:30 2022 +0400
@@ -3020,7 +3020,7 @@ ngx_ssl_sendfile(ngx_connection_t *c, ng
     n = SSL_sendfile(c->ssl->connection, file->file->fd, file->file_pos,
                      size, flags);
 
-    ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0, "SSL_sendfile: %d", n);
+    ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0, "SSL_sendfile: %z", n);
 
     if (n > 0) {
 


From pluknet at nginx.com  Fri Nov 25 11:52:14 2022
From: pluknet at nginx.com (Sergey Kandaurov)
Date: Fri, 25 Nov 2022 15:52:14 +0400
Subject: [PATCH] Linux packages: actualized supported Alpine Linux
 versions
In-Reply-To: <a20b51e84c32af154412.1669363729@QGCD7XG9R9>
References: <a20b51e84c32af154412.1669363729@QGCD7XG9R9>
Message-ID: <941BA85C-9940-4F0A-B167-EEACBE7D8F3D@nginx.com>


> On 25 Nov 2022, at 12:08, Konstantin Pavlov <thresh at nginx.com> wrote:
> 
> # HG changeset patch
> # User Konstantin Pavlov <thresh at nginx.com>
> # Date 1669360436 -14400
> #      Fri Nov 25 11:13:56 2022 +0400
> # Node ID a20b51e84c32af154412f0f11d0d890e7364d746
> # Parent  7ebe15d6c68d6a7cad639a550fdf33d5bfdfbabb
> Linux packages: actualized supported Alpine Linux versions.
> 
> [..]


Looks good.

-- 
Sergey Kandaurov


From pluknet at nginx.com  Fri Nov 25 11:56:27 2022
From: pluknet at nginx.com (Sergey Kandaurov)
Date: Fri, 25 Nov 2022 15:56:27 +0400
Subject: HTTP/3: fixed build without NGX_PCRE (broken by 0f5fc7a320db).
In-Reply-To: <6d8b1e16.2efb.184ada2a53f.Coremail.jiuzhoucui@163.com>
References: <6d8b1e16.2efb.184ada2a53f.Coremail.jiuzhoucui@163.com>
Message-ID: <ED57560F-14F6-46B3-BB7C-C46CE0FB02E6@nginx.com>


> On 25 Nov 2022, at 11:13, Jiuzhou Cui <jiuzhoucui at 163.com> wrote:
> 
> Hello!
> 
> # HG changeset patch
> # User Jiuzhou Cui <cuijiuzhou at alibaba-inc.com>
> # Date 1669360043 -28800
> #      Fri Nov 25 15:07:23 2022 +0800
> # Branch quic
> # Node ID da7d9eb93afad52bd5b231501590535be9660d8c
> # Parent  c6580dce98a8951fcecce2daa4b6bdbda8307bf7
> HTTP/3: fixed build without NGX_PCRE (broken by 0f5fc7a320db).
> ssl_servername_regex defined under macro NGX_PCRE.
> 
> diff -r c6580dce98a8 -r da7d9eb93afa src/http/v3/ngx_http_v3_request.c
> --- a/src/http/v3/ngx_http_v3_request.c Wed Nov 23 18:50:26 2022 +0400
> +++ b/src/http/v3/ngx_http_v3_request.c Fri Nov 25 15:07:23 2022 +0800
> @@ -81,7 +81,9 @@
> 
>     if (phc->ssl_servername) {
>         hc->ssl_servername = phc->ssl_servername;
> +#if (NGX_PCRE)
>         hc->ssl_servername_regex = phc->ssl_servername_regex;
> +#endif
>         hc->conf_ctx = phc->conf_ctx;
> 
>         ngx_set_connection_log(c, clcf->error_log);

Committed (with minor adjustment), thanks!

https://hg.nginx.org/nginx-quic/rev/210ad79a8853

-- 
Sergey Kandaurov


From pluknet at nginx.com  Fri Nov 25 12:00:57 2022
From: pluknet at nginx.com (Sergey Kandaurov)
Date: Fri, 25 Nov 2022 16:00:57 +0400
Subject: QUIC: fixed computation of nonce
In-Reply-To: <20221124153351.ablijt3rinybgrl5@N00W24XTQX>
References: <6e54119a.44a7.1848ba488b5.Coremail.yolkking@126.com>
 <D561E0F8-1395-4864-AEF4-C5FEA8E41307@nginx.com>
 <20221124153351.ablijt3rinybgrl5@N00W24XTQX>
Message-ID: <2D49B23D-E264-4466-B5E2-761FFD1537E1@nginx.com>


> On 24 Nov 2022, at 19:33, Roman Arutyunyan <arut at nginx.com> wrote:
> 
> Hi,
> 
> On Tue, Nov 22, 2022 at 02:29:47PM +0400, Sergey Kandaurov wrote:
>> 
>>> On 18 Nov 2022, at 20:48, Yu Zhu <yolkking at 126.com> wrote:
>>> 
>>> # HG changeset patch
>>> # User Yu Zhu <lishu.zy at alibaba-inc.com>
>>> # Date 1668789115 -28800
>>> #      Sat Nov 19 00:31:55 2022 +0800
>>> # Branch quic
>>> # Node ID 1a320805265db14904ca9deaae8330f4979619ce
>>> # Parent  6cf8ed15fd00668b7efa0226c06f47d7238f26e8
>>> QUIC: fixed computation of nonce
>>> 
>>> RFC 9001, 5.3. AEAD Usage
>>> The nonce, N, is formed by combining the packet protection IV with the packet number. The 62 bits of the reconstructed QUIC packet number in network byte order are left-padded with zeros to the size of the IV. The exclusive OR of the padded packet number and the IV forms the AEAD nonce.
>> 
>> While citing RFCs is certainly good, I would limit this
>> to a specific description what this patch intends to do.
>> 
>>> 
>>> diff -r 6cf8ed15fd00 -r 1a320805265d src/event/quic/ngx_event_quic_protection.c
>>> --- a/src/event/quic/ngx_event_quic_protection.c        Tue Nov 01 17:00:35 2022 +0400
>>> +++ b/src/event/quic/ngx_event_quic_protection.c        Sat Nov 19 00:31:55 2022 +0800
>>> @@ -969,10 +969,11 @@
>>> static void
>>> ngx_quic_compute_nonce(u_char *nonce, size_t len, uint64_t pn)
>>> {
>>> -    nonce[len - 4] ^= (pn & 0xff000000) >> 24;
>>> -    nonce[len - 3] ^= (pn & 0x00ff0000) >> 16;
>>> -    nonce[len - 2] ^= (pn & 0x0000ff00) >> 8;
>>> -    nonce[len - 1] ^= (pn & 0x000000ff);
>>> +    size_t  i;
>>> +
>>> +    for (i = 0; i < 8; i++) {
>>> +        nonce[len - 8 + i] ^= (pn >> (8 - i - 1) * 8) & 0xff;
>>> +    }
>>> }
>> 
>> Thanks for the patch.
>> I tend to prefer computation with unrolled loops, though.
>> Microbenchmarking shows that it performs better.
>> Also, this allows to peel two high bits without additional branching.
>> 
>> 100m loop, user time in seconds
>> 
>>                -O0     -O1     -O2
>> gcc12/arm64
>> old             0.609   0.233   0.024
>> new             1.030   0.454   0.683
>> clang14/arm64
>> old             0.447   0.256   0.096*
>> new             1.292   0.453   0.096*
>> gcc8/i386
>> old             1.519   0.541   0.551
>> new             2.915   1.318   1.476
>> 
>> # HG changeset patch
>> # User Sergey Kandaurov <pluknet at nginx.com>
>> # Date 1669112795 -14400
>> #      Tue Nov 22 14:26:35 2022 +0400
>> # Branch quic
>> # Node ID 29d8d12b6d35eb172465db2dff47bd8e98b36fc7
>> # Parent  0f5fc7a320db621da8731a8832f486c7167b236b
>> QUIC: fixed computation of nonce with packet numbers beyond 2^32.
>> 
>> Prodded by Yu Zhu.
>> 
>> diff --git a/src/event/quic/ngx_event_quic_protection.c b/src/event/quic/ngx_event_quic_protection.c
>> --- a/src/event/quic/ngx_event_quic_protection.c
>> +++ b/src/event/quic/ngx_event_quic_protection.c
>> @@ -969,10 +969,14 @@ ngx_quic_parse_pn(u_char **pos, ngx_int_
>> static void
>> ngx_quic_compute_nonce(u_char *nonce, size_t len, uint64_t pn)
>> {
>> -    nonce[len - 4] ^= (pn & 0xff000000) >> 24;
>> -    nonce[len - 3] ^= (pn & 0x00ff0000) >> 16;
>> -    nonce[len - 2] ^= (pn & 0x0000ff00) >> 8;
>> -    nonce[len - 1] ^= (pn & 0x000000ff);
>> +    nonce[len - 8] ^= (pn >> 56) & 0x3f;
>> +    nonce[len - 7] ^= (pn >> 48) & 0xff;
>> +    nonce[len - 6] ^= (pn >> 40) & 0xff;
>> +    nonce[len - 5] ^= (pn >> 32) & 0xff;
>> +    nonce[len - 4] ^= (pn >> 24) & 0xff;
>> +    nonce[len - 3] ^= (pn >> 16) & 0xff;
>> +    nonce[len - 2] ^= (pn >> 8) & 0xff;
>> +    nonce[len - 1] ^= pn & 0xff;
>> }
> 
> Looks ok

https://hg.nginx.org/nginx-quic/rev/70ce1e927715

-- 
Sergey Kandaurov


From ciapnz at gmail.com  Sat Nov 26 17:01:54 2022
From: ciapnz at gmail.com (Danila Vershinin)
Date: Sun, 27 Nov 2022 01:01:54 +0800
Subject: Allowing the etag directive within if in location
Message-ID: <CADeKGy-FzrxmZ8ev5Zp1ti8XegRZvHMjNm160KZkgsO994OzFw@mail.gmail.com>

Hi,

For our purposes, we need to be able to turn off etag under specific
conditions, e.g.:

location / {
    if ($condition) {
        etag off;
    }
}

However, this syntax is invalid because this directive doesn't include
the NGX_HTTP_LIF_CONF flag.

We added that flag. Then recompiled NGINX. So now:

* the syntax is accepted by NGINX
* at runtime it works as desired

Are there any potential problems with this approach? Performance or
otherwise?

Best regards,
Danila
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20221127/64782994/attachment.htm>

From mdounin at mdounin.ru  Sat Nov 26 17:37:37 2022
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Sat, 26 Nov 2022 20:37:37 +0300
Subject: Allowing the etag directive within if in location
In-Reply-To: <CADeKGy-FzrxmZ8ev5Zp1ti8XegRZvHMjNm160KZkgsO994OzFw@mail.gmail.com>
References: <CADeKGy-FzrxmZ8ev5Zp1ti8XegRZvHMjNm160KZkgsO994OzFw@mail.gmail.com>
Message-ID: <Y4JO4UTa7xWpxnAn@mdounin.ru>

Hello!

On Sun, Nov 27, 2022 at 01:01:54AM +0800, Danila Vershinin wrote:

> Hi,
> 
> For our purposes, we need to be able to turn off etag under specific
> conditions, e.g.:
> 
> location / {
>     if ($condition) {
>         etag off;
>     }
> }
> 
> However, this syntax is invalid because this directive doesn't include
> the NGX_HTTP_LIF_CONF flag.
> 
> We added that flag. Then recompiled NGINX. So now:
> 
> * the syntax is accepted by NGINX
> * at runtime it works as desired
> 
> Are there any potential problems with this approach? Performance or
> otherwise?

The generic issue with the "if" directive within a location 
context is that it creates a separate configuration, and therefore 
works in a counter-intuitive way in many cases (including being 
counter-intuitive for developers, which results in various bugs), 
see IfIsEvil wiki article for the details.  Due to this, no new 
directives are allowed to work in the if-in-location context.

Other than that, it probably should work fine assuming it works 
for you now and you are ok to maintain the patch yourself.

A more portable solution might be to use

    map $condition $etag {
        default    $sent_http_etag;
        1          "";
    }

    add_header ETag $etag;

to conditionally remove the ETag header.

-- 
Maxim Dounin
http://mdounin.ru/


From ciapnz at gmail.com  Sat Nov 26 18:31:20 2022
From: ciapnz at gmail.com (Danila Vershinin)
Date: Sun, 27 Nov 2022 02:31:20 +0800
Subject: Allowing the etag directive within if in location
In-Reply-To: <Y4JO4UTa7xWpxnAn@mdounin.ru>
References: <CADeKGy-FzrxmZ8ev5Zp1ti8XegRZvHMjNm160KZkgsO994OzFw@mail.gmail.com>
 <Y4JO4UTa7xWpxnAn@mdounin.ru>
Message-ID: <CADeKGy_cHzfc-QNWhBExtBZjnzwaVeOKo5DWHoC+mbHj216Uaw@mail.gmail.com>

Thank you for the answer. A more generic question follows:

While we understand that IfIsEvil mostly applies to "being evil inside
location context."...

We notice some directives allow being placed within "if in location" but
not within "if" in the server context.

An example is gzip module's main directive:
http://nginx.org/en/docs/http/ngx_http_gzip_module.html#gzip

Context: http, server, location, if in location

So "gzip on" can be done in "if in location", but NGINX does not allow this
to be done in just "if" (which seems not an "evil" thing to do).

Is there a specific rationale for why many NGINX directives are not allowed
in server-level "if" but are allowed in "if in location"?

Consequentially, is there any problem in allowing server-level `if` context
for such directives by adding corresponding flags?
And is there are problem in doing the same for directives that doesn't
support any kind of "if" (neither `if` in server nor `if in location`),
like `gzip_proxied` directive?

Best regards,
Danila


On Sun, Nov 27, 2022 at 1:38 AM Maxim Dounin <mdounin at mdounin.ru> wrote:

> Hello!
>
> On Sun, Nov 27, 2022 at 01:01:54AM +0800, Danila Vershinin wrote:
>
> > Hi,
> >
> > For our purposes, we need to be able to turn off etag under specific
> > conditions, e.g.:
> >
> > location / {
> >     if ($condition) {
> >         etag off;
> >     }
> > }
> >
> > However, this syntax is invalid because this directive doesn't include
> > the NGX_HTTP_LIF_CONF flag.
> >
> > We added that flag. Then recompiled NGINX. So now:
> >
> > * the syntax is accepted by NGINX
> > * at runtime it works as desired
> >
> > Are there any potential problems with this approach? Performance or
> > otherwise?
>
> The generic issue with the "if" directive within a location
> context is that it creates a separate configuration, and therefore
> works in a counter-intuitive way in many cases (including being
> counter-intuitive for developers, which results in various bugs),
> see IfIsEvil wiki article for the details.  Due to this, no new
> directives are allowed to work in the if-in-location context.
>
> Other than that, it probably should work fine assuming it works
> for you now and you are ok to maintain the patch yourself.
>
> A more portable solution might be to use
>
>     map $condition $etag {
>         default    $sent_http_etag;
>         1          "";
>     }
>
>     add_header ETag $etag;
>
> to conditionally remove the ETag header.
>
> --
> Maxim Dounin
> http://mdounin.ru/
> _______________________________________________
> nginx-devel mailing list -- nginx-devel at nginx.org
> To unsubscribe send an email to nginx-devel-leave at nginx.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20221127/408fb179/attachment.htm>

From mdounin at mdounin.ru  Sat Nov 26 21:07:19 2022
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Sun, 27 Nov 2022 00:07:19 +0300
Subject: Allowing the etag directive within if in location
In-Reply-To: <CADeKGy_cHzfc-QNWhBExtBZjnzwaVeOKo5DWHoC+mbHj216Uaw@mail.gmail.com>
References: <CADeKGy-FzrxmZ8ev5Zp1ti8XegRZvHMjNm160KZkgsO994OzFw@mail.gmail.com>
 <Y4JO4UTa7xWpxnAn@mdounin.ru>
 <CADeKGy_cHzfc-QNWhBExtBZjnzwaVeOKo5DWHoC+mbHj216Uaw@mail.gmail.com>
Message-ID: <Y4KAB1eLG89mYqH+@mdounin.ru>

Hello!

On Sun, Nov 27, 2022 at 02:31:20AM +0800, Danila Vershinin wrote:

> Thank you for the answer. A more generic question follows:
> 
> While we understand that IfIsEvil mostly applies to "being evil inside
> location context."...
> 
> We notice some directives allow being placed within "if in location" but
> not within "if" in the server context.
> 
> An example is gzip module's main directive:
> http://nginx.org/en/docs/http/ngx_http_gzip_module.html#gzip
> 
> Context: http, server, location, if in location
> 
> So "gzip on" can be done in "if in location", but NGINX does not allow this
> to be done in just "if" (which seems not an "evil" thing to do).
> 
> Is there a specific rationale for why many NGINX directives are not allowed
> in server-level "if" but are allowed in "if in location"?
> 
> Consequentially, is there any problem in allowing server-level `if` context
> for such directives by adding corresponding flags?
> And is there are problem in doing the same for directives that doesn't
> support any kind of "if" (neither `if` in server nor `if in location`),
> like `gzip_proxied` directive?

In the server context, only rewrite directives are allowed 
inside "if", and "if" works as it should: just provides an 
conditional operator within the rewrite module instructions, and 
does not try to create an implicit location configuration.

For obvious reasons you can't just allow non-rewrite directives 
within server-level "if" blocks: you have to make them work 
somehow.  Trying to do so will likely create even bigger issues 
than the current if-in-location problems.

Summing the above: yes, there is a problem.

-- 
Maxim Dounin
http://mdounin.ru/


From jerome at loyet.net  Mon Nov 28 05:40:33 2022
From: jerome at loyet.net (=?UTF-8?B?SsOpcsO0bWUgTG95ZXQ=?=)
Date: Mon, 28 Nov 2022 06:40:33 +0100
Subject: switch to ms resolution for rate limiting
Message-ID: <CAFK6my4ius+gWX=iDnZZ-V==ANJDUOJPkBOuPs+jMkt1c8tL5A@mail.gmail.com>

Hello,

I'm using rate limiting within the stream module. While it works great
for long connections it does not work on request smaller than the rate
limite size for 1 second. I set up a 1gbps rate limit (limit_rate
125m) and request smaller than 125M or not limited. This is a normal
behavioir as the rate limiting is done with a second precision. This
patch change second precision to millisecond precision. From my first
tests (still on going) it seems to works better.

What guys do you think about this patch ?

Thanks
++ Jerome


# HG changeset patch
# User Jerome Loyet <jerome at loyet.net>
# Date 1669612614 -3600
#      Mon Nov 28 06:16:54 2022 +0100
# Node ID a474ad7ca3408d2b09cd7bfbc827c8df84e3bd0c
# Parent  0b360747c74e3fa7e439e0684a8cf1da2d14d8f6
switch to ms resolution for rate limiting

rate limiting use a second resolution which can be bypassed on
requests/response smaller than the configured rate for 1 second.

This patch switches to millisecond resolution to ensure
better precision of the rate limiting.

diff -r 0b360747c74e -r a474ad7ca340 src/event/ngx_event_pipe.c
--- a/src/event/ngx_event_pipe.c    Thu Nov 24 23:08:30 2022 +0400
+++ b/src/event/ngx_event_pipe.c    Mon Nov 28 06:16:54 2022 +0100
@@ -203,8 +203,8 @@
                     break;
                 }

-                limit = (off_t) p->limit_rate * (ngx_time() - p->start_sec + 1)
-                        - p->read_length;
+                limit = (off_t) p->limit_rate * (ngx_current_msec -
p->start_msec + 1)
+                        / 1000 - p->read_length;

                 if (limit <= 0) {
                     p->upstream->read->delayed = 1;
diff -r 0b360747c74e -r a474ad7ca340 src/event/ngx_event_pipe.h
--- a/src/event/ngx_event_pipe.h    Thu Nov 24 23:08:30 2022 +0400
+++ b/src/event/ngx_event_pipe.h    Mon Nov 28 06:16:54 2022 +0100
@@ -91,7 +91,7 @@
     ngx_buf_t         *buf_to_file;

     size_t             limit_rate;
-    time_t             start_sec;
+    ngx_msec_t         start_msec;

     ngx_temp_file_t   *temp_file;

diff -r 0b360747c74e -r a474ad7ca340 src/http/ngx_http_upstream.c
--- a/src/http/ngx_http_upstream.c    Thu Nov 24 23:08:30 2022 +0400
+++ b/src/http/ngx_http_upstream.c    Mon Nov 28 06:16:54 2022 +0100
@@ -3217,7 +3217,7 @@
     p->pool = r->pool;
     p->log = c->log;
     p->limit_rate = u->conf->limit_rate;
-    p->start_sec = ngx_time();
+    p->start_msec = ngx_current_msec;

     p->cacheable = u->cacheable || u->store;

diff -r 0b360747c74e -r a474ad7ca340 src/stream/ngx_stream_proxy_module.c
--- a/src/stream/ngx_stream_proxy_module.c    Thu Nov 24 23:08:30 2022 +0400
+++ b/src/stream/ngx_stream_proxy_module.c    Mon Nov 28 06:16:54 2022 +0100
@@ -435,7 +435,7 @@
     }

     u->peer.type = c->type;
-    u->start_sec = ngx_time();
+    u->start_msec = ngx_current_msec;

     c->write->handler = ngx_stream_proxy_downstream_handler;
     c->read->handler = ngx_stream_proxy_downstream_handler;
@@ -1679,8 +1679,8 @@
             && !src->read->error)
         {
             if (limit_rate) {
-                limit = (off_t) limit_rate * (ngx_time() - u->start_sec + 1)
-                        - *received;
+                limit = (off_t) limit_rate * (ngx_current_msec -
u->start_msec + 1)
+                        / 1000 - *received;

                 if (limit <= 0) {
                     src->read->delayed = 1;
diff -r 0b360747c74e -r a474ad7ca340 src/stream/ngx_stream_upstream.h
--- a/src/stream/ngx_stream_upstream.h    Thu Nov 24 23:08:30 2022 +0400
+++ b/src/stream/ngx_stream_upstream.h    Mon Nov 28 06:16:54 2022 +0100
@@ -127,7 +127,7 @@
     ngx_chain_t                       *downstream_busy;

     off_t                              received;
-    time_t                             start_sec;
+    ngx_msec_t                         start_msec;
     ngx_uint_t                         requests;
     ngx_uint_t                         responses;
     ngx_msec_t                         start_time;


From pluknet at nginx.com  Mon Nov 28 16:11:10 2022
From: pluknet at nginx.com (Sergey Kandaurov)
Date: Mon, 28 Nov 2022 20:11:10 +0400
Subject: [PATCH 04 of 10] QUIC: reusable mode for main connection
In-Reply-To: <0a5de8e68cb8b238a1fb.1662627991@arut-laptop>
References: <patchbomb.1662627987@arut-laptop>
 <0a5de8e68cb8b238a1fb.1662627991@arut-laptop>
Message-ID: <20221128161110.ksxdmkyynd23imu2@Y9MQ9X2QVV>

On Thu, Sep 08, 2022 at 01:06:31PM +0400, Roman Arutyunyan wrote:
> # HG changeset patch
> # User Roman Arutyunyan <arut at nginx.com>
> # Date 1662478181 -14400
> #      Tue Sep 06 19:29:41 2022 +0400
> # Branch quic
> # Node ID 0a5de8e68cb8b238a1fb82da93ce583c0fa1a6a1
> # Parent  b4662fc66f1e8388f54d988777b741964892cec2
> QUIC: reusable mode for main connection.
> 
> The connection is automatically switched to this mode by transport layer when
> there are no non-cancelable streams.  Currently, cancelable streams are
> HTTP/3 encoder/decoder/control streams.
> 
> diff --git a/src/event/quic/ngx_event_quic.c b/src/event/quic/ngx_event_quic.c
> --- a/src/event/quic/ngx_event_quic.c
> +++ b/src/event/quic/ngx_event_quic.c
> @@ -341,6 +341,8 @@ ngx_quic_new_connection(ngx_connection_t
>          return NULL;
>      }
>  
> +    ngx_reusable_connection(c, 1);
> +
>      ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0,
>                     "quic connection created");
>  
> @@ -420,7 +422,7 @@ ngx_quic_input_handler(ngx_event_t *rev)
>      if (c->close) {
>          qc->error = NGX_QUIC_ERR_NO_ERROR;
>          qc->error_reason = "graceful shutdown";
> -        ngx_quic_close_connection(c, NGX_OK);
> +        ngx_quic_close_connection(c, NGX_ERROR);
>          return;
>      }
>  
> @@ -603,12 +605,17 @@ ngx_quic_finalize_connection(ngx_connect
>      ngx_quic_connection_t  *qc;
>  
>      qc = ngx_quic_get_connection(c);
> +
> +    if (qc->closing) {
> +        return;
> +    }
> +
>      qc->error = err;
>      qc->error_reason = reason;
>      qc->error_app = 1;
>      qc->error_ftype = 0;
>  
> -    ngx_quic_close_connection(c, NGX_OK);
> +    ngx_post_event(&qc->close, &ngx_posted_events);
>  }
>  
>  
> @@ -630,20 +637,13 @@ ngx_quic_shutdown_connection(ngx_connect
>  static void
>  ngx_quic_close_handler(ngx_event_t *ev)
>  {
> -    ngx_connection_t       *c;
> -    ngx_quic_connection_t  *qc;
> +    ngx_connection_t  *c;
>  
>      ngx_log_debug0(NGX_LOG_DEBUG_EVENT, ev->log, 0, "quic close handler");
>  
>      c = ev->data;
> -    qc = ngx_quic_get_connection(c);
>  
> -    if (qc->closing) {
> -        ngx_quic_close_connection(c, NGX_OK);
> -
> -    } else if (qc->shutdown) {
> -        ngx_quic_shutdown_quic(c);
> -    }
> +    ngx_quic_close_connection(c, NGX_OK);
>  }
>  
>  
> @@ -1428,31 +1428,10 @@ ngx_quic_push_handler(ngx_event_t *ev)
>  void
>  ngx_quic_shutdown_quic(ngx_connection_t *c)
>  {
> -    ngx_rbtree_t           *tree;
> -    ngx_rbtree_node_t      *node;
> -    ngx_quic_stream_t      *qs;
>      ngx_quic_connection_t  *qc;
>  
> -    qc = ngx_quic_get_connection(c);
> -
> -    if (qc->closing) {
> -        return;
> +    if (c->reusable) {
> +        qc = ngx_quic_get_connection(c);
> +        ngx_quic_finalize_connection(c, qc->shutdown_code, qc->shutdown_reason);
>      }
> -
> -    tree = &qc->streams.tree;
> -
> -    if (tree->root != tree->sentinel) {
> -        for (node = ngx_rbtree_min(tree->root, tree->sentinel);
> -             node;
> -             node = ngx_rbtree_next(tree, node))
> -        {
> -            qs = (ngx_quic_stream_t *) node;
> -
> -            if (!qs->cancelable) {
> -                return;
> -            }
> -        }
> -    }
> -
> -    ngx_quic_finalize_connection(c, qc->shutdown_code, qc->shutdown_reason);
>  }
> diff --git a/src/event/quic/ngx_event_quic.h b/src/event/quic/ngx_event_quic.h
> --- a/src/event/quic/ngx_event_quic.h
> +++ b/src/event/quic/ngx_event_quic.h
> @@ -113,6 +113,7 @@ void ngx_quic_shutdown_connection(ngx_co
>      const char *reason);
>  ngx_int_t ngx_quic_reset_stream(ngx_connection_t *c, ngx_uint_t err);
>  ngx_int_t ngx_quic_shutdown_stream(ngx_connection_t *c, int how);
> +void ngx_quic_cancelable_stream(ngx_connection_t *c);
>  ngx_int_t ngx_quic_handle_read_event(ngx_event_t *rev, ngx_uint_t flags);
>  ngx_int_t ngx_quic_handle_write_event(ngx_event_t *wev, size_t lowat);
>  ngx_int_t ngx_quic_get_packet_dcid(ngx_log_t *log, u_char *data, size_t len,
> diff --git a/src/event/quic/ngx_event_quic_streams.c b/src/event/quic/ngx_event_quic_streams.c
> --- a/src/event/quic/ngx_event_quic_streams.c
> +++ b/src/event/quic/ngx_event_quic_streams.c
> @@ -33,6 +33,7 @@ static ngx_chain_t *ngx_quic_stream_send
>  static ngx_int_t ngx_quic_stream_flush(ngx_quic_stream_t *qs);
>  static void ngx_quic_stream_cleanup_handler(void *data);
>  static ngx_int_t ngx_quic_close_stream(ngx_quic_stream_t *qs);
> +static ngx_int_t ngx_quic_can_shutdown(ngx_connection_t *c);
>  static ngx_int_t ngx_quic_control_flow(ngx_quic_stream_t *qs, uint64_t last);
>  static ngx_int_t ngx_quic_update_flow(ngx_quic_stream_t *qs, uint64_t last);
>  static ngx_int_t ngx_quic_update_max_stream_data(ngx_quic_stream_t *qs);
> @@ -51,6 +52,10 @@ ngx_quic_open_stream(ngx_connection_t *c
>      pc = c->quic ? c->quic->parent : c;
>      qc = ngx_quic_get_connection(pc);
>  
> +    if (qc->shutdown || qc->closing) {

I think the shutdown part needs to be reverted as otherwise
it would prevent creating on-demand a unidirectional stream
used to send Stream Cancelation decoder instruction.
In particular, this is possible if client sends additional
request streams after we sent GOAWAY and the decoder stream
hasn't yet been used (i.e. there were no insertions).

> +        return NULL;
> +    }
> +
>      if (bidi) {
>          if (qc->streams.server_streams_bidi
>              >= qc->streams.server_max_streams_bidi)
> @@ -161,13 +166,10 @@ ngx_quic_close_streams(ngx_connection_t 
>      ngx_pool_t         *pool;
>      ngx_queue_t        *q;
>      ngx_rbtree_t       *tree;
> +    ngx_connection_t   *sc;
>      ngx_rbtree_node_t  *node;
>      ngx_quic_stream_t  *qs;
>  
> -#if (NGX_DEBUG)
> -    ngx_uint_t          ns;
> -#endif
> -
>      while (!ngx_queue_empty(&qc->streams.uninitialized)) {
>          q = ngx_queue_head(&qc->streams.uninitialized);
>          ngx_queue_remove(q);
> @@ -185,34 +187,34 @@ ngx_quic_close_streams(ngx_connection_t 
>          return NGX_OK;
>      }
>  
> -#if (NGX_DEBUG)
> -    ns = 0;
> -#endif
> -
>      node = ngx_rbtree_min(tree->root, tree->sentinel);
>  
>      while (node) {
>          qs = (ngx_quic_stream_t *) node;
>          node = ngx_rbtree_next(tree, node);
> +        sc = qs->connection;
>  
>          qs->recv_state = NGX_QUIC_STREAM_RECV_RESET_RECVD;
>          qs->send_state = NGX_QUIC_STREAM_SEND_RESET_SENT;
>  
> -        if (qs->connection == NULL) {
> +        if (sc == NULL) {
>              ngx_quic_close_stream(qs);
>              continue;
>          }
>  
> -        ngx_quic_set_event(qs->connection->read);
> -        ngx_quic_set_event(qs->connection->write);
> +        ngx_quic_set_event(sc->read);
> +        ngx_quic_set_event(sc->write);
>  
> -#if (NGX_DEBUG)
> -        ns++;
> -#endif
> +        sc->close = 1;
> +        sc->read->handler(sc->read);
>      }
>  
> -    ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0,
> -                   "quic connection has %ui active streams", ns);
> +    if (tree->root == tree->sentinel) {
> +        return NGX_OK;
> +    }
> +
> +    ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0,
> +                   "quic connection has active streams");
>  
>      return NGX_AGAIN;
>  }
> @@ -587,6 +589,7 @@ ngx_quic_create_stream(ngx_connection_t 
>  {
>      ngx_log_t              *log;
>      ngx_pool_t             *pool;
> +    ngx_uint_t              reusable;
>      ngx_queue_t            *q;
>      ngx_connection_t       *sc;
>      ngx_quic_stream_t      *qs;
> @@ -639,10 +642,14 @@ ngx_quic_create_stream(ngx_connection_t 
>      *log = *c->log;
>      pool->log = log;
>  
> +    reusable = c->reusable;
> +    ngx_reusable_connection(c, 0);
> +
>      sc = ngx_get_connection(c->fd, log);
>      if (sc == NULL) {
>          ngx_destroy_pool(pool);
>          ngx_queue_insert_tail(&qc->streams.free, &qs->queue);
> +        ngx_reusable_connection(c, reusable);
>          return NULL;
>      }
>  
> @@ -712,6 +719,7 @@ ngx_quic_create_stream(ngx_connection_t 
>          ngx_close_connection(sc);
>          ngx_destroy_pool(pool);
>          ngx_queue_insert_tail(&qc->streams.free, &qs->queue);
> +        ngx_reusable_connection(c, reusable);
>          return NULL;
>      }
>  
> @@ -724,6 +732,31 @@ ngx_quic_create_stream(ngx_connection_t 
>  }
>  
>  
> +void
> +ngx_quic_cancelable_stream(ngx_connection_t *c)
> +{
> +    ngx_connection_t       *pc;
> +    ngx_quic_stream_t      *qs;
> +    ngx_quic_connection_t  *qc;
> +
> +    qs = c->quic;
> +    pc = qs->parent;
> +    qc = ngx_quic_get_connection(pc);
> +
> +    if (!qs->cancelable) {
> +        qs->cancelable = 1;
> +
> +        if (ngx_quic_can_shutdown(pc) == NGX_OK) {
> +            ngx_reusable_connection(pc, 1);
> +
> +            if (qc->shutdown) {
> +                ngx_quic_shutdown_quic(pc);
> +            }
> +        }
> +    }
> +}
> +
> +
>  static void
>  ngx_quic_empty_handler(ngx_event_t *ev)
>  {
> @@ -1056,14 +1089,47 @@ ngx_quic_close_stream(ngx_quic_stream_t 
>          ngx_quic_queue_frame(qc, frame);
>      }
>  
> +    if (!pc->reusable && ngx_quic_can_shutdown(pc) == NGX_OK) {
> +        ngx_reusable_connection(pc, 1);
> +    }
> +
>      if (qc->shutdown) {
> -        ngx_post_event(&qc->close, &ngx_posted_events);
> +        ngx_quic_shutdown_quic(pc);
>      }
>  
>      return NGX_OK;
>  }
>  
>  
> +static ngx_int_t
> +ngx_quic_can_shutdown(ngx_connection_t *c)
> +{
> +    ngx_rbtree_t           *tree;
> +    ngx_rbtree_node_t      *node;
> +    ngx_quic_stream_t      *qs;
> +    ngx_quic_connection_t  *qc;
> +
> +    qc = ngx_quic_get_connection(c);
> +
> +    tree = &qc->streams.tree;
> +
> +    if (tree->root != tree->sentinel) {
> +        for (node = ngx_rbtree_min(tree->root, tree->sentinel);
> +             node;
> +             node = ngx_rbtree_next(tree, node))
> +        {
> +            qs = (ngx_quic_stream_t *) node;
> +
> +            if (!qs->cancelable) {
> +                return NGX_DECLINED;
> +            }
> +        }
> +     }
> +
> +    return NGX_OK;
> +}
> +
> +
>  ngx_int_t
>  ngx_quic_handle_stream_frame(ngx_connection_t *c, ngx_quic_header_t *pkt,
>      ngx_quic_frame_t *frame)
> diff --git a/src/http/v3/ngx_http_v3_uni.c b/src/http/v3/ngx_http_v3_uni.c
> --- a/src/http/v3/ngx_http_v3_uni.c
> +++ b/src/http/v3/ngx_http_v3_uni.c
> @@ -52,7 +52,7 @@ ngx_http_v3_init_uni_stream(ngx_connecti
>          return;
>      }
>  
> -    c->quic->cancelable = 1;
> +    ngx_quic_cancelable_stream(c);
>  
>      us = ngx_pcalloc(c->pool, sizeof(ngx_http_v3_uni_stream_t));
>      if (us == NULL) {
> @@ -182,6 +182,11 @@ ngx_http_v3_uni_read_handler(ngx_event_t
>  
>      ngx_log_debug0(NGX_LOG_DEBUG_HTTP, c->log, 0, "http3 read handler");
>  
> +    if (c->close) {
> +        ngx_http_v3_close_uni_stream(c);
> +        return;
> +    }
> +
>      ngx_memzero(&b, sizeof(ngx_buf_t));
>  
>      while (rev->ready) {
> @@ -262,6 +267,11 @@ ngx_http_v3_uni_dummy_read_handler(ngx_e
>  
>      ngx_log_debug0(NGX_LOG_DEBUG_HTTP, c->log, 0, "http3 dummy read handler");
>  
> +    if (c->close) {
> +        ngx_http_v3_close_uni_stream(c);
> +        return;
> +    }
> +
>      if (rev->ready) {
>          if (c->recv(c, &ch, 1) != 0) {
>              ngx_http_v3_finalize_connection(c, NGX_HTTP_V3_ERR_NO_ERROR, NULL);
> @@ -404,7 +414,7 @@ ngx_http_v3_get_uni_stream(ngx_connectio
>          goto failed;
>      }
>  
> -    sc->quic->cancelable = 1;
> +    ngx_quic_cancelable_stream(sc);
>  
>      ngx_log_debug1(NGX_LOG_DEBUG_HTTP, c->log, 0,
>                     "http3 create uni stream, type:%ui", type);


From pluknet at nginx.com  Mon Nov 28 17:40:05 2022
From: pluknet at nginx.com (Sergey Kandaurov)
Date: Mon, 28 Nov 2022 21:40:05 +0400
Subject: [PATCH 08 of 10] QUIC: idle mode for main connection
In-Reply-To: <20221124151531.s4oldl3jeo4wtobv@N00W24XTQX>
References: <patchbomb.1662627987@arut-laptop>
 <e0634a484d9a2d82d43f.1662627995@arut-laptop>
 <20221020115015.azqjb5wd7fisgm2f@Y9MQ9X2QVV>
 <20221020142503.d327lsrunx6hzw63@N00W24XTQX>
 <20221124151531.s4oldl3jeo4wtobv@N00W24XTQX>
Message-ID: <20221128174005.esncmvmumjstgbz3@Y9MQ9X2QVV>

On Thu, Nov 24, 2022 at 07:15:31PM +0400, Roman Arutyunyan wrote:
> Hi,
> 
> On Thu, Oct 20, 2022 at 06:25:03PM +0400, Roman Arutyunyan wrote:
> > Hi,
> > 
> > On Thu, Oct 20, 2022 at 03:50:15PM +0400, Sergey Kandaurov wrote:
> > > On Thu, Sep 08, 2022 at 01:06:35PM +0400, Roman Arutyunyan wrote:
> > > > # HG changeset patch
> > > > # User Roman Arutyunyan <arut at nginx.com>
> > > > # Date 1662627133 -14400
> > > > #      Thu Sep 08 12:52:13 2022 +0400
> > > > # Branch quic
> > > > # Node ID e0634a484d9a2d82d43f565d64a0a22e989ac1cb
> > > > # Parent  1dd6fabfdcb5b52af495f9d8fc00f64ae36a537c
> > > > QUIC: idle mode for main connection.
> > > > 
> > > > Now main QUIC connection for HTTP/3 always has c->idle flag set.  This allows
> > > > the connection to receive worker shutdown notification.  It is passed to
> > > > application level via a new conf->shutdown() callback.
> > > > 
> > > > The HTTP/3 shutdown callback sends GOAWAY to client and gracefully shuts down
> > > > the QUIC connection.
> > > > 
> > > > diff --git a/src/event/quic/ngx_event_quic.c b/src/event/quic/ngx_event_quic.c
> > > > --- a/src/event/quic/ngx_event_quic.c
> > > > +++ b/src/event/quic/ngx_event_quic.c
> > > > @@ -341,6 +341,7 @@ ngx_quic_new_connection(ngx_connection_t
> > > >          return NULL;
> > > >      }
> > > >  
> > > > +    c->idle = 1;
> > > >      ngx_reusable_connection(c, 1);
> > > >  
> > > >      ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0,
> > > > @@ -420,9 +421,9 @@ ngx_quic_input_handler(ngx_event_t *rev)
> > > >      }
> > > >  
> > > >      if (c->close) {
> > > > -        qc->error = NGX_QUIC_ERR_NO_ERROR;
> > > > -        qc->error_reason = "graceful shutdown";
> > > > -        ngx_quic_close_connection(c, NGX_ERROR);
> > > > +        if (qc->conf->shutdown) {
> > > 
> > > As previously discussed in private, this will need an additional check
> > > that we are not yet in qc->closing.
> > > 
> > > > +            qc->conf->shutdown(c);
> > > > +        }
> > > >          return;
> > > >      }
> > 
> > 
> > Yes, added the check.  Also, c->close is reset here similarly to HTTP/2
> > since we want to be able to handle future packets normally.
> > 
> > Also, current code which closes the connection instantly should remain for
> > connection reuse.  To tell reuse from shutdown we can check ngx_exiting.
> > Assuming reuse does not make sense in shutdown mode, this will work good.
> > 
> > > > diff --git a/src/event/quic/ngx_event_quic.h b/src/event/quic/ngx_event_quic.h
> > > > --- a/src/event/quic/ngx_event_quic.h
> > > > +++ b/src/event/quic/ngx_event_quic.h
> > > > @@ -28,6 +28,9 @@
> > > >  #define NGX_QUIC_STREAM_UNIDIRECTIONAL       0x02
> > > >  
> > > >  
> > > > +typedef void (*ngx_quic_shutdown_pt)(ngx_connection_t *c);
> > > > +
> > > > +
> > > >  typedef enum {
> > > >      NGX_QUIC_STREAM_SEND_READY = 0,
> > > >      NGX_QUIC_STREAM_SEND_SEND,
> > > > @@ -74,6 +77,8 @@ typedef struct {
> > > >      ngx_int_t                      stream_reject_code_uni;
> > > >      ngx_int_t                      stream_reject_code_bidi;
> > > >  
> > > > +    ngx_quic_shutdown_pt           shutdown;
> > > > +
> > > >      u_char                         av_token_key[NGX_QUIC_AV_KEY_LEN];
> > > >      u_char                         sr_token_key[NGX_QUIC_SR_KEY_LEN];
> > > >  } ngx_quic_conf_t;
> > > > diff --git a/src/http/v3/ngx_http_v3.h b/src/http/v3/ngx_http_v3.h
> > > > --- a/src/http/v3/ngx_http_v3.h
> > > > +++ b/src/http/v3/ngx_http_v3.h
> > > > @@ -141,6 +141,7 @@ struct ngx_http_v3_session_s {
> > > >      uint64_t                      next_push_id;
> > > >      uint64_t                      max_push_id;
> > > >      uint64_t                      goaway_push_id;
> > > > +    uint64_t                      next_request_id;
> > > >  
> > > >      off_t                         total_bytes;
> > > >      off_t                         payload_bytes;
> > > > @@ -158,6 +159,7 @@ void ngx_http_v3_init(ngx_connection_t *
> > > >  void ngx_http_v3_reset_connection(ngx_connection_t *c);
> > > >  ngx_int_t ngx_http_v3_init_session(ngx_connection_t *c);
> > > >  ngx_int_t ngx_http_v3_check_flood(ngx_connection_t *c);
> > > > +void ngx_http_v3_shutdown(ngx_connection_t *c);
> > > >  
> > > >  ngx_int_t ngx_http_v3_read_request_body(ngx_http_request_t *r);
> > > >  ngx_int_t ngx_http_v3_read_unbuffered_request_body(ngx_http_request_t *r);
> > > > diff --git a/src/http/v3/ngx_http_v3_module.c b/src/http/v3/ngx_http_v3_module.c
> > > > --- a/src/http/v3/ngx_http_v3_module.c
> > > > +++ b/src/http/v3/ngx_http_v3_module.c
> > > > @@ -249,6 +249,8 @@ ngx_http_v3_create_srv_conf(ngx_conf_t *
> > > >      h3scf->quic.stream_reject_code_bidi = NGX_HTTP_V3_ERR_REQUEST_REJECTED;
> > > >      h3scf->quic.active_connection_id_limit = NGX_CONF_UNSET_UINT;
> > > >  
> > > > +    h3scf->quic.shutdown = ngx_http_v3_shutdown;
> > > > +
> > > >      return h3scf;
> > > >  }
> > > >  
> > > > diff --git a/src/http/v3/ngx_http_v3_request.c b/src/http/v3/ngx_http_v3_request.c
> > > > --- a/src/http/v3/ngx_http_v3_request.c
> > > > +++ b/src/http/v3/ngx_http_v3_request.c
> > > > @@ -97,6 +97,37 @@ ngx_http_v3_init(ngx_connection_t *c)
> > > >  }
> > > >  
> > > >  
> > > > +void
> > > > +ngx_http_v3_shutdown(ngx_connection_t *c)
> > > > +{
> > > > +    ngx_http_v3_session_t   *h3c;
> > > 
> > > extra indent
> > > 
> > > > +
> > > > +    ngx_log_debug0(NGX_LOG_DEBUG_HTTP, c->log, 0, "http3 shutdown");
> > > > +
> > > > +    h3c = ngx_http_v3_get_session(c);
> > > > +
> > > > +    if (h3c == NULL) {
> > > > +        ngx_quic_finalize_connection(c, NGX_HTTP_V3_ERR_NO_ERROR,
> > > > +                                     "connection shutdown");
> > > > +        return;
> > > > +    }
> > > > +
> > > > +    if (!h3c->goaway) {
> > > > +        h3c->goaway = 1;
> > > > +
> > > > +#if (NGX_HTTP_V3_HQ)
> > > > +        if (!h3c->hq)
> > > > +#endif
> > > > +        {
> > > > +            (void) ngx_http_v3_send_goaway(c, h3c->next_request_id);
> > > > +        }
> > > > +
> > > > +        ngx_http_v3_shutdown_connection(c, NGX_HTTP_V3_ERR_NO_ERROR,
> > > > +                                        "connection shutdown");
> > > > +    }
> > > 
> > > Note that this callback is used to be called from a read event as part of
> > > graceful shutdown.
> > > With ngx_quic_finalize_connection() remade in patch #4 (reusable mode)
> > > to defer closing QUIC connection to a posted event, this call now results
> > > in a posted event, which no one can fulfill, hence no further action until
> > > quic idle timeout fires.
> > > It could be fixed by executing known posted events after shutdown callback
> > > or more globally - as part of graceful shutdown itself.
> > 
> > Yes, events posted from ngx_close_idle_connections() are not handled right away.
> > Instead, they are handled at the end of the next cycle, which normally
> > happens after a timeout.  There seems to be no pretty way to fix this, unless
> > we handle posted events in ngx_worker_process_cycle() right after
> > ngx_close_idle_connections().  We are trying to avoid global changes like this.
> > 
> > I suggest posting current connection read event as a next posted event.  This
> > will effectively set next cycle timeout to be zero and eliminate the problem.
> > 
> > > > +}
> > > > +
> > > > +
> > > >  static void
> > > >  ngx_http_v3_init_request_stream(ngx_connection_t *c)
> > > >  {
> > > > @@ -137,6 +168,8 @@ ngx_http_v3_init_request_stream(ngx_conn
> > > >  
> > > >      pc = c->quic->parent;
> > > >  
> > > > +    h3c->next_request_id = c->quic->id + 0x04;
> > > > +
> > > >      if (n + 1 == clcf->keepalive_requests
> > > >          || ngx_current_msec - pc->start_time > clcf->keepalive_time)
> > > >      {
> > > > @@ -146,7 +179,7 @@ ngx_http_v3_init_request_stream(ngx_conn
> > > >          if (!h3c->hq)
> > > >  #endif
> > > >          {
> > > > -            if (ngx_http_v3_send_goaway(c, (n + 1) << 2) != NGX_OK) {
> > > > +            if (ngx_http_v3_send_goaway(c, h3c->next_request_id) != NGX_OK) {
> > > >                  ngx_http_close_connection(c);
> > > >                  return;
> > > >              }
> > > > 
> > > > _______________________________________________
> > > > nginx-devel mailing list -- nginx-devel at nginx.org
> > > > To unsubscribe send an email to nginx-devel-leave at nginx.org
> > > _______________________________________________
> > > nginx-devel mailing list -- nginx-devel at nginx.org
> > > To unsubscribe send an email to nginx-devel-leave at nginx.org
> > 
> > Attached is a diff to the current patch.
> > 
> > --
> > Roman
> 
> > # HG changeset patch
> > # User Roman Arutyunyan <arut at nginx.com>
> > # Date 1666273166 -14400
> > #      Thu Oct 20 17:39:26 2022 +0400
> > # Branch quic
> > # Node ID d6c725081a0b024886822e1cc722fdace9c32621
> > # Parent  a4ba2ac5fa55ef94bb75a66e66e0b19d792fed10
> > [mq]: quic-idle-fix1
> > 
> > diff --git a/src/event/quic/ngx_event_quic.c b/src/event/quic/ngx_event_quic.c
> > --- a/src/event/quic/ngx_event_quic.c
> > +++ b/src/event/quic/ngx_event_quic.c
> > @@ -421,9 +421,22 @@ ngx_quic_input_handler(ngx_event_t *rev)
> >      }
> >  
> >      if (c->close) {
> > -        if (qc->conf->shutdown) {
> > +        c->close = 0;
> > +
> > +        if (!ngx_exiting) {
> > +            qc->error = NGX_QUIC_ERR_NO_ERROR;
> > +            qc->error_reason = "graceful shutdown";
> > +            ngx_quic_close_connection(c, NGX_ERROR);
> > +            return;
> > +        }
> > +
> > +        if (!qc->closing && qc->conf->shutdown) {
> > +            /* do not delay events posted by shutdown() */
> > +            ngx_post_event(rev, &ngx_posted_next_events);
> 
> Following the change http://hg.nginx.org/nginx/rev/b809f53d3f5b, this part
> is no longer needed.  The new diff attached.

Note that adding support for "idle mode for main connection" combined
with b809f53d3f5b uncovers a socket leak due to qc->close.cancelable set.

In more details:
- there can be connections with stretched pto e.g. due to missing client ack,
  large enough such that a close timer (3*pto) fires after the idle timer
- now that QUIC connection is handled on graceful shutdown as idle,
  entering graceful shutdown results in setting a close timer, and closing
  can take additional cycle following the "qc->close.timer_set" condition
- pto and idle timers fired (and removed)
- only close and, possibly, keepalive timers left; both cancelable

I suggest removing the cancelable property of a close timer
given that we want to wait for 3*PTO while in a closing state
and send back sensible CONNECTION_CLOSE to any further input
(otherwise it would emit less sensible stateless reset).
This will fix leaving event cycle without non-cancelable timers set.

> 
> > +
> >              qc->conf->shutdown(c);
> >          }
> > +
> >          return;
> >      }
> 
> 
> --
> Roman Arutyunyan

> # HG changeset patch
> # User Roman Arutyunyan <arut at nginx.com>
> # Date 1669302768 -14400
> #      Thu Nov 24 19:12:48 2022 +0400
> # Branch quic
> # Node ID ec21165abb4333f77fb85e956cccf300c8ae1acf
> # Parent  6fba6061b65f0ee41c231bde6b738cfae00bb179
> [mq]: quic-idle-fix1
> 
> diff --git a/src/event/quic/ngx_event_quic.c b/src/event/quic/ngx_event_quic.c
> --- a/src/event/quic/ngx_event_quic.c
> +++ b/src/event/quic/ngx_event_quic.c
> @@ -421,9 +421,19 @@ ngx_quic_input_handler(ngx_event_t *rev)
>      }
>  
>      if (c->close) {
> -        if (qc->conf->shutdown) {
> +        c->close = 0;
> +
> +        if (!ngx_exiting) {
> +            qc->error = NGX_QUIC_ERR_NO_ERROR;
> +            qc->error_reason = "graceful shutdown";
> +            ngx_quic_close_connection(c, NGX_ERROR);
> +            return;
> +        }
> +
> +        if (!qc->closing && qc->conf->shutdown) {
>              qc->conf->shutdown(c);
>          }
> +
>          return;
>      }
>  
> diff --git a/src/http/v3/ngx_http_v3_request.c b/src/http/v3/ngx_http_v3_request.c
> --- a/src/http/v3/ngx_http_v3_request.c
> +++ b/src/http/v3/ngx_http_v3_request.c
> @@ -101,7 +101,7 @@ ngx_http_v3_init(ngx_connection_t *c)
>  void
>  ngx_http_v3_shutdown(ngx_connection_t *c)
>  {
> -    ngx_http_v3_session_t   *h3c;
> +    ngx_http_v3_session_t  *h3c;
>  
>      ngx_log_debug0(NGX_LOG_DEBUG_HTTP, c->log, 0, "http3 shutdown");
>  

Looks good.


From mdounin at mdounin.ru  Mon Nov 28 22:12:16 2022
From: mdounin at mdounin.ru (=?utf-8?q?Maxim_Dounin?=)
Date: Tue, 29 Nov 2022 01:12:16 +0300
Subject: [PATCH 2 of 2] Win32: event flags handling edge cases in
 ngx_wsarecv()
In-Reply-To: <patchbomb.1669673534@vm-bsd.mdounin.ru>
References: <patchbomb.1669673534@vm-bsd.mdounin.ru>
Message-ID: <447cea17128f6a4ee539.1669673536@vm-bsd.mdounin.ru>

# HG changeset patch
# User Maxim Dounin <mdounin at mdounin.ru>
# Date 1669672416 -10800
#      Tue Nov 29 00:53:36 2022 +0300
# Node ID 447cea17128f6a4ee5399ed56e53f8d9c163420b
# Parent  97a5a082c58ac91f278c7faf2286e0bc04b3c958
Win32: event flags handling edge cases in ngx_wsarecv().

Fixed event flags handling edge cases in ngx_wsarecv() and ngx_wsarecv_chain(),
notably to always reset rev->ready in case of errors (which wasn't the case
after ngx_socket_nread() errors), and after EOF (rev->ready was not cleared
if due to a misconfiguration a zero-sized buffer was used for reading).

diff --git a/src/os/win32/ngx_wsarecv.c b/src/os/win32/ngx_wsarecv.c
--- a/src/os/win32/ngx_wsarecv.c
+++ b/src/os/win32/ngx_wsarecv.c
@@ -78,6 +78,7 @@ ngx_wsarecv(ngx_connection_t *c, u_char 
                                      ngx_socket_nread_n " failed");
 
             if (n == NGX_ERROR) {
+                rev->ready = 0;
                 rev->error = 1;
             }
 
@@ -95,6 +96,7 @@ ngx_wsarecv(ngx_connection_t *c, u_char 
     }
 
     if (bytes == 0) {
+        rev->ready = 0;
         rev->eof = 1;
     }
 
diff --git a/src/os/win32/ngx_wsarecv_chain.c b/src/os/win32/ngx_wsarecv_chain.c
--- a/src/os/win32/ngx_wsarecv_chain.c
+++ b/src/os/win32/ngx_wsarecv_chain.c
@@ -121,6 +121,7 @@ ngx_wsarecv_chain(ngx_connection_t *c, n
     } else if (bytes == size) {
 
         if (ngx_socket_nread(c->fd, &rev->available) == -1) {
+            rev->ready = 0;
             rev->error = 1;
             ngx_connection_error(c, ngx_socket_errno,
                                  ngx_socket_nread_n " failed");
@@ -138,6 +139,7 @@ ngx_wsarecv_chain(ngx_connection_t *c, n
     }
 
     if (bytes == 0) {
+        rev->ready = 0;
         rev->eof = 1;
     }
 


From mdounin at mdounin.ru  Mon Nov 28 22:12:14 2022
From: mdounin at mdounin.ru (=?utf-8?q?Maxim_Dounin?=)
Date: Tue, 29 Nov 2022 01:12:14 +0300
Subject: [PATCH 0 of 2] unbuffered proxying CPU hog (ticket #2418)
Message-ID: <patchbomb.1669673534@vm-bsd.mdounin.ru>

Hello!

The following patch fixes CPU hog observed with unbuffered SSL proxying 
after SSL errors (ticket #2418).  Fix is to always clear c->read->ready
flag when returning errors from ngx_ssl_recv().

An additional patch cleans up some win32-specific edge cases (not expected
to appear in practice though) when c->read->ready is not cleared when
errors or EOFs are returned from ngx_wsarecv() and ngx_wsarecv_chain().

-- 
Maxim Dounin


From mdounin at mdounin.ru  Mon Nov 28 22:12:15 2022
From: mdounin at mdounin.ru (=?utf-8?q?Maxim_Dounin?=)
Date: Tue, 29 Nov 2022 01:12:15 +0300
Subject: [PATCH 1 of 2] SSL: fixed ngx_ssl_recv() to reset c->read->ready
 after errors
In-Reply-To: <patchbomb.1669673534@vm-bsd.mdounin.ru>
References: <patchbomb.1669673534@vm-bsd.mdounin.ru>
Message-ID: <97a5a082c58ac91f278c.1669673535@vm-bsd.mdounin.ru>

# HG changeset patch
# User Maxim Dounin <mdounin at mdounin.ru>
# Date 1669622597 -10800
#      Mon Nov 28 11:03:17 2022 +0300
# Node ID 97a5a082c58ac91f278c7faf2286e0bc04b3c958
# Parent  0b360747c74e3fa7e439e0684a8cf1da2d14d8f6
SSL: fixed ngx_ssl_recv() to reset c->read->ready after errors.

With this change, behaviour of ngx_ssl_recv() now matches ngx_unix_recv(),
which used to always reset c->read->ready to 0 when returning errors.

This fixes an infinite loop in unbuffered SSL proxying if writing to the
client is blocked and an SSL error happens (ticket #2418).

With this change, the fix for a similar issue in the stream module
(6868:ee3645078759), which used a different approach of explicitly
testing c->read->error instead, is no longer needed and was reverted.

diff --git a/src/event/ngx_event_openssl.c b/src/event/ngx_event_openssl.c
--- a/src/event/ngx_event_openssl.c
+++ b/src/event/ngx_event_openssl.c
@@ -2204,6 +2204,7 @@ ngx_ssl_recv(ngx_connection_t *c, u_char
 #endif
 
     if (c->ssl->last == NGX_ERROR) {
+        c->read->ready = 0;
         c->read->error = 1;
         return NGX_ERROR;
     }
@@ -2270,6 +2271,7 @@ ngx_ssl_recv(ngx_connection_t *c, u_char
 #if (NGX_HAVE_FIONREAD)
 
                     if (ngx_socket_nread(c->fd, &c->read->available) == -1) {
+                        c->read->ready = 0;
                         c->read->error = 1;
                         ngx_connection_error(c, ngx_socket_errno,
                                              ngx_socket_nread_n " failed");
@@ -2306,6 +2308,7 @@ ngx_ssl_recv(ngx_connection_t *c, u_char
             return 0;
 
         case NGX_ERROR:
+            c->read->ready = 0;
             c->read->error = 1;
 
             /* fall through */
@@ -2326,6 +2329,7 @@ ngx_ssl_recv_early(ngx_connection_t *c, 
     size_t     readbytes;
 
     if (c->ssl->last == NGX_ERROR) {
+        c->read->ready = 0;
         c->read->error = 1;
         return NGX_ERROR;
     }
@@ -2425,6 +2429,7 @@ ngx_ssl_recv_early(ngx_connection_t *c, 
             return 0;
 
         case NGX_ERROR:
+            c->read->ready = 0;
             c->read->error = 1;
 
             /* fall through */
diff --git a/src/stream/ngx_stream_proxy_module.c b/src/stream/ngx_stream_proxy_module.c
--- a/src/stream/ngx_stream_proxy_module.c
+++ b/src/stream/ngx_stream_proxy_module.c
@@ -1675,9 +1675,8 @@ ngx_stream_proxy_process(ngx_stream_sess
 
         size = b->end - b->last;
 
-        if (size && src->read->ready && !src->read->delayed
-            && !src->read->error)
-        {
+        if (size && src->read->ready && !src->read->delayed) {
+
             if (limit_rate) {
                 limit = (off_t) limit_rate * (ngx_time() - u->start_sec + 1)
                         - *received;


From arut at nginx.com  Tue Nov 29 14:00:40 2022
From: arut at nginx.com (Roman Arutyunyan)
Date: Tue, 29 Nov 2022 18:00:40 +0400
Subject: [PATCH 08 of 10] QUIC: idle mode for main connection
In-Reply-To: <20221128174005.esncmvmumjstgbz3@Y9MQ9X2QVV>
References: <patchbomb.1662627987@arut-laptop>
 <e0634a484d9a2d82d43f.1662627995@arut-laptop>
 <20221020115015.azqjb5wd7fisgm2f@Y9MQ9X2QVV>
 <20221020142503.d327lsrunx6hzw63@N00W24XTQX>
 <20221124151531.s4oldl3jeo4wtobv@N00W24XTQX>
 <20221128174005.esncmvmumjstgbz3@Y9MQ9X2QVV>
Message-ID: <20221129140040.t7rhdnhoipzslmoe@N00W24XTQX>

Hi,

On Mon, Nov 28, 2022 at 09:40:05PM +0400, Sergey Kandaurov wrote:
> On Thu, Nov 24, 2022 at 07:15:31PM +0400, Roman Arutyunyan wrote:
> > Hi,
> > 
> > On Thu, Oct 20, 2022 at 06:25:03PM +0400, Roman Arutyunyan wrote:
> > > Hi,
> > > 
> > > On Thu, Oct 20, 2022 at 03:50:15PM +0400, Sergey Kandaurov wrote:
> > > > On Thu, Sep 08, 2022 at 01:06:35PM +0400, Roman Arutyunyan wrote:
> > > > > # HG changeset patch
> > > > > # User Roman Arutyunyan <arut at nginx.com>
> > > > > # Date 1662627133 -14400
> > > > > #      Thu Sep 08 12:52:13 2022 +0400
> > > > > # Branch quic
> > > > > # Node ID e0634a484d9a2d82d43f565d64a0a22e989ac1cb
> > > > > # Parent  1dd6fabfdcb5b52af495f9d8fc00f64ae36a537c
> > > > > QUIC: idle mode for main connection.
> > > > > 
> > > > > Now main QUIC connection for HTTP/3 always has c->idle flag set.  This allows
> > > > > the connection to receive worker shutdown notification.  It is passed to
> > > > > application level via a new conf->shutdown() callback.
> > > > > 
> > > > > The HTTP/3 shutdown callback sends GOAWAY to client and gracefully shuts down
> > > > > the QUIC connection.
> > > > > 
> > > > > diff --git a/src/event/quic/ngx_event_quic.c b/src/event/quic/ngx_event_quic.c
> > > > > --- a/src/event/quic/ngx_event_quic.c
> > > > > +++ b/src/event/quic/ngx_event_quic.c
> > > > > @@ -341,6 +341,7 @@ ngx_quic_new_connection(ngx_connection_t
> > > > >          return NULL;
> > > > >      }
> > > > >  
> > > > > +    c->idle = 1;
> > > > >      ngx_reusable_connection(c, 1);
> > > > >  
> > > > >      ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0,
> > > > > @@ -420,9 +421,9 @@ ngx_quic_input_handler(ngx_event_t *rev)
> > > > >      }
> > > > >  
> > > > >      if (c->close) {
> > > > > -        qc->error = NGX_QUIC_ERR_NO_ERROR;
> > > > > -        qc->error_reason = "graceful shutdown";
> > > > > -        ngx_quic_close_connection(c, NGX_ERROR);
> > > > > +        if (qc->conf->shutdown) {
> > > > 
> > > > As previously discussed in private, this will need an additional check
> > > > that we are not yet in qc->closing.
> > > > 
> > > > > +            qc->conf->shutdown(c);
> > > > > +        }
> > > > >          return;
> > > > >      }
> > > 
> > > 
> > > Yes, added the check.  Also, c->close is reset here similarly to HTTP/2
> > > since we want to be able to handle future packets normally.
> > > 
> > > Also, current code which closes the connection instantly should remain for
> > > connection reuse.  To tell reuse from shutdown we can check ngx_exiting.
> > > Assuming reuse does not make sense in shutdown mode, this will work good.
> > > 
> > > > > diff --git a/src/event/quic/ngx_event_quic.h b/src/event/quic/ngx_event_quic.h
> > > > > --- a/src/event/quic/ngx_event_quic.h
> > > > > +++ b/src/event/quic/ngx_event_quic.h
> > > > > @@ -28,6 +28,9 @@
> > > > >  #define NGX_QUIC_STREAM_UNIDIRECTIONAL       0x02
> > > > >  
> > > > >  
> > > > > +typedef void (*ngx_quic_shutdown_pt)(ngx_connection_t *c);
> > > > > +
> > > > > +
> > > > >  typedef enum {
> > > > >      NGX_QUIC_STREAM_SEND_READY = 0,
> > > > >      NGX_QUIC_STREAM_SEND_SEND,
> > > > > @@ -74,6 +77,8 @@ typedef struct {
> > > > >      ngx_int_t                      stream_reject_code_uni;
> > > > >      ngx_int_t                      stream_reject_code_bidi;
> > > > >  
> > > > > +    ngx_quic_shutdown_pt           shutdown;
> > > > > +
> > > > >      u_char                         av_token_key[NGX_QUIC_AV_KEY_LEN];
> > > > >      u_char                         sr_token_key[NGX_QUIC_SR_KEY_LEN];
> > > > >  } ngx_quic_conf_t;
> > > > > diff --git a/src/http/v3/ngx_http_v3.h b/src/http/v3/ngx_http_v3.h
> > > > > --- a/src/http/v3/ngx_http_v3.h
> > > > > +++ b/src/http/v3/ngx_http_v3.h
> > > > > @@ -141,6 +141,7 @@ struct ngx_http_v3_session_s {
> > > > >      uint64_t                      next_push_id;
> > > > >      uint64_t                      max_push_id;
> > > > >      uint64_t                      goaway_push_id;
> > > > > +    uint64_t                      next_request_id;
> > > > >  
> > > > >      off_t                         total_bytes;
> > > > >      off_t                         payload_bytes;
> > > > > @@ -158,6 +159,7 @@ void ngx_http_v3_init(ngx_connection_t *
> > > > >  void ngx_http_v3_reset_connection(ngx_connection_t *c);
> > > > >  ngx_int_t ngx_http_v3_init_session(ngx_connection_t *c);
> > > > >  ngx_int_t ngx_http_v3_check_flood(ngx_connection_t *c);
> > > > > +void ngx_http_v3_shutdown(ngx_connection_t *c);
> > > > >  
> > > > >  ngx_int_t ngx_http_v3_read_request_body(ngx_http_request_t *r);
> > > > >  ngx_int_t ngx_http_v3_read_unbuffered_request_body(ngx_http_request_t *r);
> > > > > diff --git a/src/http/v3/ngx_http_v3_module.c b/src/http/v3/ngx_http_v3_module.c
> > > > > --- a/src/http/v3/ngx_http_v3_module.c
> > > > > +++ b/src/http/v3/ngx_http_v3_module.c
> > > > > @@ -249,6 +249,8 @@ ngx_http_v3_create_srv_conf(ngx_conf_t *
> > > > >      h3scf->quic.stream_reject_code_bidi = NGX_HTTP_V3_ERR_REQUEST_REJECTED;
> > > > >      h3scf->quic.active_connection_id_limit = NGX_CONF_UNSET_UINT;
> > > > >  
> > > > > +    h3scf->quic.shutdown = ngx_http_v3_shutdown;
> > > > > +
> > > > >      return h3scf;
> > > > >  }
> > > > >  
> > > > > diff --git a/src/http/v3/ngx_http_v3_request.c b/src/http/v3/ngx_http_v3_request.c
> > > > > --- a/src/http/v3/ngx_http_v3_request.c
> > > > > +++ b/src/http/v3/ngx_http_v3_request.c
> > > > > @@ -97,6 +97,37 @@ ngx_http_v3_init(ngx_connection_t *c)
> > > > >  }
> > > > >  
> > > > >  
> > > > > +void
> > > > > +ngx_http_v3_shutdown(ngx_connection_t *c)
> > > > > +{
> > > > > +    ngx_http_v3_session_t   *h3c;
> > > > 
> > > > extra indent
> > > > 
> > > > > +
> > > > > +    ngx_log_debug0(NGX_LOG_DEBUG_HTTP, c->log, 0, "http3 shutdown");
> > > > > +
> > > > > +    h3c = ngx_http_v3_get_session(c);
> > > > > +
> > > > > +    if (h3c == NULL) {
> > > > > +        ngx_quic_finalize_connection(c, NGX_HTTP_V3_ERR_NO_ERROR,
> > > > > +                                     "connection shutdown");
> > > > > +        return;
> > > > > +    }
> > > > > +
> > > > > +    if (!h3c->goaway) {
> > > > > +        h3c->goaway = 1;
> > > > > +
> > > > > +#if (NGX_HTTP_V3_HQ)
> > > > > +        if (!h3c->hq)
> > > > > +#endif
> > > > > +        {
> > > > > +            (void) ngx_http_v3_send_goaway(c, h3c->next_request_id);
> > > > > +        }
> > > > > +
> > > > > +        ngx_http_v3_shutdown_connection(c, NGX_HTTP_V3_ERR_NO_ERROR,
> > > > > +                                        "connection shutdown");
> > > > > +    }
> > > > 
> > > > Note that this callback is used to be called from a read event as part of
> > > > graceful shutdown.
> > > > With ngx_quic_finalize_connection() remade in patch #4 (reusable mode)
> > > > to defer closing QUIC connection to a posted event, this call now results
> > > > in a posted event, which no one can fulfill, hence no further action until
> > > > quic idle timeout fires.
> > > > It could be fixed by executing known posted events after shutdown callback
> > > > or more globally - as part of graceful shutdown itself.
> > > 
> > > Yes, events posted from ngx_close_idle_connections() are not handled right away.
> > > Instead, they are handled at the end of the next cycle, which normally
> > > happens after a timeout.  There seems to be no pretty way to fix this, unless
> > > we handle posted events in ngx_worker_process_cycle() right after
> > > ngx_close_idle_connections().  We are trying to avoid global changes like this.
> > > 
> > > I suggest posting current connection read event as a next posted event.  This
> > > will effectively set next cycle timeout to be zero and eliminate the problem.
> > > 
> > > > > +}
> > > > > +
> > > > > +
> > > > >  static void
> > > > >  ngx_http_v3_init_request_stream(ngx_connection_t *c)
> > > > >  {
> > > > > @@ -137,6 +168,8 @@ ngx_http_v3_init_request_stream(ngx_conn
> > > > >  
> > > > >      pc = c->quic->parent;
> > > > >  
> > > > > +    h3c->next_request_id = c->quic->id + 0x04;
> > > > > +
> > > > >      if (n + 1 == clcf->keepalive_requests
> > > > >          || ngx_current_msec - pc->start_time > clcf->keepalive_time)
> > > > >      {
> > > > > @@ -146,7 +179,7 @@ ngx_http_v3_init_request_stream(ngx_conn
> > > > >          if (!h3c->hq)
> > > > >  #endif
> > > > >          {
> > > > > -            if (ngx_http_v3_send_goaway(c, (n + 1) << 2) != NGX_OK) {
> > > > > +            if (ngx_http_v3_send_goaway(c, h3c->next_request_id) != NGX_OK) {
> > > > >                  ngx_http_close_connection(c);
> > > > >                  return;
> > > > >              }
> > > > > 
> > > > > _______________________________________________
> > > > > nginx-devel mailing list -- nginx-devel at nginx.org
> > > > > To unsubscribe send an email to nginx-devel-leave at nginx.org
> > > > _______________________________________________
> > > > nginx-devel mailing list -- nginx-devel at nginx.org
> > > > To unsubscribe send an email to nginx-devel-leave at nginx.org
> > > 
> > > Attached is a diff to the current patch.
> > > 
> > > --
> > > Roman
> > 
> > > # HG changeset patch
> > > # User Roman Arutyunyan <arut at nginx.com>
> > > # Date 1666273166 -14400
> > > #      Thu Oct 20 17:39:26 2022 +0400
> > > # Branch quic
> > > # Node ID d6c725081a0b024886822e1cc722fdace9c32621
> > > # Parent  a4ba2ac5fa55ef94bb75a66e66e0b19d792fed10
> > > [mq]: quic-idle-fix1
> > > 
> > > diff --git a/src/event/quic/ngx_event_quic.c b/src/event/quic/ngx_event_quic.c
> > > --- a/src/event/quic/ngx_event_quic.c
> > > +++ b/src/event/quic/ngx_event_quic.c
> > > @@ -421,9 +421,22 @@ ngx_quic_input_handler(ngx_event_t *rev)
> > >      }
> > >  
> > >      if (c->close) {
> > > -        if (qc->conf->shutdown) {
> > > +        c->close = 0;
> > > +
> > > +        if (!ngx_exiting) {
> > > +            qc->error = NGX_QUIC_ERR_NO_ERROR;
> > > +            qc->error_reason = "graceful shutdown";
> > > +            ngx_quic_close_connection(c, NGX_ERROR);
> > > +            return;
> > > +        }
> > > +
> > > +        if (!qc->closing && qc->conf->shutdown) {
> > > +            /* do not delay events posted by shutdown() */
> > > +            ngx_post_event(rev, &ngx_posted_next_events);
> > 
> > Following the change http://hg.nginx.org/nginx/rev/b809f53d3f5b, this part
> > is no longer needed.  The new diff attached.
> 
> Note that adding support for "idle mode for main connection" combined
> with b809f53d3f5b uncovers a socket leak due to qc->close.cancelable set.
> 
> In more details:
> - there can be connections with stretched pto e.g. due to missing client ack,
>   large enough such that a close timer (3*pto) fires after the idle timer
> - now that QUIC connection is handled on graceful shutdown as idle,
>   entering graceful shutdown results in setting a close timer, and closing
>   can take additional cycle following the "qc->close.timer_set" condition
> - pto and idle timers fired (and removed)
> - only close and, possibly, keepalive timers left; both cancelable
> 
> I suggest removing the cancelable property of a close timer
> given that we want to wait for 3*PTO while in a closing state
> and send back sensible CONNECTION_CLOSE to any further input
> (otherwise it would emit less sensible stateless reset).
> This will fix leaving event cycle without non-cancelable timers set.

Setting "cancelable" for a client-related entity looks wrong both in QUIC and
HTTP/3.  IMHO it should be removed, see attach.

> > > +
> > >              qc->conf->shutdown(c);
> > >          }
> > > +
> > >          return;
> > >      }
> > 
> > 
> > --
> > Roman Arutyunyan
> 
> > # HG changeset patch
> > # User Roman Arutyunyan <arut at nginx.com>
> > # Date 1669302768 -14400
> > #      Thu Nov 24 19:12:48 2022 +0400
> > # Branch quic
> > # Node ID ec21165abb4333f77fb85e956cccf300c8ae1acf
> > # Parent  6fba6061b65f0ee41c231bde6b738cfae00bb179
> > [mq]: quic-idle-fix1
> > 
> > diff --git a/src/event/quic/ngx_event_quic.c b/src/event/quic/ngx_event_quic.c
> > --- a/src/event/quic/ngx_event_quic.c
> > +++ b/src/event/quic/ngx_event_quic.c
> > @@ -421,9 +421,19 @@ ngx_quic_input_handler(ngx_event_t *rev)
> >      }
> >  
> >      if (c->close) {
> > -        if (qc->conf->shutdown) {
> > +        c->close = 0;
> > +
> > +        if (!ngx_exiting) {
> > +            qc->error = NGX_QUIC_ERR_NO_ERROR;
> > +            qc->error_reason = "graceful shutdown";
> > +            ngx_quic_close_connection(c, NGX_ERROR);
> > +            return;
> > +        }
> > +
> > +        if (!qc->closing && qc->conf->shutdown) {
> >              qc->conf->shutdown(c);
> >          }
> > +
> >          return;
> >      }
> >  
> > diff --git a/src/http/v3/ngx_http_v3_request.c b/src/http/v3/ngx_http_v3_request.c
> > --- a/src/http/v3/ngx_http_v3_request.c
> > +++ b/src/http/v3/ngx_http_v3_request.c
> > @@ -101,7 +101,7 @@ ngx_http_v3_init(ngx_connection_t *c)
> >  void
> >  ngx_http_v3_shutdown(ngx_connection_t *c)
> >  {
> > -    ngx_http_v3_session_t   *h3c;
> > +    ngx_http_v3_session_t  *h3c;
> >  
> >      ngx_log_debug0(NGX_LOG_DEBUG_HTTP, c->log, 0, "http3 shutdown");
> >  
> 
> Looks good.
> _______________________________________________
> nginx-devel mailing list -- nginx-devel at nginx.org
> To unsubscribe send an email to nginx-devel-leave at nginx.org
-------------- next part --------------
# HG changeset patch
# User Roman Arutyunyan <arut at nginx.com>
# Date 1669730306 -14400
#      Tue Nov 29 17:58:26 2022 +0400
# Branch quic
# Node ID 174c09dedc046aac2ca5308ab8957fc61b27c778
# Parent  576d9decab8bdb34a85b0cf52acf3afb3ae97b8d
imported patch quic-idle-fix2

diff --git a/src/event/quic/ngx_event_quic.c b/src/event/quic/ngx_event_quic.c
--- a/src/event/quic/ngx_event_quic.c
+++ b/src/event/quic/ngx_event_quic.c
@@ -275,22 +275,18 @@ ngx_quic_new_connection(ngx_connection_t
     qc->pto.log = c->log;
     qc->pto.data = c;
     qc->pto.handler = ngx_quic_pto_handler;
-    qc->pto.cancelable = 1;
 
     qc->push.log = c->log;
     qc->push.data = c;
     qc->push.handler = ngx_quic_push_handler;
-    qc->push.cancelable = 1;
 
     qc->close.log = c->log;
     qc->close.data = c;
     qc->close.handler = ngx_quic_close_handler;
-    qc->close.cancelable = 1;
 
     qc->path_validation.log = c->log;
     qc->path_validation.data = c;
     qc->path_validation.handler = ngx_quic_path_validation_handler;
-    qc->path_validation.cancelable = 1;
 
     qc->conf = conf;
 
diff --git a/src/http/v3/ngx_http_v3.c b/src/http/v3/ngx_http_v3.c
--- a/src/http/v3/ngx_http_v3.c
+++ b/src/http/v3/ngx_http_v3.c
@@ -55,7 +55,6 @@ ngx_http_v3_init_session(ngx_connection_
     h3c->keepalive.log = pc->log;
     h3c->keepalive.data = pc;
     h3c->keepalive.handler = ngx_http_v3_keepalive_handler;
-    h3c->keepalive.cancelable = 1;
 
     h3c->table.send_insert_count.log = pc->log;
     h3c->table.send_insert_count.data = pc;

From mdounin at mdounin.ru  Wed Nov 30 03:41:59 2022
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Wed, 30 Nov 2022 06:41:59 +0300
Subject: switch to ms resolution for rate limiting
In-Reply-To: <CAFK6my4ius+gWX=iDnZZ-V==ANJDUOJPkBOuPs+jMkt1c8tL5A@mail.gmail.com>
References: <CAFK6my4ius+gWX=iDnZZ-V==ANJDUOJPkBOuPs+jMkt1c8tL5A@mail.gmail.com>
Message-ID: <Y4bRB8xi5OEr2S6C@mdounin.ru>

Hello!

On Mon, Nov 28, 2022 at 06:40:33AM +0100, Jérôme Loyet wrote:

> Hello,
> 
> I'm using rate limiting within the stream module. While it works great
> for long connections it does not work on request smaller than the rate
> limite size for 1 second. I set up a 1gbps rate limit (limit_rate
> 125m) and request smaller than 125M or not limited. This is a normal
> behavioir as the rate limiting is done with a second precision. This
> patch change second precision to millisecond precision. From my first
> tests (still on going) it seems to works better.
> 
> What guys do you think about this patch ?

A while ago a similar patch was considered for both stream and 
http, though there is an open question on how to handle 
long-running requests on 32-bit platforms (where ngx_msec_t will 
overflow after ~48 days).  With the naive approach, which is also 
seen in your patch, all traffic on a such connection is likely to 
completely stop after the overflow, which does not seem to be a 
good outcome.

-- 
Maxim Dounin
http://mdounin.ru/


From mdounin at mdounin.ru  Wed Nov 30 04:22:58 2022
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Wed, 30 Nov 2022 07:22:58 +0300
Subject: [PATCH v2] Removed the casts within ngx_memcmp()
In-Reply-To: <Y2vBPDFSnKfWNmDX@mdounin.ru>
References: <d8fed8e2-0316-5c5e-cea1-6a5d894cf76c@gmail.com>
 <20221108105539.3924-1-alx@nginx.com>
 <Y2vBPDFSnKfWNmDX@mdounin.ru>
Message-ID: <Y4baoj/FOWDUnaUM@mdounin.ru>

Hello!

Ping.

On Wed, Nov 09, 2022 at 06:03:24PM +0300, Maxim Dounin wrote:

[...]

> # HG changeset patch
> # User Maxim Dounin <mdounin at mdounin.ru>
> # Date 1668004692 -10800
> #      Wed Nov 09 17:38:12 2022 +0300
> # Node ID fc79ea0724a92c1f463625a11ed4cb785cd342b7
> # Parent  42bc158a47ecb3c2bd0396c723c307c757f2770e
> Fixed alignment of ngx_memmove()/ngx_movemem() macro definitions.
> 
> diff --git a/src/core/ngx_string.h b/src/core/ngx_string.h
> --- a/src/core/ngx_string.h
> +++ b/src/core/ngx_string.h
> @@ -140,8 +140,8 @@ ngx_copy(u_char *dst, u_char *src, size_
>  #endif
>  
>  
> -#define ngx_memmove(dst, src, n)   (void) memmove(dst, src, n)
> -#define ngx_movemem(dst, src, n)   (((u_char *) memmove(dst, src, n)) + (n))
> +#define ngx_memmove(dst, src, n)  (void) memmove(dst, src, n)
> +#define ngx_movemem(dst, src, n)  (((u_char *) memmove(dst, src, n)) + (n))
>  
>  
>  /* msvc and icc7 compile memcmp() to the inline loop */
> # HG changeset patch
> # User Maxim Dounin <mdounin at mdounin.ru>
> # Date 1668005196 -10800
> #      Wed Nov 09 17:46:36 2022 +0300
> # Node ID 5269880f00df1e5ae08299165ec43435b759c5a3
> # Parent  fc79ea0724a92c1f463625a11ed4cb785cd342b7
> Removed casts from ngx_memcmp() macro.
> 
> Casts are believed to be not needed, since memcmp() has "const void *"
> arguments since introduction of the "void" type in C89.  And on pre-C89
> platforms nginx is unlikely to compile without warnings anyway, as there
> are no casts in memcpy() and memmove() calls.
> 
> These casts were added in 1648:89a47f19b9ec without any details on why they
> were added, and Igor does not remember details either.  The most plausible
> explanation is that they were copied from ngx_strcmp() and were not really
> needed even at that time.
> 
> Prodded by Alejandro Colomar.
> 
> diff --git a/src/core/ngx_string.h b/src/core/ngx_string.h
> --- a/src/core/ngx_string.h
> +++ b/src/core/ngx_string.h
> @@ -145,7 +145,7 @@ ngx_copy(u_char *dst, u_char *src, size_
>  
>  
>  /* msvc and icc7 compile memcmp() to the inline loop */
> -#define ngx_memcmp(s1, s2, n)  memcmp((const char *) s1, (const char *) s2, n)
> +#define ngx_memcmp(s1, s2, n)     memcmp(s1, s2, n)
>  
>  
>  u_char *ngx_cpystrn(u_char *dst, u_char *src, size_t n);

-- 
Maxim Dounin
http://mdounin.ru/


From arut at nginx.com  Wed Nov 30 10:11:15 2022
From: arut at nginx.com (Roman Arutyunyan)
Date: Wed, 30 Nov 2022 14:11:15 +0400
Subject: [PATCH 08 of 10] QUIC: idle mode for main connection
In-Reply-To: <20221129140040.t7rhdnhoipzslmoe@N00W24XTQX>
References: <patchbomb.1662627987@arut-laptop>
 <e0634a484d9a2d82d43f.1662627995@arut-laptop>
 <20221020115015.azqjb5wd7fisgm2f@Y9MQ9X2QVV>
 <20221020142503.d327lsrunx6hzw63@N00W24XTQX>
 <20221124151531.s4oldl3jeo4wtobv@N00W24XTQX>
 <20221128174005.esncmvmumjstgbz3@Y9MQ9X2QVV>
 <20221129140040.t7rhdnhoipzslmoe@N00W24XTQX>
Message-ID: <20221130101115.zdfikbxzvs356tho@N00W24XTQX>

Hi,

On Tue, Nov 29, 2022 at 06:00:40PM +0400, Roman Arutyunyan wrote:
> Hi,
> 
> On Mon, Nov 28, 2022 at 09:40:05PM +0400, Sergey Kandaurov wrote:
> > On Thu, Nov 24, 2022 at 07:15:31PM +0400, Roman Arutyunyan wrote:
> > > Hi,
> > > 
> > > On Thu, Oct 20, 2022 at 06:25:03PM +0400, Roman Arutyunyan wrote:
> > > > Hi,
> > > > 
> > > > On Thu, Oct 20, 2022 at 03:50:15PM +0400, Sergey Kandaurov wrote:
> > > > > On Thu, Sep 08, 2022 at 01:06:35PM +0400, Roman Arutyunyan wrote:
> > > > > > # HG changeset patch
> > > > > > # User Roman Arutyunyan <arut at nginx.com>
> > > > > > # Date 1662627133 -14400
> > > > > > #      Thu Sep 08 12:52:13 2022 +0400
> > > > > > # Branch quic
> > > > > > # Node ID e0634a484d9a2d82d43f565d64a0a22e989ac1cb
> > > > > > # Parent  1dd6fabfdcb5b52af495f9d8fc00f64ae36a537c
> > > > > > QUIC: idle mode for main connection.
> > > > > > 
> > > > > > Now main QUIC connection for HTTP/3 always has c->idle flag set.  This allows
> > > > > > the connection to receive worker shutdown notification.  It is passed to
> > > > > > application level via a new conf->shutdown() callback.
> > > > > > 
> > > > > > The HTTP/3 shutdown callback sends GOAWAY to client and gracefully shuts down
> > > > > > the QUIC connection.
> > > > > > 
> > > > > > diff --git a/src/event/quic/ngx_event_quic.c b/src/event/quic/ngx_event_quic.c
> > > > > > --- a/src/event/quic/ngx_event_quic.c
> > > > > > +++ b/src/event/quic/ngx_event_quic.c
> > > > > > @@ -341,6 +341,7 @@ ngx_quic_new_connection(ngx_connection_t
> > > > > >          return NULL;
> > > > > >      }
> > > > > >  
> > > > > > +    c->idle = 1;
> > > > > >      ngx_reusable_connection(c, 1);
> > > > > >  
> > > > > >      ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0,
> > > > > > @@ -420,9 +421,9 @@ ngx_quic_input_handler(ngx_event_t *rev)
> > > > > >      }
> > > > > >  
> > > > > >      if (c->close) {
> > > > > > -        qc->error = NGX_QUIC_ERR_NO_ERROR;
> > > > > > -        qc->error_reason = "graceful shutdown";
> > > > > > -        ngx_quic_close_connection(c, NGX_ERROR);
> > > > > > +        if (qc->conf->shutdown) {
> > > > > 
> > > > > As previously discussed in private, this will need an additional check
> > > > > that we are not yet in qc->closing.
> > > > > 
> > > > > > +            qc->conf->shutdown(c);
> > > > > > +        }
> > > > > >          return;
> > > > > >      }
> > > > 
> > > > 
> > > > Yes, added the check.  Also, c->close is reset here similarly to HTTP/2
> > > > since we want to be able to handle future packets normally.
> > > > 
> > > > Also, current code which closes the connection instantly should remain for
> > > > connection reuse.  To tell reuse from shutdown we can check ngx_exiting.
> > > > Assuming reuse does not make sense in shutdown mode, this will work good.
> > > > 
> > > > > > diff --git a/src/event/quic/ngx_event_quic.h b/src/event/quic/ngx_event_quic.h
> > > > > > --- a/src/event/quic/ngx_event_quic.h
> > > > > > +++ b/src/event/quic/ngx_event_quic.h
> > > > > > @@ -28,6 +28,9 @@
> > > > > >  #define NGX_QUIC_STREAM_UNIDIRECTIONAL       0x02
> > > > > >  
> > > > > >  
> > > > > > +typedef void (*ngx_quic_shutdown_pt)(ngx_connection_t *c);
> > > > > > +
> > > > > > +
> > > > > >  typedef enum {
> > > > > >      NGX_QUIC_STREAM_SEND_READY = 0,
> > > > > >      NGX_QUIC_STREAM_SEND_SEND,
> > > > > > @@ -74,6 +77,8 @@ typedef struct {
> > > > > >      ngx_int_t                      stream_reject_code_uni;
> > > > > >      ngx_int_t                      stream_reject_code_bidi;
> > > > > >  
> > > > > > +    ngx_quic_shutdown_pt           shutdown;
> > > > > > +
> > > > > >      u_char                         av_token_key[NGX_QUIC_AV_KEY_LEN];
> > > > > >      u_char                         sr_token_key[NGX_QUIC_SR_KEY_LEN];
> > > > > >  } ngx_quic_conf_t;
> > > > > > diff --git a/src/http/v3/ngx_http_v3.h b/src/http/v3/ngx_http_v3.h
> > > > > > --- a/src/http/v3/ngx_http_v3.h
> > > > > > +++ b/src/http/v3/ngx_http_v3.h
> > > > > > @@ -141,6 +141,7 @@ struct ngx_http_v3_session_s {
> > > > > >      uint64_t                      next_push_id;
> > > > > >      uint64_t                      max_push_id;
> > > > > >      uint64_t                      goaway_push_id;
> > > > > > +    uint64_t                      next_request_id;
> > > > > >  
> > > > > >      off_t                         total_bytes;
> > > > > >      off_t                         payload_bytes;
> > > > > > @@ -158,6 +159,7 @@ void ngx_http_v3_init(ngx_connection_t *
> > > > > >  void ngx_http_v3_reset_connection(ngx_connection_t *c);
> > > > > >  ngx_int_t ngx_http_v3_init_session(ngx_connection_t *c);
> > > > > >  ngx_int_t ngx_http_v3_check_flood(ngx_connection_t *c);
> > > > > > +void ngx_http_v3_shutdown(ngx_connection_t *c);
> > > > > >  
> > > > > >  ngx_int_t ngx_http_v3_read_request_body(ngx_http_request_t *r);
> > > > > >  ngx_int_t ngx_http_v3_read_unbuffered_request_body(ngx_http_request_t *r);
> > > > > > diff --git a/src/http/v3/ngx_http_v3_module.c b/src/http/v3/ngx_http_v3_module.c
> > > > > > --- a/src/http/v3/ngx_http_v3_module.c
> > > > > > +++ b/src/http/v3/ngx_http_v3_module.c
> > > > > > @@ -249,6 +249,8 @@ ngx_http_v3_create_srv_conf(ngx_conf_t *
> > > > > >      h3scf->quic.stream_reject_code_bidi = NGX_HTTP_V3_ERR_REQUEST_REJECTED;
> > > > > >      h3scf->quic.active_connection_id_limit = NGX_CONF_UNSET_UINT;
> > > > > >  
> > > > > > +    h3scf->quic.shutdown = ngx_http_v3_shutdown;
> > > > > > +
> > > > > >      return h3scf;
> > > > > >  }
> > > > > >  
> > > > > > diff --git a/src/http/v3/ngx_http_v3_request.c b/src/http/v3/ngx_http_v3_request.c
> > > > > > --- a/src/http/v3/ngx_http_v3_request.c
> > > > > > +++ b/src/http/v3/ngx_http_v3_request.c
> > > > > > @@ -97,6 +97,37 @@ ngx_http_v3_init(ngx_connection_t *c)
> > > > > >  }
> > > > > >  
> > > > > >  
> > > > > > +void
> > > > > > +ngx_http_v3_shutdown(ngx_connection_t *c)
> > > > > > +{
> > > > > > +    ngx_http_v3_session_t   *h3c;
> > > > > 
> > > > > extra indent
> > > > > 
> > > > > > +
> > > > > > +    ngx_log_debug0(NGX_LOG_DEBUG_HTTP, c->log, 0, "http3 shutdown");
> > > > > > +
> > > > > > +    h3c = ngx_http_v3_get_session(c);
> > > > > > +
> > > > > > +    if (h3c == NULL) {
> > > > > > +        ngx_quic_finalize_connection(c, NGX_HTTP_V3_ERR_NO_ERROR,
> > > > > > +                                     "connection shutdown");
> > > > > > +        return;
> > > > > > +    }
> > > > > > +
> > > > > > +    if (!h3c->goaway) {
> > > > > > +        h3c->goaway = 1;
> > > > > > +
> > > > > > +#if (NGX_HTTP_V3_HQ)
> > > > > > +        if (!h3c->hq)
> > > > > > +#endif
> > > > > > +        {
> > > > > > +            (void) ngx_http_v3_send_goaway(c, h3c->next_request_id);
> > > > > > +        }
> > > > > > +
> > > > > > +        ngx_http_v3_shutdown_connection(c, NGX_HTTP_V3_ERR_NO_ERROR,
> > > > > > +                                        "connection shutdown");
> > > > > > +    }
> > > > > 
> > > > > Note that this callback is used to be called from a read event as part of
> > > > > graceful shutdown.
> > > > > With ngx_quic_finalize_connection() remade in patch #4 (reusable mode)
> > > > > to defer closing QUIC connection to a posted event, this call now results
> > > > > in a posted event, which no one can fulfill, hence no further action until
> > > > > quic idle timeout fires.
> > > > > It could be fixed by executing known posted events after shutdown callback
> > > > > or more globally - as part of graceful shutdown itself.
> > > > 
> > > > Yes, events posted from ngx_close_idle_connections() are not handled right away.
> > > > Instead, they are handled at the end of the next cycle, which normally
> > > > happens after a timeout.  There seems to be no pretty way to fix this, unless
> > > > we handle posted events in ngx_worker_process_cycle() right after
> > > > ngx_close_idle_connections().  We are trying to avoid global changes like this.
> > > > 
> > > > I suggest posting current connection read event as a next posted event.  This
> > > > will effectively set next cycle timeout to be zero and eliminate the problem.
> > > > 
> > > > > > +}
> > > > > > +
> > > > > > +
> > > > > >  static void
> > > > > >  ngx_http_v3_init_request_stream(ngx_connection_t *c)
> > > > > >  {
> > > > > > @@ -137,6 +168,8 @@ ngx_http_v3_init_request_stream(ngx_conn
> > > > > >  
> > > > > >      pc = c->quic->parent;
> > > > > >  
> > > > > > +    h3c->next_request_id = c->quic->id + 0x04;
> > > > > > +
> > > > > >      if (n + 1 == clcf->keepalive_requests
> > > > > >          || ngx_current_msec - pc->start_time > clcf->keepalive_time)
> > > > > >      {
> > > > > > @@ -146,7 +179,7 @@ ngx_http_v3_init_request_stream(ngx_conn
> > > > > >          if (!h3c->hq)
> > > > > >  #endif
> > > > > >          {
> > > > > > -            if (ngx_http_v3_send_goaway(c, (n + 1) << 2) != NGX_OK) {
> > > > > > +            if (ngx_http_v3_send_goaway(c, h3c->next_request_id) != NGX_OK) {
> > > > > >                  ngx_http_close_connection(c);
> > > > > >                  return;
> > > > > >              }
> > > > > > 
> > > > > > _______________________________________________
> > > > > > nginx-devel mailing list -- nginx-devel at nginx.org
> > > > > > To unsubscribe send an email to nginx-devel-leave at nginx.org
> > > > > _______________________________________________
> > > > > nginx-devel mailing list -- nginx-devel at nginx.org
> > > > > To unsubscribe send an email to nginx-devel-leave at nginx.org
> > > > 
> > > > Attached is a diff to the current patch.
> > > > 
> > > > --
> > > > Roman
> > > 
> > > > # HG changeset patch
> > > > # User Roman Arutyunyan <arut at nginx.com>
> > > > # Date 1666273166 -14400
> > > > #      Thu Oct 20 17:39:26 2022 +0400
> > > > # Branch quic
> > > > # Node ID d6c725081a0b024886822e1cc722fdace9c32621
> > > > # Parent  a4ba2ac5fa55ef94bb75a66e66e0b19d792fed10
> > > > [mq]: quic-idle-fix1
> > > > 
> > > > diff --git a/src/event/quic/ngx_event_quic.c b/src/event/quic/ngx_event_quic.c
> > > > --- a/src/event/quic/ngx_event_quic.c
> > > > +++ b/src/event/quic/ngx_event_quic.c
> > > > @@ -421,9 +421,22 @@ ngx_quic_input_handler(ngx_event_t *rev)
> > > >      }
> > > >  
> > > >      if (c->close) {
> > > > -        if (qc->conf->shutdown) {
> > > > +        c->close = 0;
> > > > +
> > > > +        if (!ngx_exiting) {
> > > > +            qc->error = NGX_QUIC_ERR_NO_ERROR;
> > > > +            qc->error_reason = "graceful shutdown";
> > > > +            ngx_quic_close_connection(c, NGX_ERROR);
> > > > +            return;
> > > > +        }
> > > > +
> > > > +        if (!qc->closing && qc->conf->shutdown) {
> > > > +            /* do not delay events posted by shutdown() */
> > > > +            ngx_post_event(rev, &ngx_posted_next_events);
> > > 
> > > Following the change http://hg.nginx.org/nginx/rev/b809f53d3f5b, this part
> > > is no longer needed.  The new diff attached.
> > 
> > Note that adding support for "idle mode for main connection" combined
> > with b809f53d3f5b uncovers a socket leak due to qc->close.cancelable set.
> > 
> > In more details:
> > - there can be connections with stretched pto e.g. due to missing client ack,
> >   large enough such that a close timer (3*pto) fires after the idle timer
> > - now that QUIC connection is handled on graceful shutdown as idle,
> >   entering graceful shutdown results in setting a close timer, and closing
> >   can take additional cycle following the "qc->close.timer_set" condition
> > - pto and idle timers fired (and removed)
> > - only close and, possibly, keepalive timers left; both cancelable
> > 
> > I suggest removing the cancelable property of a close timer
> > given that we want to wait for 3*PTO while in a closing state
> > and send back sensible CONNECTION_CLOSE to any further input
> > (otherwise it would emit less sensible stateless reset).
> > This will fix leaving event cycle without non-cancelable timers set.
> 
> Setting "cancelable" for a client-related entity looks wrong both in QUIC and
> HTTP/3.  IMHO it should be removed, see attach.
> 
> > > > +
> > > >              qc->conf->shutdown(c);
> > > >          }
> > > > +
> > > >          return;
> > > >      }
> > > 
> > > 
> > > --
> > > Roman Arutyunyan
> > 
> > > # HG changeset patch
> > > # User Roman Arutyunyan <arut at nginx.com>
> > > # Date 1669302768 -14400
> > > #      Thu Nov 24 19:12:48 2022 +0400
> > > # Branch quic
> > > # Node ID ec21165abb4333f77fb85e956cccf300c8ae1acf
> > > # Parent  6fba6061b65f0ee41c231bde6b738cfae00bb179
> > > [mq]: quic-idle-fix1
> > > 
> > > diff --git a/src/event/quic/ngx_event_quic.c b/src/event/quic/ngx_event_quic.c
> > > --- a/src/event/quic/ngx_event_quic.c
> > > +++ b/src/event/quic/ngx_event_quic.c
> > > @@ -421,9 +421,19 @@ ngx_quic_input_handler(ngx_event_t *rev)
> > >      }
> > >  
> > >      if (c->close) {
> > > -        if (qc->conf->shutdown) {
> > > +        c->close = 0;
> > > +
> > > +        if (!ngx_exiting) {
> > > +            qc->error = NGX_QUIC_ERR_NO_ERROR;
> > > +            qc->error_reason = "graceful shutdown";
> > > +            ngx_quic_close_connection(c, NGX_ERROR);
> > > +            return;
> > > +        }
> > > +
> > > +        if (!qc->closing && qc->conf->shutdown) {
> > >              qc->conf->shutdown(c);
> > >          }
> > > +
> > >          return;
> > >      }
> > >  
> > > diff --git a/src/http/v3/ngx_http_v3_request.c b/src/http/v3/ngx_http_v3_request.c
> > > --- a/src/http/v3/ngx_http_v3_request.c
> > > +++ b/src/http/v3/ngx_http_v3_request.c
> > > @@ -101,7 +101,7 @@ ngx_http_v3_init(ngx_connection_t *c)
> > >  void
> > >  ngx_http_v3_shutdown(ngx_connection_t *c)
> > >  {
> > > -    ngx_http_v3_session_t   *h3c;
> > > +    ngx_http_v3_session_t  *h3c;
> > >  
> > >      ngx_log_debug0(NGX_LOG_DEBUG_HTTP, c->log, 0, "http3 shutdown");
> > >  
> > 
> > Looks good.
> > _______________________________________________
> > nginx-devel mailing list -- nginx-devel at nginx.org
> > To unsubscribe send an email to nginx-devel-leave at nginx.org

> # HG changeset patch
> # User Roman Arutyunyan <arut at nginx.com>
> # Date 1669730306 -14400
> #      Tue Nov 29 17:58:26 2022 +0400
> # Branch quic
> # Node ID 174c09dedc046aac2ca5308ab8957fc61b27c778
> # Parent  576d9decab8bdb34a85b0cf52acf3afb3ae97b8d
> imported patch quic-idle-fix2
> 
> diff --git a/src/event/quic/ngx_event_quic.c b/src/event/quic/ngx_event_quic.c
> --- a/src/event/quic/ngx_event_quic.c
> +++ b/src/event/quic/ngx_event_quic.c
> @@ -275,22 +275,18 @@ ngx_quic_new_connection(ngx_connection_t
>      qc->pto.log = c->log;
>      qc->pto.data = c;
>      qc->pto.handler = ngx_quic_pto_handler;
> -    qc->pto.cancelable = 1;
>  
>      qc->push.log = c->log;
>      qc->push.data = c;
>      qc->push.handler = ngx_quic_push_handler;
> -    qc->push.cancelable = 1;
>  
>      qc->close.log = c->log;
>      qc->close.data = c;
>      qc->close.handler = ngx_quic_close_handler;
> -    qc->close.cancelable = 1;
>  
>      qc->path_validation.log = c->log;
>      qc->path_validation.data = c;
>      qc->path_validation.handler = ngx_quic_path_validation_handler;
> -    qc->path_validation.cancelable = 1;
>  
>      qc->conf = conf;
>  
> diff --git a/src/http/v3/ngx_http_v3.c b/src/http/v3/ngx_http_v3.c
> --- a/src/http/v3/ngx_http_v3.c
> +++ b/src/http/v3/ngx_http_v3.c
> @@ -55,7 +55,6 @@ ngx_http_v3_init_session(ngx_connection_
>      h3c->keepalive.log = pc->log;
>      h3c->keepalive.data = pc;
>      h3c->keepalive.handler = ngx_http_v3_keepalive_handler;
> -    h3c->keepalive.cancelable = 1;
>  
>      h3c->table.send_insert_count.log = pc->log;
>      h3c->table.send_insert_count.data = pc;

Full patch.

--
Roman

-------------- next part --------------
# HG changeset patch
# User Roman Arutyunyan <arut at nginx.com>
# Date 1669802948 -14400
#      Wed Nov 30 14:09:08 2022 +0400
# Branch quic
# Node ID 97844d0c98127588932f04948b4ef0a144fd5826
# Parent  f9f8bdc571ea0a0b5cc1dc97e3c168db37d32c67
QUIC: removed cancelable flag from QUIC and HTTP/3 events.

All these events are created in context of a client connection and are deleted
when the connection is closed.  Setting ev->cancelable could trigger premature
connection closure and a socket leak alert.

diff --git a/src/event/quic/ngx_event_quic.c b/src/event/quic/ngx_event_quic.c
--- a/src/event/quic/ngx_event_quic.c
+++ b/src/event/quic/ngx_event_quic.c
@@ -275,22 +275,18 @@ ngx_quic_new_connection(ngx_connection_t
     qc->pto.log = c->log;
     qc->pto.data = c;
     qc->pto.handler = ngx_quic_pto_handler;
-    qc->pto.cancelable = 1;
 
     qc->push.log = c->log;
     qc->push.data = c;
     qc->push.handler = ngx_quic_push_handler;
-    qc->push.cancelable = 1;
 
     qc->close.log = c->log;
     qc->close.data = c;
     qc->close.handler = ngx_quic_close_handler;
-    qc->close.cancelable = 1;
 
     qc->path_validation.log = c->log;
     qc->path_validation.data = c;
     qc->path_validation.handler = ngx_quic_path_validation_handler;
-    qc->path_validation.cancelable = 1;
 
     qc->conf = conf;
 
diff --git a/src/http/v3/ngx_http_v3.c b/src/http/v3/ngx_http_v3.c
--- a/src/http/v3/ngx_http_v3.c
+++ b/src/http/v3/ngx_http_v3.c
@@ -55,7 +55,6 @@ ngx_http_v3_init_session(ngx_connection_
     h3c->keepalive.log = pc->log;
     h3c->keepalive.data = pc;
     h3c->keepalive.handler = ngx_http_v3_keepalive_handler;
-    h3c->keepalive.cancelable = 1;
 
     h3c->table.send_insert_count.log = pc->log;
     h3c->table.send_insert_count.data = pc;

From alx.manpages at gmail.com  Wed Nov 30 10:49:18 2022
From: alx.manpages at gmail.com (Alex Colomar)
Date: Wed, 30 Nov 2022 11:49:18 +0100
Subject: [PATCH v2] Removed the casts within ngx_memcmp()
In-Reply-To: <Y4baoj/FOWDUnaUM@mdounin.ru>
References: <d8fed8e2-0316-5c5e-cea1-6a5d894cf76c@gmail.com>
 <20221108105539.3924-1-alx@nginx.com> <Y2vBPDFSnKfWNmDX@mdounin.ru>
 <Y4baoj/FOWDUnaUM@mdounin.ru>
Message-ID: <e20c38dc-30f5-3219-8702-76662d8cea49@gmail.com>

Hello Maxim!

On 11/30/22 05:22, Maxim Dounin wrote:
> Hello!
> 
> Ping.

Sorry, I didn't know you were waiting for my confirmation.

> 
> On Wed, Nov 09, 2022 at 06:03:24PM +0300, Maxim Dounin wrote:
> 
> [...]
> 
>> # HG changeset patch
>> # User Maxim Dounin <mdounin at mdounin.ru>
>> # Date 1668004692 -10800
>> #      Wed Nov 09 17:38:12 2022 +0300
>> # Node ID fc79ea0724a92c1f463625a11ed4cb785cd342b7
>> # Parent  42bc158a47ecb3c2bd0396c723c307c757f2770e
>> Fixed alignment of ngx_memmove()/ngx_movemem() macro definitions.

Makes sense.

>>
>> diff --git a/src/core/ngx_string.h b/src/core/ngx_string.h
>> --- a/src/core/ngx_string.h
>> +++ b/src/core/ngx_string.h
>> @@ -140,8 +140,8 @@ ngx_copy(u_char *dst, u_char *src, size_
>>   #endif
>>   
>>   
>> -#define ngx_memmove(dst, src, n)   (void) memmove(dst, src, n)
>> -#define ngx_movemem(dst, src, n)   (((u_char *) memmove(dst, src, n)) + (n))
>> +#define ngx_memmove(dst, src, n)  (void) memmove(dst, src, n)
>> +#define ngx_movemem(dst, src, n)  (((u_char *) memmove(dst, src, n)) + (n))
>>   
>>   
>>   /* msvc and icc7 compile memcmp() to the inline loop */
>> # HG changeset patch
>> # User Maxim Dounin <mdounin at mdounin.ru>
>> # Date 1668005196 -10800
>> #      Wed Nov 09 17:46:36 2022 +0300
>> # Node ID 5269880f00df1e5ae08299165ec43435b759c5a3
>> # Parent  fc79ea0724a92c1f463625a11ed4cb785cd342b7
>> Removed casts from ngx_memcmp() macro.
>>
>> Casts are believed to be not needed, since memcmp() has "const void *"
>> arguments since introduction of the "void" type in C89.  And on pre-C89
>> platforms nginx is unlikely to compile without warnings anyway, as there
>> are no casts in memcpy() and memmove() calls.
>>
>> These casts were added in 1648:89a47f19b9ec without any details on why they
>> were added, and Igor does not remember details either.  The most plausible
>> explanation is that they were copied from ngx_strcmp() and were not really
>> needed even at that time.
>>
>> Prodded by Alejandro Colomar.

And of course, this patch LGTM :)

Cheers,

Alex

>>
>> diff --git a/src/core/ngx_string.h b/src/core/ngx_string.h
>> --- a/src/core/ngx_string.h
>> +++ b/src/core/ngx_string.h
>> @@ -145,7 +145,7 @@ ngx_copy(u_char *dst, u_char *src, size_
>>   
>>   
>>   /* msvc and icc7 compile memcmp() to the inline loop */
>> -#define ngx_memcmp(s1, s2, n)  memcmp((const char *) s1, (const char *) s2, n)
>> +#define ngx_memcmp(s1, s2, n)     memcmp(s1, s2, n)
>>   
>>   
>>   u_char *ngx_cpystrn(u_char *dst, u_char *src, size_t n);
> 

-- 
<http://www.alejandro-colomar.es/>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20221130/0abf1819/attachment.bin>

From yolkking at 126.com  Wed Nov 30 12:10:29 2022
From: yolkking at 126.com (Yu Zhu)
Date: Wed, 30 Nov 2022 20:10:29 +0800 (CST)
Subject: QUIC: position of RTT and congestion
Message-ID: <56b10dbb.3bca.184c8720fd8.Coremail.yolkking@126.com>


Hi,

    As described in "rfc 9002 6. Loss Detection",  "RTT and congestion control are properties of the path", so moves first_rtt, latest_rtt, avg_rtt, min_rtt, rttvar and congestion from ngx_quic_connection_t to struct ngx_quic_path_t looks more reasonable?





--
Yu Zhu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20221130/6e74646e/attachment.htm>

From pluknet at nginx.com  Wed Nov 30 12:17:15 2022
From: pluknet at nginx.com (Sergey Kandaurov)
Date: Wed, 30 Nov 2022 16:17:15 +0400
Subject: [PATCH v2] Removed the casts within ngx_memcmp()
In-Reply-To: <Y2vBPDFSnKfWNmDX@mdounin.ru>
References: <d8fed8e2-0316-5c5e-cea1-6a5d894cf76c@gmail.com>
 <20221108105539.3924-1-alx@nginx.com>
 <Y2vBPDFSnKfWNmDX@mdounin.ru>
Message-ID: <20221130121715.45yx5g4upwxhrs3k@Y9MQ9X2QVV>

On Wed, Nov 09, 2022 at 06:03:24PM +0300, Maxim Dounin wrote:
> # HG changeset patch
> # User Maxim Dounin <mdounin at mdounin.ru>
> # Date 1668004692 -10800
> #      Wed Nov 09 17:38:12 2022 +0300
> # Node ID fc79ea0724a92c1f463625a11ed4cb785cd342b7
> # Parent  42bc158a47ecb3c2bd0396c723c307c757f2770e
> Fixed alignment of ngx_memmove()/ngx_movemem() macro definitions.
> 
> diff --git a/src/core/ngx_string.h b/src/core/ngx_string.h
> --- a/src/core/ngx_string.h
> +++ b/src/core/ngx_string.h
> @@ -140,8 +140,8 @@ ngx_copy(u_char *dst, u_char *src, size_
>  #endif
>  
>  
> -#define ngx_memmove(dst, src, n)   (void) memmove(dst, src, n)
> -#define ngx_movemem(dst, src, n)   (((u_char *) memmove(dst, src, n)) + (n))
> +#define ngx_memmove(dst, src, n)  (void) memmove(dst, src, n)
> +#define ngx_movemem(dst, src, n)  (((u_char *) memmove(dst, src, n)) + (n))
>  
>  
>  /* msvc and icc7 compile memcmp() to the inline loop */
> # HG changeset patch
> # User Maxim Dounin <mdounin at mdounin.ru>
> # Date 1668005196 -10800
> #      Wed Nov 09 17:46:36 2022 +0300
> # Node ID 5269880f00df1e5ae08299165ec43435b759c5a3
> # Parent  fc79ea0724a92c1f463625a11ed4cb785cd342b7
> Removed casts from ngx_memcmp() macro.
> 
> Casts are believed to be not needed, since memcmp() has "const void *"
> arguments since introduction of the "void" type in C89.  And on pre-C89
> platforms nginx is unlikely to compile without warnings anyway, as there
> are no casts in memcpy() and memmove() calls.
> 
> These casts were added in 1648:89a47f19b9ec without any details on why they
> were added, and Igor does not remember details either.  The most plausible
> explanation is that they were copied from ngx_strcmp() and were not really
> needed even at that time.
> 
> Prodded by Alejandro Colomar.
> 
> diff --git a/src/core/ngx_string.h b/src/core/ngx_string.h
> --- a/src/core/ngx_string.h
> +++ b/src/core/ngx_string.h
> @@ -145,7 +145,7 @@ ngx_copy(u_char *dst, u_char *src, size_
>  
>  
>  /* msvc and icc7 compile memcmp() to the inline loop */
> -#define ngx_memcmp(s1, s2, n)  memcmp((const char *) s1, (const char *) s2, n)
> +#define ngx_memcmp(s1, s2, n)     memcmp(s1, s2, n)
>  
>  
>  u_char *ngx_cpystrn(u_char *dst, u_char *src, size_t n);
> 

Looks good.

Indeed, old memcmp definition is traced back to pre-ANSI.
In particular, you can find old implementation in 4.3BSD-Tahoe
(named as "Routines described in memory(BA_LIB); System V compatibility")
that uses "char *" arguments, until they were rewritten in 4.3BSD-Reno
in ANSI C by Chris Torek.
Also, it seems that VC 6.0 has memcmp with non-const void argument
as pre-C++98 (but I cannot support this clame with real facts).


From vl at inspert.ru  Wed Nov 30 12:31:33 2022
From: vl at inspert.ru (Vladimir Homutov)
Date: Wed, 30 Nov 2022 15:31:33 +0300
Subject: QUIC: position of RTT and congestion
In-Reply-To: <56b10dbb.3bca.184c8720fd8.Coremail.yolkking@126.com>
References: <56b10dbb.3bca.184c8720fd8.Coremail.yolkking@126.com>
Message-ID: <Y4dNJafRL1eZY5L+@vlws>

On Wed, Nov 30, 2022 at 08:10:29PM +0800, Yu Zhu wrote:
>
> Hi,
>
>     As described in "rfc 9002 6. Loss Detection",  "RTT and congestion
>     control are properties of the path", so moves first_rtt,
>     latest_rtt, avg_rtt, min_rtt, rttvar and congestion from
>     ngx_quic_connection_t to struct ngx_quic_path_t looks more
>     reasonable?

yes, you are right.

Currently per-path calculations are not implemented, as well as path mtu
discovery and some other things.


From pluknet at nginx.com  Wed Nov 30 15:29:35 2022
From: pluknet at nginx.com (Sergey Kandaurov)
Date: Wed, 30 Nov 2022 19:29:35 +0400
Subject: [PATCH 0 of 2] unbuffered proxying CPU hog (ticket #2418)
In-Reply-To: <patchbomb.1669673534@vm-bsd.mdounin.ru>
References: <patchbomb.1669673534@vm-bsd.mdounin.ru>
Message-ID: <8BB66B75-7321-4A51-9E97-FF66A3FD1A9D@nginx.com>


> On 29 Nov 2022, at 02:12, Maxim Dounin <mdounin at mdounin.ru> wrote:
> 
> Hello!
> 
> The following patch fixes CPU hog observed with unbuffered SSL proxying 
> after SSL errors (ticket #2418).  Fix is to always clear c->read->ready
> flag when returning errors from ngx_ssl_recv().
> 
> An additional patch cleans up some win32-specific edge cases (not expected
> to appear in practice though) when c->read->ready is not cleared when
> errors or EOFs are returned from ngx_wsarecv() and ngx_wsarecv_chain().
> 

Hello!

Both patches look good to me.

-- 
Sergey Kandaurov


From A.Bavshin at F5.com  Wed Nov 30 23:33:26 2022
From: A.Bavshin at F5.com (Aleksei Bavshin)
Date: Wed, 30 Nov 2022 23:33:26 +0000
Subject: [PATCH] HTTP: add internal_redirect directive.
In-Reply-To: <Y3cIhd0li6k6Bzdh@mdounin.ru>
References:  <CO6PR01MB7482906A6CA9DC1847971291C0069@CO6PR01MB7482.prod.exchangelabs.com>
 <Y3cIhd0li6k6Bzdh@mdounin.ru>
Message-ID:  <CO6PR01MB74826E503808C72D26AA7918C0159@CO6PR01MB7482.prod.exchangelabs.com>

Hello,

> > The directive in question provides direct access to the
> > ngx_http_internal_redirect/ngx_http_named_location APIs,
> > allowing to simplify and optimize several real-life
> > configurations where the modules should be executed multiple
> > times/in a different order/with a different configuration/etc,
> > sometimes based on a condition evaluated during the request
> > processing.
> 
> No, thanks.
> 
> Discussions about adding the "goto" directive date back to at
> least 2009.  Igor's and my position on this are summarized here:
> 
> https://mailman.nginx.org/pipermail/nginx-ru/2017-April/059736.html
> 
> Hope this helps.

Thank you, I missed these threads while looking through the archives. I'm still getting used to the vast amount of knowledge that is available only in Russian, and sometimes forget to retry the search with a translated query.

So, Igor's main point was that a directive like that allows creating unmaintainable configurations and will be heavily abused? It is a fair point and it made me hesitate before submitting the patch. But there are already roundabout ways of achieving the same and providing an official and well-documented one may be beneficial.

I also wanted to point out that, unlike the previous requests, we're not looking for a way to reduce duplication in the configuration.  There are genuine cases, where applying several location configs one by one is the most optimal way to get a desired behavior. E.g., step-up authentication or reuse of the result of another access module -- the existing methods of implementing such configurations are complex and may negatively affect max RPS/request processing time.

---
It doesn't have to be a redirect either; that was just the most direct approach. The `error_page ...; return` idiom could work for these scenarios, if we could have a conditional return-like directive processed at POST_ACCESS or PRECONTENT. Although it would be less obvious and with its own share of pitfalls.

    location /protected {
        ...

        include                regular_error_pages.inc;

        error_page             418 = @extra_auth;
        recursive_error_pages  on;
        post_auth_require      $not_suspicious  $var_from_subrequest ... error=418; # semantics similar to auth_jwt_require
    }

If I failed to convince you on the topic of redirect, would you be open to a patch with that kind of directive?